An evaluation of the Therapy Competency Logbook from the … · Service Evaluation Project Evaluation of the TCL Prepared on the Leeds D.Clin.Psychol. Programme, 2016 3 1. Introduction

An evaluation of the TherapyCompetency Logbook from the

supervisor perspective.

Mandip Dosanjh

Commissioned by Dr Tracey Smith and Dr Ciara Masterson

Service Evaluation Project Evaluation of the TCL

Prepared on the Leeds D.Clin.Psychol. Programme, 2016 2

Contents Page

1. Introduction............................................................................................................... 31.1 DClinPsy Programme Accreditation......................................................................... 31.2 Evidencing Competency ........................................................................................... 31.4 Existing Competency Frameworks ........................................................................... 41.5 The Leeds DClinPsy Competency Framework......................................................... 51.6 Placements ................................................................................................................ 5

2. Methodology .................................................................................................................. 62.1 Design Amendments.................................................................................................. 62.2 Design ....................................................................................................................... 62.3 Participants............................................................................................................... 72.4 Procedure .................................................................................................................. 72.4.1 Recruitment............................................................................................................ 72.4.2 Consent .................................................................................................................. 72.4.3 Data Collection ...................................................................................................... 82.5 Data Analysis ............................................................................................................ 82.6 Credibility checks ..................................................................................................... 82.7 Ethics......................................................................................................................... 9

3. Results ............................................................................................................................ 93.1 Interviews.................................................................................................................. 93.2 Validation process................................................................................................... 173.3 Recommendations................................................................................................... 18

4. Discussion..................................................................................................................... 194.1 Key findings............................................................................................................ 194.2 Strengths ................................................................................................................. 224.3 Limitations .............................................................................................................. 234.4 Dissemination of results.......................................................................................... 23

5. References .................................................................................................................... 256. Appendices................................................................................................................... 27



1. Introduction

This service evaluation project (SEP) investigated placement supervisors’ views on using

the Therapy Competency Logbook (TCL) to assess trainees’ competencies on the

Doctorate in Clinical Psychology (DClinPsy) training programme, at the University of

Leeds (UoL).

1.1 DClinPsy Programme Accreditation

DClinPsy programmes must comply with the Health Care Professions Council

(HCPC;2015) standards and the British Psychological Society will accredit programmes

according to their own standards for accreditation (BPS, 2017). The BPS and HCPC

provide standards of proficiency to ensure safe and effective practice, thus, developing

competent “psychologists who will be fit for purpose” (BPS, 2014, p. 1). These

professional bodies have a significant role in developing guidelines to inform the

curriculum of DClinPsy programmes; therefore, courses are often revising their

curriculum to maintain their accreditation. Whilst the BPS (2014) expect all standards to

be met, they are not prescriptive in how standards are achieved. They recognise that

programmes will achieve the standards in different ways based upon their distinctive

course identities (BPS, 2014).

1.2 Evidencing Competency

Demonstrating trainees’ competence in the delivery of therapy has always been a course

requirement; however, in 2014 the BPS revised guidelines stated that courses must find a

way to ‘evidence’ trainee’s therapy skills “in at least two evidence-based models of

formal psychological interventions, of which one must be cognitive-behaviour therapy”

(BPS, 2014, p. 12). This shift to evidencing reflects professionals’ opinion that

psychological interventions are delivered based upon a set of underpinning skills and that



these skills should be understood and deployed by those delivering such interventions

Roth & Pilling, 2015).

The BPS were not prescriptive in how courses operationalised this new requirement;

however, therapy skills were required to be evidenced against a “credible and robust”

framework (BPS, 2014, p.24) and where possible “benchmarked against recognised

criteria” (BPS, 2014, p. 24) as this would produce a credible competency framework “to

quality assure trainee work against” (BPS, 2014, p.29). The transition to competency-

based training presents many challenges for how clinical competence can be

implemented and assessed (Tweed, Graber & Wang, 2010). For instance, competencies

described at a general level result in everyone meeting the specification whereas

competencies that define too exhaustively result in many not fulfilling the criteria (Roth

& Pilling, 2008).

1.4 Existing Competency Frameworks

The BPS (2015) have provided their own version of a competency framework, it is called

‘the BPS Centre for Outcomes Research and Effectiveness’. The BPS encourage courses

to use or adapt their competency framework to evaluate trainee skill development.

The BPS (2015) centre for outcomes research and effectiveness (CORE) produced a

competency framework to evaluate trainee’s skills development and they encouraged

courses to use or adapt this framework. The BPS provide examples of therapy

frameworks in CBT, Psychodynamic and Systemic Therapies. In addition, courses could

also draw upon existing frameworks, such as Roth and Pilling’s (2008) CBT competency

framework and the Cognitive Therapy Rating Scale (CTS-R). Roth and Pilling’s (2008)

CBT framework was informed by manualised CBT interventions used within randomised

controlled trials in the treatment of anxiety and depression. Their framework is

considered a tool to inform programme curriculum, whereas the CTS-R is more useful to

capture assessment of adherence and competence, as it is a validated measure of clinical

competence in the delivery of CBT during training (Blackburn, James, Milne, & Reichelt,



2001). Without such clear and formal processes to evaluate trainees’ competencies, the

evaluation process becomes an idiosyncratic and individualised rating process (Tweed et

al., 2010); therefore, the introduction of evidencing trainees’ therapy skills systematically

provides a quality check system of competence.

1.5 The Leeds DClinPsy Competency Framework

In 2014, the Leeds DClinPsy programme developed the TCL in response to the BPS

guidelines. The document includes a range of therapeutic models for trainees to evidence

their plus one other model of therapy from: Cognitive Analytic Therapy, Systemic,

Acceptance and Commitment Therapy, Compassion Focused Therapy and Intensive

Short-Term Dynamic Psychotherapy and psychodynamic. The TCL was developed partly

based on Roth and Pilling’s (2008) framework and partly from consultation with local

experts and supervisors to develop each model-specific competency framework. The

involvement of local clinicians may have increased the ecological validity of the

competency framework, as Roth and Pilling’s (2008) competency framework emerged

from clinical trials, and thus, lacks representation of how routine clinical practice

corresponds to the framework.

1.6 Placements

In accordance with the BPS (2014) regulations, all placements are supervised by clinical

psychologists who have completed training in supervision that has either been recognised

by the BPS or provided by the UoL. The UoL also provides regular workshops on

supervisory skills and training events to ensure their trainees receive effective

supervision. Trainees are provided with the TCL before their first placement commences.

They are advised to focus on completing the CBT competency within the first two years

of training, whilst focusing on the plus one model in the final year. The course expects

both supervisors and trainees to contribute to the completion of the TCL.



Aims

The course was interested in the supervisor’s experiences of using the TCL. The SEP

sought to understand whether:

the TCL is a useful and meaningful document to assess trainees’ competencies.

supervisors understand how to use the TCL to assess trainees’ competencies.

any adaptions or changes are required to the TCL.

supervisors have any training needs in relation to the TCL.

2. Methodology

2.1 Design Amendments

The final design was qualitative methodology; however, the original design was mixed

methods. Data from the focus group (stage 1) was intended to generate the online

questionnaire (stage 2). Recruitment for the focus group was successful, however, it was

impossible to get everyone together and therefore represented an unfeasible data

collection method. An amendment was requested to conduct short individual interviews

(stage 1) to then inform the online questionnaire (stage 2); however, unexpectedly, the

initial analysis of the individual interviews provided rich data. Therefore, it was

considered more ethical to use the resource of time to analyse the data set to its fullest

potential as opposed to beginning to develop the questionnaire. An amendment was then

requested to withdraw stage 2 and a validation process was implemented instead.

2.2 Design

Participants took part in a brief individual interview. Participants were given a choice of

telephone or face-to-face interview. A semi-structured interview schedule was used to

guide the interview (see Appendix 1). Interviews were then transcribed and analysed

using thematic analysis (Braun & Clarke 2006).



2.3 Participants

Inclusion criteria: qualified clinical psychologists who were supervisors for the DClinPsy

placements, who either were currently supervising or had supervised within the last three

years.

Exclusion criteria: not a qualified clinical psychologist or had not completed the

supervisor training.

The sample size was determined by current thematic analysis guidelines.

Thematic analysis can be applied to a minimum sample size of two participants (Fugard

& Potts, 2015). In addition, the project aimed to recruit at least one representative from

the five main clinical areas that clinical psychologists work in (health, forensics, child,

adult mental health and learning disabilities); therefore, the sample size aimed for 2-5

participants. 260 supervisors were emailed to take part in the project. The response rate

was five participants (child n=3, health n=1 and forensic n=1) and I checked that they had

offered a placement over the last three years.

2.4 Procedure

2.4.1 Recruitment

Participants were recruited from the placement supervisor list. The DClinPsy

administrative team sent an email invitation to the supervisors, with the participant

information sheet (PIS) (see Appendix 2). Participants responded to the principal

investigator by email if they wanted to participate. Recruitment operated on a first come

first served basis, with a view to selecting the most representative sample in terms of the

clinical areas; thus, using purposive sampling to ensure a representative sample.

2.4.2 Consent



The participants signed a consent form (see Appendix 3) before the interview was

conducted.

2.4.3 Data Collection

All participants chose a telephone interview. Interviews lasted approximately 20-30

minutes and were recorded. The audio file was then transferred to a secure drive on

university computers.

2.5 Data Analysis

The interviews were transcribed verbatim and analysed using thematic analysis (Braun &

Clark, 2006). The researcher followed Braun and Clarke’s (2006) four phases of data

analysis (see Appendix 4). Thematic analysis is one type of qualitative methodology and

“is a method for recognising and organizing patterns in content and meaning in

qualitative data” (Willig, 2013, p.58). Thematic analysis is appropriate to explore realist,

phenomenological or social constructionist research questions (Willig, 2013); therefore, it

was a suitable method of analysis, as the research question was interested in the

supervisors’ experiences of using the TCL. Other qualitative approaches were considered,

such as grounded theory (GT), but GT was considered inappropriate as the project did not

aim to generate a theory from the data to explain the phenomenon being investigated

(Charmaz, 1995).

2.6 Credibility checks

The initial themes and refinement of themes were quality checked with the commissioner

on several occasions and the commissioner was in agreement with how the themes were

developing. Furthermore, raw data extracts have been incorporated to support the themes.

In addition, a validation process was incorporated and the themes were presented at a

supervisor training event to see whether the themes fitted with their experiences of using

the TCL. The supervisors chose from responses: agree, disagree or don’t know (see

Appendix 5).



2.7 Ethics

Ethical approval was provided by the UoL School of Medicine Research and Ethics

Committee on the 5 July 2017. Informed consent was provided by participants to take

part in the project after reading the PIS and in addition, they were given the opportunity

to ask questions.

3. Results

3.1 Interviews

The analysis of data produced four themes and 12 subthemes (see figure 1). These will be

discussed in more detail below.



Figure 1: thematic map

Theme 1: Challenges

The supervisors can experience challenges when using the TCL.

Pressure: Supervisors can experience a sense of pressure to provide the right experiences

on placements to ensure successful completion of the TCL. They feel that either the

service context (for example, forensic service) or their own limited experience is often in

conflict with the TCL or trainees’ expectations; therefore, at times, making it difficult to

meet competencies.



“we have a family therapy team here…she was more the reflecting team...because

she couldn’t be the interviewer… I think some of it was harder, to score her on.”

Participant 3

“… if it was just up to me, to say that I am passing somebody on… say a CAT

placement, that would be really worrying…I’ve had third years come before,

wanting CAT and I said we need to get you extra CAT therapy because I’m not,

erm, au fait with that.” Participant 5

Supervisor Opportunities: Some supervisors would like to be able to provide more

expertise in other models, however, these opportunities are limited due to funding or

time. Therefore, sometimes supervisors can feel deskilled in meeting the trainees’ and the

TCLs expectations.

“…they happen to be supervised by somebody who has taken the time and effort

and possibly money to do that extra training…which is hard given that there is

hardly any time or funding available for extra training.” Participant 2

“…if this is what trainees are coming to me and they need, I need to be able to

provide that… I know very little about ACT, but I would like to know more.”

Participant 5

Finding a way: In light of such challenges, there is successful navigation and supervisors

are able to find a way to complete the TCL. Supervisors have learnt to think flexibly

when evidencing trainees’ competencies.

“we do use narrative and solution focused… and I suppose the skills of either, sort

of wangled in the generic competencies or we’ve looked at how closely aligned

they are to aspects of other models...” Participant 1



Theme 2: Uncertainty

Supervisors do have some doubts concerning the purpose of, and how to use, the TCL.

Level of evidence: Supervisors are unsure about how much evidence is required and

what format the evidence should take to prove a trainee’s competencies.

“how much evidence do we need to be able to confidently tick a box? Is it enough

for a trainee to say oh yeah I did that in a session or is it something that we have

to see on video or hear on audio or to actually see that, you know a formulation,

to see the visual product of that.” Participant 1

Truthful Reflection: Some supervisors are curious as to whether the TCL can truly

capture a trainee’s competence. They wonder what ticking a box represents, as doing

therapy and doing it well are two different things.

“…it tells you that they’ve done something but it does not tell you the quality of

that generic assessment or the quality of the initial formulation.” Participant 5

“…could somebody tick all the boxes and still be not very good at doing therapy?

Well, er, yes…” Participant 4

Also, some supervisors feel that a missed opportunity within the TCL is not being able to

evidence a trainee’s skills in models that are not listed in the TCL.

“It does sort of constrain, you know, what you can mark people on. They might

have lots of fantastic skills but, they’ve got to fit, you know, those six boxes.”

Participant 3

Capturing Therapy: Supervisors are hesitant as to whether any document could truly

capture therapy skills, as this is a difficult task. They perceive the quantifying of therapy



skills as a reductionist approach that potentially conflicts with what clinical psychology

is.

“I don’t think it’s useful as a mechanistic, yes you can do this, no you can’t do

this…I’m not sure anything works that well as a kind of a black or white, pass or

fail assessment…because it’s really experiential isn’t it? therapy and really hard

to capture that and to quantify it…” Participant 4

“…relying purely on, kind of accumulative description of things is rather removed

from what that skill will actually in tail in practice after qualification.” Participant

2

In addition, some supervisors think that the TCL is ‘missing something’ when assessing

trainees competence.

“It’s very much an approach and an attitude it’s not just, about ticking off doing

certain things. You could, do a lot of these things and still not really be working

systemically…” Participant 3

“…there is something that happens with a therapist and a client and I think it’s

missing that… it’s more than the alliance… whatever it is that makes, helps that

person come back to a session week after week. It’s that, that I don’t think is

being captured…” Participant 5

A living document: Some of the supervisors have noticed that the TCL is not used as an

active document throughout placement.

“… when it’s been presented to me, it’s usually been, around a mid-placement or

end of placement. I rarely see it at the beginning...” Participant 5



Furthermore, some supervisors seem unsure of how to integrate the TCL within the

placement and, because of this, the TCL can feel like an ‘add on’ and consequently a

tick-box exercise.

“clarity, in maybe… whether it’s intended to be, its primary function is around,

kind of, a memory aid for trainees after the course or kind of thinking about what

they’ve have achieved. Or whether it’s meant to be something that is meant to be

used more actively in supervision, that could be helpful.” Participant 1

“It felt like an add on, erm which perhaps it shouldn’t do, but it did… I do

wonder, where there’s a bit more help in thinking about embedding it, how we

embed it within supervision would be really helpful actually… But actually, a lot

of the things in there are really, really important. To make sure it just doesn’t

become a ticking exercise.” Participant 3

Some supervisors wonder whether the TCL should have a more active role within the

placement.

“…when I get my next one, erm, definitely on my radar now, to make better use

of it. Because it is really a comprehensive document, I think it could be used a bit

more.” Participant 5

Theme 3: Responsibility

This theme illustrates the role of responsibility and whose responsibility it is to complete

the TCL.

Partnership: When it comes to completing the TCL, trainees and supervisors have

different roles. Supervisors perceive the trainees to be responsible for taking the lead.



“I think the main thing is having a trainee who can understand what it is for and

how to use it. So they are able to guide the supervisor in how to most

appropriately use it as a tool.” Participant 2

Supervisors see their role as supporting as opposed to being responsible for the

completion of the TCL. Supervisors support the trainees to notice skills that they have not

logged but are competent in and they use the TCL as a reflective tool to support the

trainee’s development.

“Some of the trainees, have not remembered what they’ve done or they’ve not

given themselves credit, what they’ve done work wise and I’ve said no no,

highlight to them, that was a rupture, yes you dealt with that.” Participant 5

“I see our job as supervisors to help them think about where there might have

been challenges in achieving certain skills, perhaps the skills they’ve not managed

to tick off. Why that might be or if it’s just they missed something or not noticed

that they were doing something.” Participant 1

Collaboration: This partnership of leading and supporting produces a collaborative

process in completing the TCL. The collaborative process has been a positive experience

and has facilitated meaningful conversations.

“Yes, it’s defiantly been collaborative and there had been times where I have

reminded them, and they have reminded me and we’ve had real discussions about

it.” Participant 5

Change: Supervisors are wondering whether they need to take on more responsibility in

how the TCL is used. This shift to joint responsibility, may encourage better use of the

TCL within placement.



“I need to find a way of embedding it a bit better, rather than it being a, a bit more

of a tick box exercise.” Participant 3

Theme 4: Document Composition

Supervisors are generally satisfied with the format of the TCL and the document is fit for

purpose.

Direction: The supervisors feel that the TCL provides clear direction on how to assess

whether a trainee is competent as it breaks down the therapy skills, which helps the

supervisors to focus on what to look for.

“…it’s really helpful for having a framework for understanding what is it that you

are doing. You know, breaking it down…” Participant 4

“it’s really nice to have something that is really specific, so you know what you

are looking for. I was talking to colleagues just now, how we often don’t, talking

about how I, observing an trainee and kind of, what I’m sort of looking out for

and you know, how you do feedback. Erm, it’s nice to have a different model.”

Participant 3

The TCL also provides a framework for thinking as it directs the supervisor to identify

any gaps in the trainees’ development.

“…So it’s easy to see isn’t it, cos you’ve got a boxes, oh I’ve done that but we

haven’t done anything to do with that, so it does kind of prompt you, whereas if,

if you didn’t have it you might not think about all those aspects.” Participant 4



Structure: The supervisors think that the layout is clear, accessible to use, flexible as to

the information that can be entered, and the content is appropriate.

“I like that it’s got little columns and that it er directs you into where to put things

into it… I like that…almost, you do have a bit more freedom to put in what you

feel is relevant…” Participant 2

“…most things seemed really appropriate.” Participant 3

In addition, most of the supervisors think the reflective box is a good way of evidencing

learning and competencies.

“an opportunity to write down and reflect, what you have done with it…It’s more

about, kind of, being exposed to the concepts and using them. Having to write in

the reflection box, I guess is, is evidence of that isn’t it. That someone is engaging

with the concepts.” Participant 4

3.2 Validation process

The graph shows how strongly the themes resonated with a wider pool of supervisors

(n=11) (see figure 2). There was strong agreement for most of the subthemes, with the

exception of ‘supervisor opportunities’ and ‘collaboration’, which were less supported

but still relatively strong. On the rating document (see Appendix 5), some of the

supervisors (n=5) had not provided a rating for some of the subthemes but they had

written a qualitative entry, therefore, I made an assumption to determine their agreement

rating where possible, based on their qualitative comment.



0

20

40

60

80

100

120

Chart Title

Agree Do not agree Not sure

Figure 2: Graph to show the results from the validation process.

3.3 Recommendations

The recommendations from the interviews and the validation process have not been

arranged in relation to the themes, as they are standalone. The list of recommendations

was generated by the participants. Recommendations were not considered if they were

not feasible, not clearly expressed or impractical.

Evidence format: the TCL could have a column to indicate which format the

skill was evidenced in: for example, audio or live.

Model the TCL on the neuropsychology document: as it is better at evidencing

how the skill was proven.

Confidence rating in competency of that skill: doing a skill once may not be a

marker of confidence and thus competency; therefore, it could be useful to track

progress in confidence.

Further training in therapy models: the course could upskill the supervisors.



Consider evaluating process and relational elements of therapy.

Transparent rating scale: different levels of competency status: for example, a)

started doing, b) getting better, c) advanced.

4. Discussion

4.1 Key findings

The results indicated that generally the supervisors are satisfied with the document and it

is fit for purpose. In addition, the results have also identified adaptations that could

improve the TCL.

Aim 1: to establish whether the TCL is a useful and meaningful document to assess

trainees’ competencies.

The TCL is perceived to be a meaningful document as it can promote rich discussions

(‘Collaboration’) and some supervisors’ willingness to have a more active role in

completing the TCL (‘Change’) indicates that they see value in the document. The

supervisors also feel that the TCL is a useful resource to guide the assessment of the

trainees’ competencies (‘Document Composition’). The TCL also appears to be a useful

reflective tool to support trainees’ development in skills competency (‘Partnership’). The

supervisors also feel that the reflective element is a more meaningful process of

evaluation, compared to a standalone tick column (‘Structure’). The importance of

reflective practice has been argued by Schön to be just as important as technical

knowledge (for example, specific CBT techniques); and both are required to enable good

clinical decisions (as cited in Fisher et al., 2015, p. 731-732). Furthermore, the BPS

(2015. p. 45) state that trainees should “monitor and review their own progress and

develop skills in self-reflection and critical reflection on practice”. The TCLs

incorporation of logging reflections may be an important developmental skill for trainees.

Indeed a study found that clinical psychologists report the use of reflection and reflective



practice within their clinical work as benefitting them and their clients (Fisher, Chew &

Leow, 2015).

Aim 2: to establish whether supervisors understand how to use the TCL to assess

trainees’ competencies.

Generally, supervisors understood how to use the TCL to assess trainees’ competencies

(‘Document Composition’), although there are two issues. Firstly, supervisors are unsure

about how much and what format the evidence should take to verify a trainee’s

competence (‘Level of evidence’). Beck et al., (2014) outlined several different

approaches to assess trainees’ skills, other than hearing the trainee’s account of therapy.

This should include a combination of live observations, joint working and watching

videos of the therapy sessions, both during the trainee’s initial sessions and as they begin

mastering specific intervention techniques; all of which are methods of evidencing that

what the course advocates.

Secondly, supervisors are uncertain about how to integrate the TCL as an active

document within placement (‘A living document’). It has been suggested that even

though competence frameworks have been developed to inform a curriculum, such

frameworks can also guide the supervision process (Roth & Pilling, 2008) and it is

apparent from the analysis that this does not appear to be routinely happening on

placements. Therefore, the TCL may need a more active role within placement as it tends

to be introduced at the mid or end of placement visit. If the integration of the TCL within

placement was improved and embedded within supervision, it might be less vulnerable to

becoming a tick-box exercise or an afterthought, thus transitioning into a more

meaningful document with a purpose.

The subtheme ‘pressure’ does also indicate that supervisors are not entirely clear

on how to use the TCL as the DClinPsy programme does not expect supervisors to

comment on model skills that they themselves are not competent within. Furthermore,



Recommendations Training

- on the frequency and type of evidence required to verify a trainee’sskill competency.

- supporting supervisors to integrate the TCL as an active documentthroughout placement. For instance, embedding within supervision.

Changes to TCLTCL may benefit from a column to indicate which format the skill wasevidenced in: for example, audio. This may encourage the supervisorsuse of different observation techniques. Furthermore, this tangiblestructure, may provide supervisors with some clarity and confidence inevidencing competency.

Trainees and supervisors owning joint responsibility for completion ofTCL.

Supervisors need to go back to the TCL guideline documents. Inaddition, the course may need to be more explicit in explaining theirexpectations of supervisors.

this is also explained in the document, ‘guidance for supervisors’, which is given to

supervisors by the programme. If supervisors were more familiar with this document, it

could alleviate their anxieties around a need to meet the TCL and trainees’ expectations.

To improve the supervisors understanding on how to use the TCL to assess

trainees’ competencies, the course may want to consider the following recommendations

listed in figure 3.

Figure 3: Recommendation box

Aim 3: to establish whether any adaptions or changes are required to the TCL.

The results have shown that the TCL may benefit from being adapted, in light of

supervisors’ current experiences of using the document; some of these changes have



already been listed in figure 3, in addition further recommendations for changes are listed

in figure 4.

Figure 4: Recommendation box

Aim 4: to establish whether supervisors have any training needs in relation to the TCL

The results in relation to Aim 4 were highlighted in Aim 2. It was discussed in Aim 2 that

supervisors require further training to improve their understanding of how to use the

TCL. Firstly, supervisors are unsure about how much and what format the evidence

should take to verify a trainee’s competence and secondly, supervisors are uncertain

about how to integrate the TCL as an active document within placement. The

recommendations for these training needs are listed in figure 3.

4.2 Strengths

The initial themes and refinement of themes were quality checked with the commissioner

on several occasions. This credibility check reduced potential bias, and enhanced the

reliability of the findings and was conducted in adherence to guidelines for good

qualitative research (Elliot et al.,1999). Furthermore, raw data extracts were used to

support the themes. This transparency of analysis is considered an important check of

Recommendations: DClinPsy course to consider the following adaptations

Evaluating a trainee’s personal embodiment of a model. Evaluating a trainee’s competency in engaging with process and

relational issues within therapy. Transparent rating scale: recording skill competency in a more

transparent way in an attempt to determine the difference between beingable to do therapy well and doing therapy. It has been recommended thatrating scales can be a useful measure of professional competency as theyreflect growth and the development of skills as well indicating areas ofimprovement (Leigh et al., 2007).

A section for models that are not within the TCL.



quality, reliability and validity within qualitative research (Elliot’s et al.,1999; Yardley,

2008). In addition, a validation process was incorporated to see whether the themes

resonated with a wider pool of supervisors. This process of triangulation provides further

validity to the findings.

4.3 Limitations

The SEP aimed to recruit the most representative sample in terms of the clinical areas in

which that trainees work, however, the SEP was unsuccessful in securing a wide

representative sample, as there was no representation from adult mental health (AMH)

and learning disabilities (LD). Therefore, the findings may not necessarily resonate with

supervisors within these areas. A missed opportunity at the validation process that could

have rectified this would have been to collect information on which clinical areas the

supervisors worked within. This information could have provided a consensus on whether

the themes resonated with clinicians from AMH and LD. In addition, the nature of

qualitative methodology and the small sample size impacts upon the generalisability of

results to the wider population. There may also be bias in the participants who chose to

participate, for instance in their motivations.

4.4 Future research considerations

The course may want to consider commissioning a quantitative SEP that evaluates the

TCL from the supervisor’s perspective. The project could generate an online

questionnaire that has been informed by the findings of this SEP. A quantitative approach

will provide a broader perspective on the findings. In addition, in light of this project’s

limitations, this future research may allow for a more representative sample in terms of

the clinical areas that the supervisors work in.

4.4 Dissemination of results



A poster and oral presentation for this SEP were presented at the UoL SEP conference on

Friday 26th October 2018.



5. References

Beck, J. G., Castonguay, L. G., Chronis‐Tuscano, A., Klonsky, E. D., McGinn, L. K., & Youngstrom, E. A. (2014). Principles for training in evidence‐based psychology: Recommendations for the graduate curricula in clinical psychology. ClinicalPsychology: Science and Practice, 21(4), 410-424.

Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., &Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): psychometricproperties. Behavioural and cognitive psychotherapy, 29(4), 431-446

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitativeresearch in psychology, 3(2), 77-101.

British Psychological Society, BPS. (2014). Standards for Doctoral programmes inClinical Psychology. Retrieved September 30, 2018 from:https://www1.bps.org.uk/system/files/Public%20files/PaCT/dclinpsy_standards_approved_may_2014.pdf

British Psychological Society, BPS. (2015). Standards for the accreditation of Doctoralprogrammes in clinical psychology. Retrieved September 30, 2018 from:https://www1.bps.org.uk/system/files/Public%20files/PaCT/clinical_accreditation_2015_web.pdf

British Psychological Society, BPS. (2017). Standards for Doctoral programmes inClinical Psychology. Retrieved September 30, 2018 from:https://www.bps.org.uk/sites/bps.org.uk/files/Accreditation/Clinical%20Accreditation%20Handbook%20(2017).pdf

Charmaz, K. (1995). Grounded Theory. In J.A., Smith, R, Harré & L, Van Langenhove.(Eds.), Rethinking methods in psychology (pp. 27-49). London: Sage.

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication ofqualitative research studies in psychology and related fields. British journal of clinicalpsychology, 38(3), 215-229.

Fisher, P., Chew, K., & Leow, Y. J. (2015). Clinical psychologists’ use of reflection andreflective practice within clinical work. Reflective Practice, 16(6), 731-743.

Fugard, A. J., & Potts, H. W. (2015). Supporting thinking on sample sizes for thematicanalyses: a quantitative tool. International Journal of Social ResearchMethodology, 18(6), 669-684.

Leigh, I. W., Smith, I. L., Bebeau, M. J., Lichtenberg, J. W., Nelson, P. D., Portnoy, S., ... &Kaslow, N. J. (2007). Competency assessment models. Professional Psychology:Research and Practice, 38(5), 463.

Roth, A. D., & Pilling, S. (2008). Using an evidence-based methodology to identify thecompetences required to deliver effective cognitive and behavioural therapy fordepression and anxiety disorders. Behavioural and Cognitive Psychotherapy, 36(2), 129-147.

Roth, A., & Pilling, S. (2015). A competence framework for psychological interventionswith people with persistent physical health conditions report. NHS Education forScotland and the Improving Access to Psychological Therapies programme. UniversityCollege London. Retrieved February 2, 2019 from:



https://www.uea.ac.uk/documents/246046/11991919/Physical+LTC+Competence+Framework.pdf/fa58d42d-8df9-4b4f-bb00-a8681eaa0321

Smith, J., Flowers, P., & Larkin, M. (2009). Interpretative Phenomenological Analysis:theory, method and research. London: Sage.

Tweed, A., Graber, R., & Wang, M. (2010). Assessing trainee clinical psychologists' clinicalcompetence. Psychology Learning & Teaching, 9(2), 50-60.

Willig, C. (2013). Introducing qualitative research in psychology. London: McGraw-HillEducation (UK).

Yardley, L. (2008a). Demonstrating validity in qualitative psychology. Qualitativepsychology: A practical guide to research methods, 2, 235-251.



6. Appendices

Appendix 1: Semi Structured Interview Guide (Version 1) (30.06.18)

Introductions Thank them for taking part. Outline SEP aims and ask if they have any questions about the

evaluation. Explain confidentiality.

Setting the scene questions How many trainees have you used the TCL with? What parts of the TCL have you used with your trainees? Have you ever used other competency frameworks, other than the TCL?

Topic area: usefulness/practicalities of TCL

1. What do you think about the concept of competency description and evaluation?

2. How useful is the document to assess and record the trainee’s competencies?(Prompts: is there anything you like about the TCL document? is there anything you do not likeabout the TCL document? Any suggestions for improvements?)

Topic area: Support

3. How confident do you feel in completing the TCL?(Prompts: do you feel like you need more support in how to complete the TCL? What parts of theTCL may you struggle with/works well? What do you think contributes to this low/highconfidence? What would you suggest would help e.g. further training?)

Topic area: Understanding

4. When you have completed the form with a trainee, how exactly have you gone about it? E.g. haveyou completed it together?(Prompts: Do all trainees approach the task in the same way? Do you always approach it in thesame way?)

5. How have you found this experience?(Prompts: Is there anything that makes it easier or harder to complete?)

Topic area: Meaningfulness

6. How meaningful do you think the TCL is, as an assessment document to capture and record thetrainees’ competencies?(Prompts: What do you see as the value in completing the TCL? are there any drawbacks? how doyou feel about completing the TCL?)

Closing: Is there anything else that you think might be helpful for me to know?



Appendix 2: Participant information sheet (version 1) (30/06/18)

“Evaluating the Therapy Competency Logbook from the supervisor perspective”

You are being invited to participate in this service evaluation because you are aplacement supervisor for the Leeds Dclin course. This information sheet describes theService Evaluation Project (SEP) and explains what will be involved if you choose totake part.

Who is conducting the study?

My name is …….. and I am a Psychologist in Clinical Training on the Leeds DClinProgramme.

What is the purpose of this study?

The British Psychological Society (BPS) is a professional body in the UK forpsychologists and psychology. The BPS stipulates that for accreditation, the ClinicalPsychology training programme must evidence that trainees are competent in the deliveryof therapeutic models. The guidelines require trainees to be competent in the delivery ofCognitive Behavioural Therapy plus one other model of therapy.

In 2016, the course evaluated the Therapy Competency Logbook from the traineeperspective and there is now a need to complete this from the supervisor perspective. Weare interested in finding out about your experiences of using the log book. The interviewsare being held to generate data which can then be used to develop a questionnaire for thesecond part of the evaluation, which will be an online survey.

What will participating in this evaluation involve?

If you agree to take part in the evaluation you will be invited to take part in the first stage,which is a brief individual interview. The interview will involve having a conversationabout your experiences of using the logbook. The purpose of the interview is to informthe development of a questionnaire (second stage). The interview will either be face toface or over the telephone and will last around 20-30 minutes. If you choose a face toface interview, the venue will either be your place of work or the University of Leeds.The interview will be recorded using an encrypted Dictaphone.

Right to withdrawYour participation is voluntary and you have the right to change your mind if you wish tono longer take part. You may withdraw prior to or during the interview. Also, you haveone week to withdraw your data from the date of interview.



What will happen to the information I give?

Any information that you provide will be kept confidential. Any notes will not personallyidentify you and will be held securely (locked away when not in use) and shredded at thefirst opportunity. The audio recording of the interview will be immediately transferredfrom the recording device into an audio file, which will be stored on a secure server onthe university of Leeds computer network. The audio recording will then be immediatelyerased from the recording device after transfer to the computer. Your data on the audiofile will then be transcribed for the purpose of analysis and then permanently deletedfrom the server. Your transcript will be anonymous and will not have any identifiableinformation. Some of your anonymised quotations may appear in a report.

Dissemination of findings The SEP is a university assignment and a report will be written on the findings

and submitted for marking. The report will be disseminated to the clinical tutors at the university. The findings will be shared at the placement poster event at the University of

Leeds DClin Psychology course in September 2018. The findings may be submitted and published in the Clinical Psychology Forum,

in order to share good practice amongst training courses.

What will happen to my information after completion of the project?

All evaluation documentation will be hand delivered to the DClin research office for safekeeping. All electronic files will be copied over and stored on a secure drive and madeavailable to the department, which is restricted to a small number of research supportstaff. After 3 years all respective evaluation related documentation will be shredded andelectronic files will be deleted from the secure drive by research support staff

Confidentiality

The principle investigator will maintain confidentiality. Please be aware that a breach ofconfidentiality may be required if in the unlikely event there are concerns raised overyour practice and/or you disclose any information that is a cause for concern. In thisevent, the principle investigator will raise their concerns with you and wherever possiblewill discuss any action required with you first. Discussion may then need to take placewith third parties e.g. commissioner and supervisor of the SEP project.

What are the possible benefits of taking part?

The supervisor perspective could provide valuable information as to whether anythingneeds to be adapted or changed to ensure that trainee’s competencies are being assessedin a useful and time efficient way. Furthermore, it is important to know whethersupervisors feel confident in using the logbook to support trainees in documenting their



competencies. Also, the findings may indicate a need to change and/or offer additionaltraining and support to supervisors.

What are the possible disadvantages and risks of taking part?

You will need to give up some of your time and it is acknowledged that your availabilitymay be limited. There are no known risks of taking part in the project, however in theunlikely event of experiencing distress because of participating in this project; you cancontact Dr Tracey Smith (Clinical Tutor) at the University of Leeds.

Contact details

If you have any queries please email …….. (Psychologist in Clinical Training) on……...Alternatively, you can contact the commissioner of the project, Dr Tracey Smith [email protected]

This study has been reviewed and given ethical approval from the School of MedicineResearch Ethics Committee on the (05/07/2018), ethics project number (…….. ).



Appendix 3: Consent form (version 1) (30/06/18)

Evaluating the Therapy Competency Logbook from the supervisor perspective

I have read and understand the information sheet provided forthis SEP evaluation.

I have been given the opportunity to ask questions about theSEP and the answers have been satisfactory.

I understand that my participation is voluntary.

I understand that I may withdraw prior to or during theinterview without giving a reason and I can withdraw my dataone week after the completed interview date.

I know that the interview will be digitally recorded and thentranscribed. I understand that any personally identifableinformation will be removed from the transcript of the interviewand that I will not be identifable in any future reports,publications or presentations.

I understand that the data collected will be securely stored on auniversity server and password protected. I am aware of whatwill happen to my data after the SEP is complete.

If you would like a summary of the SEP, please provide anemail address, so it can be emailed

………………………………………………………………………………………………………………………………………………

Participant’s Name (Printed): ________________________Participant’s signature: _________________________ Date: __________Principle investigator’s signature: _______________________ Date: __________

Thank you for agreeing to take part in this study.Your contribution is very much appreciated



Appendix 4: Table summary of the phase of Thematic Analysis (Braun & Clarke, 2006)

Phase Description of the process

Phase 1

Familiarising yourself with

your data

involves immersion within the data in search of meaning and

patterns (reading and re-reading).

Note down initial ideas

Phase 2

Generating initial codes Generate initial list of ideas and what is interesting in the

data.

Produce initial codes from the data. The code identifies parts

of the data that are interesting to the analyst.

Organising data into meaningful groups.

Working systematically through the entire data set,

identifying interesting aspects in the data may form the basis

of repeated patterns across the data set.

Phase 3

Searching for themes

Sorting the codes into potential themes.

Beginning to analyse the codes and how differenee codes

may combiner to produce an overarching theme.

Phase 4

Reviewing themes

Refinement of themes is done by checking that they work in

related to the coded extracts (phase 1) and the entire data set

(phase 2), generating a thematic map.

Phase 5

Defining and naming themes

Define & further refine themes by identifying the essence of

the theme, the overall story and clear names for the themes.



Appendix 5: Rating Document for validation process

Tick, Cross, Question mark next to subtheme

Qualitative box

X

?

Conflict

1. Pressure

2. SupervisorOpportunities

3. Finding away

Documents

An evaluation of the Therapy Competency Logbook from the … · Service Evaluation Project Evaluation of the TCL Prepared on the Leeds D.Clin.Psychol. Programme, 2016 3 1. Introduction