Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
•
•
•
•
•
•
•
•
Evidence base for VBR methodologies
Dr Máire Kerrin
Work Psychology Group
www.hee.nhs.ukwww.hee.nhs.uk
What are values?
• There is a complex relationship between values and other attributes
such as personality, ability and motivation.
– Values are motivational goals that influence behaviour.
– Values primarily impact the goals that individuals choose to pursue
(goal content) while personality traits primarily impact the amount of
effort that individuals exhibit in pursuit of those goals (goal striving).
– Personality represents behaviours that come most naturally, whereas
values reflect effort (a choice) to behave a certain way. This is an
important distinction when considering selection tools.
• Assessing values in recruitment is more challenging than assessing
abilities and skills.
www.hee.nhs.ukwww.hee.nhs.uk
Key Messages from the VBR Literature Review
• VBR is only one part of embedding the NHS values, research emphasises the
need for a multi-faceted approach.
• Individuals recruited with optimal values for the delivery of high quality
compassionate care, could lose these values if placed within teams with
suboptimal values.
• Evidence relating to VBR directly is limited, yet there exists a plethora of
measurement tools claimed to be of use.
• A single VBR tool used for attraction and marketing purposes may be
appropriate.
• When making selection decisions, a tailored approach is more likely to
accurately assess the diverse requirements of different job roles across the
NHS.
• Recruiters (particularly when interviewing) should represent the values the NHS
is seeking to attract and be trained to assess for values during recruitment.
www.hee.nhs.ukwww.hee.nhs.uk
Evaluation criteria for selection methods
• Evaluation criteria with which to judge the effectiveness and efficiency of
methods for selection methods have been established.
(See Arnold et al, 2010; Smith & Robertson, 1993; Schmitt, 2012;
Patterson, 2008; 2012; Cleland et al, 2012)
• Sixteen key evaluation criteria relevant to the VBR agenda.
• Each criterion is not mutually exclusive - it is possible for some criteria to be
at odds with one another.
• Recruiters may decide to prioritise (weight) certain criteria.
www.hee.nhs.ukwww.hee.nhs.uk
Evaluation CriteriaAccuracy and
effectiveness
1. Evidence of reliability
2. Evidence of validity
3. Arrangements for on-going validation, evaluation and
development are in place
4. Susceptibility to coaching
5. Fairness, promotes diversity/widening access
6. Legality
Costs and efficiency 7. Scalability for high volume recruitment
8. Efficiency
9. Utility
10. Generality of use
Practicalities and
implementation
11. Practicality (ease of administration)
12. Expertise required for analysis of information generated by
the tool
13. Ease of interpretation
Stakeholder
acceptance and
feedback
14. Positive employee/trainee/student perceptions
15. Generates appropriate feedback
16. Educational impact/value
Evaluation criteria for selection methods
www.hee.nhs.ukwww.hee.nhs.uk
•
Review of selection methods
Selection Method for
VBR
Reliability Validity Candidate
acceptability
Cost (to the
organisation)
Promotes
diversity
Susceptibility
to coaching
Traditional Interviews Low Low High Moderate to high Low High
Structured Interviews e.g. competency-based,
situational, multi-mini
interviews
Moderate to
high
Moderate High Moderate to high Moderate Moderate
Group Interviews Low Low Moderate Moderate Low High
Personal statements Low Low High Low to moderate Low High
References Low Low High Low to moderate Low N/A
Situational judgement
tests
High High (only if
based on a robust
psychometric
methodology)
Moderate to
high
Low to moderate High Low to
moderate
Personality testing High Moderate Low to
moderate
Low to moderate Moderate Moderate to
high
Selection centres
using work samples e.g. group exercise,
written/in-tray task,
presentations, interactive
exercises
Moderate to
high
High (only if
exercises are
used in
combination
based on a multi-
trait, method
approach)
High High Moderate Moderate
www.hee.nhs.ukwww.hee.nhs.uk
Implications for pre selection in VBRSelection
Methods
Implications of the evidence for implementing VBR
Personal
statements
Whilst candidate acceptability is high, susceptibility to coaching is also
high. (Ineffective method for VBR)
References Use of references remains widespread despite little research supporting
validity or reliability. (Ineffective method for VBR)
Situational
judgement tests
Improved validity over other selection tools (IQ and personality tests),
and can be mapped to organisational values. Whilst SJTs can be
relatively costly to design, SJTs are machine-markable and can be
delivered on-line, producing cost savings in high volume selection.
(Effective method for VBR)
Personality
testing
Concerns regarding faking in operational selection. Where there is a
high risk of susceptibility to coaching, personality assessment is best
used to drive more focused questioning at interviews (rather than a
stand-alone instrument without verification). (Personality tests may be
more useful at the attraction phase of VBR as part of self
assessment/selection.)
www.hee.nhs.ukwww.hee.nhs.uk
Selection Methods Implications of the research evidence for VBR
Traditional Interviews Across most evaluation criteria, traditional interviews perform
poorly. (Ineffective for VBR)
Structured Interviews
e.g. competency-based,
situational, multiple-mini
interviews
Interviews based on a thorough role analysis, use structured and
standardised questions with trained interviewers and appropriate
scoring can be reliable and valid. Candidates prefer interviews to
other methods although they are relatively resource intensive.
(Effective method for VBR)
Group Interviews Whilst group interviews appear more cost efficient in terms of
assessor time, evidence for reliability, validity and fairness is
lacking. (Ineffective method for VBR)
Selection centres
using work samples e.g.
group exercises,
written/in-tray task,
presentations, interactive
exercises
When designed appropriately (using a multi-trait, multi-method
approach with work samples), SCs are valid predictors of job
performance. Candidates are positive towards SCs as they have
multiple opportunities to perform. SCs are relatively expensive to
design and implement. (Effective method for VBR)
Implications for selection in VBR
Values-Based Recruitment Interviews (VBI)
Train the Trainer
Overview of Training Package Content
2014
VBI Train the Trainer: Objective
“To ensure individuals attending the
workshop have the requisite skills,
knowledge and confidence to be able to
train colleagues within their own
organisation on VBR interview technique”
Train the Trainer: Workshop Objectives• Create opportunities for learning, sharing knowledge and discussing
experience in relation to VBR.
• Refresh and clarify knowledge of assessment best practice, with
particular reference to VBR.
• To summarise the VBR evidence base, including recent literature review.
• Provide information about effective VBI practices.
• To increase understanding of VBI techniques, including interviewer
skills.
• Provide support in critically appraising existing VBI practices and
effectively implementing new VBI practices.
• Provide opportunities for interaction, participation and feedback.
• Facilitate peer support networks.
Workshop Content: Overview
• Two days (10am-4pm), approx. 10 delegates & 1
facilitator; designed with flexibility.
• Blended approach to learning: classroom based learning
and practical activity; emphasis on experiential learning.
.Day One (All Delegates)
• Overview of VBR: Background, Context & Evidence Base
• Understanding of VBI Techniques
• Refresh of Assessment / Interview Skills
• VBI in Practice: Practical Skills Development Sessions
Workshop Content: Overview
Day Two – Option One
(VBI Implemented)
• Key Considerations for
VBI Implementation
• Enhancing
Understanding of VBI in
Others
• Training Others in VBI
techniques
Day Two – Option Two
(VBI not implemented/in refinement)
• Review of VBI progress
• Key Considerations for
VBI Implementation
• VBI Action Planning
• Training Others in VBI
techniques
Evaluation Strategy
• Pre- and post-training evaluation utilising evidence-based
model1, focused on three key learning outcomes.
• As a result of this training intervention…
– Cognitive - Has knowledge increased (moving from novice to
expert) relating to values-based recruitment interviews?
– Behaviour / Skill - Has task proficiency and competence
relating to values-based recruitment interviews improved?
– Motivational / Affective - Has motivation and commitment to
the concept of values-based recruitment increased?
Documentation: Overview & Key Features
• High-quality, relevant content
• Ensure consistent approach to train the trainer delivery
– Clear, detailed materials and guidance documentation
• Consideration of ‘paperless’ delivery
• Two Suites of Materials:
1. Train the Trainer Workshop
2. Delegate Materials: ‘take home’, VBI training in Trusts
• NHS Employers Corporate Branding
• Flexibility of Delegate Materials
– Editing enabled for adaptability
Documentation: Train the Trainer
• Train the Trainer Presentation Slides (x 3)
• Facilitator Guidance
• Delegate Pre-Reading
• Delegate Learning Log (x 2)
• Supporting Exercise Material (x 1)
• Certificate of Completion
Documentation: Delegate Materials
• VBI Example Training Content (Slides)
• Example Exercise Templates
• Example VBI Training Pre-Reading
• VBI Training Facilitator Guidance
• VBI Training Evaluation Templates
• VBI: A Resource for Managers
Documentation: Evaluation
• Train the Trainer: Guidance for Evaluation
– Including Six Month Follow-Up Interview Guide
• Pre-Training Evaluation Questionnaire
• Immediate Post-Training Evaluation Questionnaire
•
•
•
•
•
•
•
Situation
Task
Action
Result•What did you learn from it?
•What impact did it have on you?
•What would you do differently?
•How did you feel about it?
•Why did you decide to…?
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Thank You
Any Questions?
Recruiting for Values in HEIs
Values Based Recruitment (VBR) Partner Workshop
Sandra Hatton, Director Project One
18 July 2014
Etc Venues, Birmingham
www.hee.nhs.ukwww.hee.nhs.uk
NHS Employment Journey – A Continuum of Values Based Employment
Values Based Recruitment (Values tested at multiple assessment points)
Attracting Candidates
Values of NHS Constitution marketed to prospective candidates
(students, trainees and employees),including use of
NHS Careers Service.
Pre-selection Selection
Values Based Environment
Values Based Employment Systems
Post SelectionEntry into
Employment & Beyond
Values based short-listing
criteria.Pre-selection
tools to assess values.
Use of selection tools, methods and approaches to assess values.
Evidence of values in
education, training,
development and organisation
culture.
Embedding values in organisation processes and
continuous learning and professional
development.
Culture & Leadership
Education, Training & Continuous Development
Values Driven Behaviour
Recruitment Post selection
NHS Constitution
www.hee.nhs.ukwww.hee.nhs.uk
Programme deliverables
By October 2014, the VBR programme will deliver a national values
based recruitment framework and suite of associated tools and
resources.
Other key deliverables include:
• Common working definition of values based activity
• Evidence to support employers and HEIs in recruiting for values
• Tool to support NHS organisations in testing and mapping their locally
agreed values to those of the NHS Constitution
• Access to values based interview and assessment training
• Methodology for measurement of long term impact and benefits
www.hee.nhs.ukwww.hee.nhs.uk
National VBR framework
HEE will oversee delivery of a national values based recruitment
framework and suite of associated tools and resources by October 2014.
The framework will:
• provide a common set of evidence-based national VBR principles and
standards against each stage of the recruitment process (attraction,
screening, selection and induction);
• provide guidance and access to resources to successfully recruit in HEIs,
NHS employers and LETBs and to prepare organisations for VBR;
• provide easy access to a toolkit of resources and evaluated techniques;
and
• access to good practice and case studies.
www.hee.nhs.ukwww.hee.nhs.uk
Stage Good practice standards for VBR
Attraction
• Marketing materials, websites and job descriptions and person
specifications include specific reference to the NHS
Constitution / Local Values.
• Local values (where they exist) are mapped to those of the
NHS Constitution. Evidence of
involvement
of patients
and public in
some stage
of the
recruitment
process.
Screening
and/or
Selection
Where designed appropriately, the following are considered to be
evidence-based approaches for VBR:
• Situational Judgement Tests (SJTs);
• Structured interviews;
• Multiple Mini Interviews (MMIs); and
• Selection Centres.
As good practice, the overall selection process should encompass
a structured face-to-face interview on an individual basis.
Post
selection
• Providing feedback to unsuccessful candidates if requested.
• Embedding the NHS Constitution in induction processes.
Professional and regulatory standards are integral to the overall selection process.
National VBR Framework
In October 2014, HEE will launch a national Values Based Recruitment
framework, including associated resources and good practice standards, to
support HEIs and providers to effectively recruit for values.
Case Study HEIs: Using Multiple Mini-Interviews (MMIs)
to recruit midwifery students with the right values
Alison Callwood, Teaching Fellow, University of Surrey
Presented by Jan Zietara,
Project Manager for Project 1 ‘Recruiting into HEIs’
www.hee.nhs.ukwww.hee.nhs.uk
Case study: Using MMIs to recruit midwifery students with the right values
• MMI development informed by current evidence including: K. Eva, McMaster
University, Ontario; C. Roberts, Sydney Medical School; A. O’Brien St George’s,
University of London
• Objective structured clinical examination (OSCE) style format
• Candidates respond to questions relating to a specific scenario at a ‘station’ in a
timed circuit e.g. x7, 4 minute stations with 1 minute between
• Each station scenario is designed to assess a specific attribute/NHS value
• No prompt questions and no dialogue between interviewer and candidate
• Practitioners integral to process
• Applicants assessed and rated against generic and station specific criteria
• Preliminary research shows reliability (internal consistency) is excellent
• Predictive validity: awaiting data
www.hee.nhs.ukwww.hee.nhs.uk
Scenario Example: Initiative and problem solving
You have recently taken the initiative to volunteer at a homeless shelter. On
one particular day you meet a 42 year old woman who is dying from an alcohol
related liver disease. Her skin and eyes are very yellow from a lifetime of
alcohol abuse and she complains of significant generalised pain which
worsens with movement.
She confides in you that she hasn’t had a drink in two days and “wants to get
hammered” before she dies but cannot get the bottle of vodka out of her
locker. You know that there is a strict policy for residents that alcohol is not
allowed in the shelter. There is another woman lying on her side three beds
away who is watching your interactions closely.
What would you say or do? Please explain…
www.hee.nhs.ukwww.hee.nhs.uk
Selected FeedbackStudents
“The questions are challenging but this is what interviews are supposed to be about,
pushing yourself to show your true personality and nature not rehearsed answers”
“You can move from a bad station in the MMI and not let it impact on other stations”
“In the traditional interview I felt I could relax more, in the MMI it felt like I was starting
all over again at each station”
“It doesn’t allow you to get a feel for staff like the personal interview…we are picking
the University as well as the University picking us”
Interviewers
“I think it gives you a much better idea than our current interview of whether they are
able to rationalise, analyse, synthesise …you can see those students who have very
concrete thinking and those who are able to look at both sides of the argument”
“I think it does demonstrate caring, certainly in my scenario with the alcoholic, it came
over very clearly… I found the compassion came out much clearer…”
“What’s quite nice about this is that… they come in and sit down and they are the
person they are and we are not biased by anything on their application forms”
www.hee.nhs.ukwww.hee.nhs.uk
On-going Research: Longitudinal Studies
(predictive validity)
• Sept 2014 BSc midwifery studies students exploring correlations between
their MMI score at interview and subsequent programme performance at
end of years 1, 2, 3 (n = 60).
• The University of Surrey is replacing the personal interview with MMIs for
all pre-registration health care student selection in 2014-15 recruitment
cycle. A large follow-up study is planned (n > 350), commencing Autumn
2014.
• For further information [email protected]
www.hee.nhs.ukwww.hee.nhs.uk
Thank you &
Questions
www.hee.nhs.ukwww.hee.nhs.uk
Contact details and further information
• Via email: [email protected]
• http://hee.nhs.uk/work-programmes/values-based-recruitment
• Via twitter: @NHS_HealthEdEng
@NHSE_Caroline
www.hee.nhs.ukwww.hee.nhs.uk
Key Questions
Q1. Looking at the table in the draft framework, please discuss and feedback
any concerns with the good practice standards included?
Q2. Are there any gaps we need to consider?
Q3. What are your challenges to using values based recruitment methods?
NHS Jobs
Values based recruitment
Agenda
• Introduction
• NHS Jobs update
• Pre-application and additional questions
• Scorecards
• Interviews
• Questions
52
Introduction
• OJEU
• Design and development sessions
• User acceptance testing sessions
• Pilot – June 2013
• Train the trainer – October 2013
• Launch date – 4 March 2014
• Planned activity
• Helpdesk activity
• Future development
53
NHS Jobs Usage
54
0
5,000
10,000
15,000
20,000
25,000
March (13/14) April (13/14) May (13/14) June (13/14)
Adverts Placed
NHS Jobs 1
NHS Jobs 2
NHS Jobs Usage
55
0
100,000
200,000
300,000
400,000
500,000
March (13/14) April (13/14) May (13/14) June (13/14)
Applications submitted
NHS Jobs 1
NHS Jobs 2
News
56
System Administrators were reduced to max 3 per organisation on 31st May – However, please ensure you have more than 1!
There was a software update on 4th June, which contained:– Fix for some issues when printing interview schedules– Fix for jobseeker error when requesting re-send of activation code– Fix for error when changing the vacancy team– Fix for error received when finalising shortlisting in some scenarios
On 12th June, the software update contained:– Supporting Information on applications from 1000 to 1500 words– Fix for Twitter adverts– Fix to prevent referees from being chased after email address is
changed– IIP standard logo added
News
57
On 26th June, the software update contained:– Improvements to candidate search– Improvement to allow Vacancy Bulletin report to be downloaded– Ability to report on primary and/or secondary organisation
Today, 10 July, the software update contained:- Ability to print the application form declaration section- Include information on when interview invitations were sent to
applicants- Include interviewees that have declined or withdrawn in the
interview schedule - Recruitment teams having the ability to un-reject applicants,
returning them to the logical previous status- Ability for Recruiting Managers to record interview outcome appoint
recommendations
Documents and Links
Documents and Links
Pre application and additional
application questions
Creating a scorecard template
NHS Jobs
Creating a scorecard template
Questions
Creating a scorecard template
NHS Jobs
Creating a scorecard template
Creating an interview
Creating an interview
Creating an interview
Questions
Questions
Questions