Practice Placement Quality Assurance (PPQA) website ... Student Evaluation... · Practice Placement Quality Assurance (PPQA) website: Student Evaluation Questions A review update

Practice Placement Quality Assurance (PPQA) website: Student Evaluation Questions

A review update >>>>>>> Fiona Bates Quality Assurance Systems Project Manager 7th October 2013

http://webyd01/documents/documents/HE-Yorkshire-and-the-Humber-Col-v2.png

PPQA: Student evaluation review Page 2 of 17 7th October 2013

Contents

Please click on the section title to access the information required

Section Page

Management Summary 3

Background 4

Outcome of the 19th June workshop 5

Student Survey: Data Analysis 7

Recommendations 11

Appendices

Appendix 1: Proposed Student Evaluation Questions 12

Appendix 2: Invitation to Students 14

Appendix 3: Student Survey Data 15

Acknowledgements 17


Management Summary

Background

The Practice Placement Quality Assurance (PPQA) website www.healthcareplacements.co.uk was conceived initially in 2004 as an evaluation tool for healthcare students to complete an online evaluation following each practice placement. The website now includes a lot more quality assurance functionality, including placement profile and audit. Requests to review the student evaluations questions have been expressed this year by stakeholders across the region. A workshop was arranged for the morning of 19

th June to review the student

evaluation questions, and responses, and propose alternatives. The outcome of the workshop and the subsequent consultation activities, including a student online survey are included in this report. At the workshop the purpose and format of the questionnaire was agreed. Linking student names to their comments was also confirmed. The proposed questions were refined via email and consultation during June and July and an online survey was developed to obtain student feedback on the proposed evaluation questions (see Appendix 1). An invitation to the students was issued in August for return by 30

th September (see Appendix 2).

Student Survey: Data Analysis 294 students complete the online survey. During the survey they were asked two questions about each of the proposed questions in Appendix 1 as follows:

A. Do you think Question X is clear and unambiguous? B. Do you think a Yes, No (and in some case N/A (see Appendix 1)) response is appropriate for

question X? C.

The responses were rated using a red, amber, green scale (see Appendix 3). The comments for the 11 proposed questions with an amber or red rating for question A are summarised in the report and included points about multiple questions and suggestions for alternatives. Comments specifically related to the actual responses (Y,N,N/A) in Question B were not significant. A number of students requested the ability to leave a comment, which is already included in the software, and others referred to the actual question (similar responses to Question A). The students were also asked a final question to establish if there was anything not covered by the 25 questions.

Recommendations

The recommendations are as follows:

The purpose of the questionnaire should be advertised to students by HEI staff and also on the PPQA website. Timescales to be decided by the Practice Placement Quality Working Group (PPQWG).

The proposed student questions with a green rating in Appendix 3 for Question A are agreed by the PPQWG as valid and will be included on the PPQA website when the student evaluation area is updated.

The proposed student questions which are listed in “Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A” should be reviewed to take into account the comments received by the students on each question and also the final question comments. A half day workshop is suggested, involving the individuals who contributed to the 19

th June workshop.

Contents

http://www.healthcareplacements.co.uk/


Background

The Practice Placement Quality Assurance (PPQA) website www.healthcareplacements.co.uk was conceived initially in 2004 as an evaluation tool for healthcare students to complete an online evaluation following each practice placement. The website now includes the following functionality:

Student and Mentor Evaluations

Mentor/Practice Educator Register

Practice Placement Profile and Audit

Student Capacity information

Documentation area

The student evaluation questions were reviewed by the Regional Educational Audit Tool (REAT) sub group in 2010 and the mentor / practice educator (PE) evaluation questions were also developed at this point. The software was updated in January 2011 to include the mentor evaluation functionality and also the ability for students and mentors to leave comments about each question they answered. It was compulsory to leave a comment if the response was negative and the names were linked to the comments. The actual questions for the students changed at the end of 2010/11 academic year as this was felt easier to manage.

Requests to review each of the student evaluation questions have been expressed this year by members of the Practice Placement Quality Working Group (PPQWG), Higher Education Institution (HEI) staff, Practice Learning Facilitators and via the online surveys undertaken as part of the PPQA review (reported in June 2013). The issue of comments being attributable to individual students was also being raised.

A workshop was arranged for the morning of 19th June to review student anonymity, student

evaluation questions, and responses, and propose alternatives. The outcome of the workshop, including the purpose of the student evaluations, and the subsequent consultation activities, including a student online survey, are contained within this report.

Contents

http://www.healthcareplacements.co.uk/


Outcome of the 19th June workshop

There were 22 delegates at the workshop. The issue of anonymity was discussed prior to reviewing the questions and it was agreed that the student names should be linked to the student comments because of the following:

The responses have never been completely anonymous and it is therefore unfair on students to give the impression they are anonymous

Good practice indicates follow up to the responses with the students, lecturers or mentors; particularly if there are professional conduct issues

Students should be encouraged to take a professional responsibility for their comments

It is important to promote a culture of transparency with the students, particularly in light of the Francis Report.

Purpose of the student questionnaire The purpose of the questionnaire was agreed at the commencement of the workshop (see below). It is recommended this should be advertised to the students on the PPQA website in the area where they complete the questionnaire and by the Higher Education Institutions when preparing the students for placement.

“Feedback on your learning experience will contribute to the on-going evaluation of education within

the placement setting and support from your education provider.

You have a responsibility as a healthcare practitioner of the future to participate in the quality

assurance processes in an open and transparent manner.”

Sections and Responses

Sections It was agreed that although the students should not see the section titles the questions need to follow a logical sequence from start to finish of the placement. The following sections were agreed:

Preparation for practice experience.

Orientation

Learning Opportunities/Resources

Supervision/Support

Feedback/Assessment

Recommendations/Concerns

Responses It was agreed that the responses should be changed from the current “strongly agree” through to “strongly disagree”. The responses should be as follows:

Yes/No, and not applicable available for identified questions (see Appendix 1)

Comment compulsory for “No”.

No check box required re terms and conditions

Additional comment box required at the end of the questionnaire

“There is a responsibility on healthcare providers to deliver high quality

education and training, not just for their students, but for all their staff in order to

ensure high quality and safe patient care.” A mandate from the government to

Health Education England (2013)


Consultation

The PPQA Project Manager collated the questions developed during the workshop by the 5 work groups under the section headings above, avoiding duplication and ensuring the questions were generic and suitable for any placement or placement period. The results were circulated to the attendees of the workshop for comment and onward circulation as appropriate. A draft student online survey, using the Bristol Online Surveys Tool (BOS), was also circulated at the same time for comment and amendment. The feedback received from the workshop group, and their colleagues, was then used to amend the questions. All comments and changes were circulated again to the workshop group with the rationale for changes and omissions. As agreed at the workshop the proposed questions (see Appendix 1) were then included in the online survey to obtain the student feedback.

Online Survey to the Students An invitation (see Appendix 2) to complete the evaluation by 30

th September was issued on the 6

th

August to the PPQWG HEI representatives who then contacted the students. 294 students had completed the survey by the closing date. The results of the survey responses are included in the section Student Feedback: Data Analysis below.

Contents


Student Survey: Data Analysis

There were 294 responses to the online survey including students from 7 HEIs, plus the Yorkshire Ambulance Service, and 11 professions were represented (See Questions 1 and 2 below).

Question 1. Please select your education provider from the list below.*

University of Bradford:

25.2% 74

University of Huddersfield:

21.1% 62

Leeds Metropolitan University:

20.7% 61

University of Leeds:

19.4% 57

The Open University:

6.5% 19

York St John University:

4.8% 14

Yorkshire Ambulance Service:

2.4% 7

*No Sheffield Hallam University, University of Sheffield and University of York students completed the

survey.

Question 2. Please select your profession from the list below.**

Nursing:

55.1% 162

Dietetics:

11.2% 33

Podiatry:

6.8% 20

Midwifery:

6.5% 19

Physiotherapy:

5.8% 17

Occupational Therapy:

5.4% 16

Radiography:

3.7% 11

Paramedics:

2.4% 7

Speech and Language Therapy:

2.0% 6

Clinical Language Sciences:

1.0% 3

Speech and Language Therapy:

2.0% 6

**No audiology, operating department practitioners or radiotherapy students completed the survey.

The students were asked if the abbreviation of N/A for “Not Applicable” is universally understood. 263 (90%) felt it was whereas 31(10%) said no. Two comments really sum up why we should not use an abbreviation without explaining it first.

“All abbreviations

should be avoided

as in practice”

“Might it not be a good idea to

write Not Applicable (N/A)

once at the beginning to

facilitate both international

students and understanding

and use of the abbreviation.”


On subsequent pages the students were presented with each of the proposed 25 questions and asked the following two questions:

D. Do you think Question X is clear and unambiguous? E. Do you think a Yes, No (and in some case N/A (see Appendix 1)) response is appropriate for

question X? There were options of either a Yes/No response to both questions and if the response was “No” a comment was compulsory)

It had been suggested to the students in the introductory text that they should consider their most recent placement when responding to the survey. Although it was a minority, there were a few comments where the students answered the actual question under discussion rather than the two questions A and B above. This was clearly a fault of the introductory text and numbers have been identified in the table below. The data for the responses are included as Appendix 3 with the responses categorised using a Red, Amber, Green (RAG) rating, for example the red rating indicates over 10% of the students answered the question negatively (Amber = 5-10%), Green = <5%). The results can be summarised as follows:

Question A 3 8 14

Question B 6 16 3

Question A

The comments for Question A resulting in a red or amber rating are summarised in the following table:

Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A

No. Proposed Student Evaluation Question

Qu.A Comments summarised

N

3. The placement profile was informative.

44 15%

Don’t know what a profile is / don’t know where to find it /not sure what is being asked (17)

What is meant by informative? (4)

Comments re profiles not being helpful / limited information / out of date / incorrect / relevant reading not available / profile not available for placement (11)

Suggestions for alternatives (9)

Other (3)

23. I had concerns about patient care on my placement but these were dealt with appropriately

34 12%

Two questions in one i.e. concerns (y/n), dealt with appropriately (y/n) (8)

“No concerns” was the student response (this could be related to their placement or a suggestion for a change to the wording). (9)

Student referred to their actual placement (5)

“N/A” was the student response(4)

What does the question mean? / needs rewriting (6)

First year students may not have confidence to answer, responses should be anonymous (2)

24. I had concerns about my educational experience on my placement but these were dealt with appropriately

32 11%

Very similar responses to Question 23 above

7. There was sufficient access to internet IT facilities

23 8%

Access for student research or patient care? (3)


Difficulties accessing internet whilst on placement (13)

5. I had somewhere safe to store my belongings.

22 7%




Question irrelevant (1)

4. I had a planned, timely and comprehensive induction which orientated me to my environment and relevant policies and procedures, including health and safety information

17 6%

Too many areas covered in the question (10)

Induction on placement or by the university? (2)


N/A, member of the trust, other (4)

1. I received information and support from my education provider, which was appropriate for the stage of my programme, so that I was prepared for my placement.

19 6%

Three questions in one (6)

Badly written, muddled, doesn’t make sense had to read it twice (6)

Should be “..this stage…” (1)

Please define “support” is it learning or practical support (1)


Student referred to their actual placement (2) 12. I received timely support from

my course/programme team whilst on placement when I needed it

16 5%


Ambiguity / vague (7)


13. The learning needs appropriate to the stage of my programme were discussed with my mentor(s) / practice educator(s) early in the placement.

16 5%


Needs rewording / clumsy (10)


14. My named mentor(s) / practice educator(s) and the wider team were able to facilitate my development and recognised the importance of my learning needs

14 5%

More than one question (4)

Needs rewording (7)


Other (1)

20. Where applicable, the placement encouraged service users/carers to contribute to the assessment of my performance in practice.

14 5%

Question confusing (2)

Suggested terminology 'provide performance feedback' (1)

N/A, other (11)

Question B

The comments for Question B were reviewed and the highest number of responses was requesting the ability to leave a qualitative comment. This functionality is currently available on the PPQA software. For example, for Question 1 which received the greatest number of negative responses (47, 16%) there were 24 students asking to leave a comment, 5 asking for a range of responses, e.g. a Likert scale and 18 commenting further on the actual question (similar to Question A). The number of students actually requesting a change to the Y/N/NA responses was very low.

Final Question As a final question the students were asked the following:

“Is there anything we have not covered in the 25 questions? Please let us know of any aspect of your placements which you feel you would not have had the opportunity to comment on.”

49 of the students left comments. 11 students requested a final optional comment box after the evaluations questions had been completed, which was also suggested at the 19

th June workshop (see


Consultation: Responses section above). 11 students wanted to be able to comment on the standard of the mentorship received i.e. in terms of time and quality and a review of the comments alongside the proposed questions 14 to 16 is therefore indicated (see Appendix 1). Comments and suggested questions included the following:

The following comment was left by one student praising the new questions but also expressing their

view regarding anonymity.

“Do you think your skills

and confidence in practice

have grown due to this

practice placement?”

“Do you feel more confident

in your abilities to practice

after completing the

placement?”

“….Support offered around

accommodation and travel

…”

“….relevance of coursework

to practice…”

“Were mentors afforded

sufficient time (by

management) in their

schedule to facilitate, support

and reflect on learning and

practice” “There should include a

question about having an

allocated mentor and being

able to work with them

often enough.” “I believe that what other

students have written should

be available to the students

that are next visiting to give

them ideas of how to get the

best out of their placement

and….”

“Were you provided with the

appropriate workwear/PPE

in order for you to carry out

your work in the placement

area.”

“Very good selection of questions - much

better improved than previous questions,

they deal with more relevant and realistic

situations. The only improvement I would

make is to make the questionnaire

anonymous as this can cause students to

feel scared to voice their true opinions on

all matters for fear of being reprimanded

by mentors, tutors etc.”

Contents


Recommendations

The recommendations are as follows:

The purpose of the questionnaire should be advertised to students by HEI staff and also on the PPQA website. Timescales to be decided by the PPQWG.

The proposed student questions with a green rating in Appendix 3 for Question A are agreed by the PPQWG as valid and will be included on the PPQA website when the student evaluation area is updated.

The proposed student questions which are listed above in “Table1: Proposed Student Evaluation Questions with a Red or Amber rating for Question A” should be reviewed to take into account the comments received by the students on each question and also the final question comments. A half day workshop is suggested, involving the individuals who contributed to the 19

th June workshop.

Contents


Appendix 1: Proposed Student Evaluation Questions

Preparation for Practice Experience Response Y/N/NA


Y/N

2. I was informed of my placement allocation with sufficient time for me to make contact with the placement.

Y/N/NA

3. The placement profile was informative. Y/N/NA

Orientation


Y/N

5. I had somewhere safe to store my belongings. Y/N/NA

Learning Opportunities/Resources

6. The placement provided me with access to resources which helped me with my learning

Y/N

7. There was sufficient access to internet IT facilities Y/N

8. The learning opportunities available in this placement enabled me to achieve my learning outcomes

Y/N

9. I was encouraged to take part in practice activities appropriate to my level of experience.

Y/N

10. I had sufficient opportunities to receive feedback after being observed in practice.

Y/N

11. The placement provided me with the experience of working inter-professionally

Y/N/NA

Supervision/Support

12. I received timely support from my course/programme team whilst on placement when I needed it

Y/N/NA


Y/N


Y/N

15. With the support of my mentor(s) / practice educator(s) I was able to work towards my placement learning outcomes

Y/N

16. Staff in the practice area were friendly, helpful, included me as part of the team and had time to answer my questions

Y/N

17. I understood the process for raising concerns within the placement Y/N

Feedback/Assessment

18. My mentor(s) / practice educator(s) understood and followed the student assessment process.

Y/N

19. My development in practice was informed by on-going constructive feedback Y/N


Y/N/NA

21. My final assessment was fairly assessed and focused on feedback about my performance across the whole placement.

Y/N

Recommendations/Concerns

22. I would recommend the placement setting to friends and family if they needed similar care or treatment.

Y/N



Y/N/NA


Y/N/NA

25. I would recommend this placement as a valuable learning experience Y/N

Contents


Appendix 2: Invitation to students

Contents


Appendix 3: Student Survey Data

A. Do you think Question X is clear and unambiguous?

B. Do you think a Yes, No (and in some case N/A see Appendix 1)) response is appropriate for

question X?

Key

Over 10% of the students answered “No” to the question

5 to 10 % of the students answered “No” to the question

Less than 5 % of the students answered “No” to the question

No. Proposed Student Evaluation Question Question A. Question B.

Y N Y N

3. The placement profile was informative. 250 85%

44 15%

254 86%

40 14%


260 88%

34 12%

249 85%

45 15%


262 89%

32 11%

255 87%

39 13%

7. There was sufficient access to internet IT facilities 271 92%

23 8%

273 93%

21 7%

5. I had somewhere safe to store my belongings. 272 93%

22 7%

275 94%

19 6%


277 94%

17 6%

262 89%

32 11%


275 94%

19 6%

247 84%

47 16%

12. I received timely support from my course/programme team whilst on placement when I needed it

278 95%

16 5%

273 93%

21 7%


278 95%

16 5%

287 98%

7 2%


280 95%

14 5%

275 94%

19 6%


280 95%

14 5%

278 95%

16 5%

9. I was encouraged to take part in practice activities appropriate to my level of experience.

283 96%

11 4%

265 90%

29 10%

6. The placement provided me with access to resources which helped me with my learning

283 96%

11 4%

269 92%

25 8%

16. Staff in the practice area were friendly, helpful, included me as part of the team and had time to answer my questions

283 96%

11 4%

268 91%

26 9%

22. I would recommend the placement setting to friends and family if they needed similar care or treatment.

283 96%

11 4%

275 94%

19 6%

8. The learning opportunities available in this placement enabled me to achieve my learning outcomes

284 97%

10 3%

250 85%

44 15%

19. My development in practice was informed by on-going constructive feedback

285 97%

9 3%

284 97%

10 3%

11. The placement provided me with the experience of working inter- 286 8 273 21


professionally 97% 3% 93% 7% 18. My mentor(s) / practice educator(s) understood and followed the

student assessment process. 287 98%

7 2%

276 94%

18 6%

21. My final assessment was fairly assessed and focused on feedback about my performance across the whole placement.

287 98%

7 2%

272 93%

22 7%

10. I had sufficient opportunities to receive feedback after being observed in practice.

288 98%

6 2%

272 93%

22 7%

17. I understood the process for raising concerns within the placement

288 98%

6 2%

290 99%

4 1%

15. With the support of my mentor(s) / practice educator(s) I was able to work towards my placement learning outcomes

289 98%

5 2%

280 95%

14 5%

25. I would recommend this placement as a valuable learning experience

289 98%

5 2%

271 92%

23 8%

2. I was informed of my placement allocation with sufficient time for me to make contact with the placement.

290 99%

4 1%

277 94%

17 6%

Contents


Acknowledgements

Thank you to all partners in the region who contributed to the consultation undertaken with the

outcome of the 19th June workshop.

Thank you to all the students who completed the online survey and the members of the PPQWG who

liaised with them.

Thank you to the colleagues below who were involved in the June 19th workshop.

Name Organisation

Anne Brown PLF Bradford and Airedale

Catherine Coates Leeds Metropolitan University

Steve Himsworth University of Hull

Corinna Klaentschi York St John University

Migaela Scorah Leeds Metropolitan University

Fiona McInnes University of York

Gillian Swan Leeds Institute of Medical Education

Heidi Cheung Sheffield Hallam University

Helen Green University of Leeds

Isabel Brislen Sheffield Health and Social Care NHS Foundation Trust

Jenny Shaw PLF Leeds

Kath Bowles University of Leeds

Katie Cook PLF North Humber

Sharon Parker PLF Bradford and Airedale

Linda Jackson Sheffield Teaching Hospital NHS Foundation Trust

Sally Underwood University of Sheffield

Niall Dew University of Huddersfield

Nicky Hughes The Open University

Samantha Smithson Yorkshire Ambulance Service

Sherree Hamburg PLF Bradford and Airedale

Valerie Hales PLF South Yorkshire

Janice High University of Bradford

Contents

Documents

Practice Placement Quality Assurance (PPQA) website ... Student Evaluation... · Practice Placement Quality Assurance (PPQA) website: Student Evaluation Questions A review update