25
Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena Harper, Linda Li, Sue Prentice

Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Embed Size (px)

Citation preview

Page 1: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Counting what counts in individual consultations:

developing and applying a set of evaluation instruments

Lynn Berry, Garry Collins, Peter Copeman

Rowena Harper, Linda Li, Sue Prentice

Page 2: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Background of our study• Individual Consultations (ICs) are still core ASC

business

• Increasing demand for (ICs) at UC but decreasing capacity to meet this demand

• Need to ensure quality & effectiveness of ICs

• Inspiration from Stevenson & Kokkinn’s (2009) AALL paper on evaluating ICs

Page 3: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Purpose

• Develop, trial and analyse the effectiveness of a set of instruments for evaluating individual consultations

• Secondary purpose – to enhance our teaching and student learning in ICs

Page 4: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The research design

Peer observations

Student evaluations

Reflective practice

Page 5: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Methodology

• Instruments: • Peer-observation (PO) tool• Self-reflection (SR) tool• Student questionnaire (Q-tool)

• Participants: • 7 ASC staff; • Students

• Time: 4 week period - Semester 2, 2011

Page 6: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The research process

Pre-PO

PO

IC

Post PO

SR

StudentEval

Anonymous student feedback

PO checklist de-identified & stripped of comments

Responses to request for contributions about SR process (non-identifiable)

Shared data

Page 7: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The peer observations: a collaborative model

A structured approach

Establishing a set of

principles and criteria

inclusive, staff active involvement,non- judgmental, a collaborative process

a collaboratively developed peer-observation tool, scheduled pre- and post-observation discussions integrated with self-reflection

Page 8: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Peer-observations: positive outcomes

• Enthusiastic staff participation

• Sharing of approaches to good practice and strategies

• Strengthening the collegiality of the team

• Benefits of learning from each other

Page 9: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

An example of staff comment

Thank you for the opportunity to observe your consultation. In spite of my observer status, I was actively involved in the learning and teaching process as I observed its development as the consultation unfolded. The consultation made an important contribution to my learning and professional development by stimulating reflection on my own teaching.

Page 10: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Peer-observations: challenges and constraints

• Some initial staff reservation

• The limiting effect of the peer-observation tool

• The constraint of time

Page 11: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The self-reflection process: reflective dialogues with self and others

• Scheduled self-reflection time

• A guided approach with a self-reflection tool aligned with the peer observation tool

• Linked to the peer observations through post-observation discussions with colleagues

• Team debrief session to reflect on the whole process

Page 12: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Self-reflections: positive outcomes

• The self-reflection tool served as a reminder or structural tool

• Self-reflection was stimulated by observing another and through peer-exchanges.

• Reflection as a process with ongoing informal discussions

Page 13: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

An example of staff self-reflection

One thing I would remind myself in future sessions is not to jump to conclusion too quickly… It was only after listening to his explanations that I realised I made an assumption too quickly without getting the real picture! This reminds me of the importance of listening to the student, not just at the beginning of the session, but throughout the session, and the importance of holding back my own judgment and letting the student make his/her own decision.

Page 14: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Self-reflections: challenges and constraints

• Some staff criticism of the self-reflection checklist

• Need to modify the tool

• ‘Self-reflection can’t take place in a vacuum’

Page 15: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

An example of staff self-reflection

By and large, my reflection took place while I was observing other staff. Upon seeing a particular approach, my instinct was to consider how I would approach the situation if I was the adviser. I learned a lot of useful strategies this way…I’m not sure I can apply ‘reflection’ to my own teaching in a sort of vacuum. When I try to do this, I am only evaluating how I fell short of my own tried and tested approaches. I feel it’s difficult to improve this way. I feel I need to see alternative approaches in practice so I have points for comparison.

Page 16: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The student questionnaire: focus on student learning

• An improved version: more than quality assurance

• Encourage the students to reflect more explicitly on their learning

Page 17: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The student questionnaire: focus on student learning

Anticipatedlearning IC Actual learning

Evaluation of learning

EvaluationOf

teaching

Page 18: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The student questionnaire: positive outcomes

• A constructivist view of learning

• Student expectation and student learning

• Focus on most needed aspects of learning

Page 19: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Student comments• I am very happy with consultation. It gave me a lot of

confidence.

• Thank you for giving me the chance to correct my assignment.

• I am happy with the ASC service and it helped me to improve my knowledge and my result.

• Thank you for all your help it has greatly advanced my grades and understanding!!

Page 20: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

The student questionnaire:challenges and constraints

• Focus on what rather than how and how well

• Unlinked to peer-observations and self-reflections

Page 21: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Have we achieved our goals?

Peer observations

Student evaluations

Reflective practice

Page 22: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Have we achieved our goals?• We successfully developed, trialled and analysed the

effectiveness of Q-tool, PO & SR

• The Q-tool contributed to greater student awareness of their own learning, but did not add significantly to feedback on our teaching

• PO and SR contributed to better shared understanding of ICs and improved IC teaching practices

Page 23: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

What have we learnt from this project?

• Better , shared teaching strategies for ICs

• We learnt more from observation than being observed or reflection

• Necessity to link PO and SR to improve IC teaching practices

• Using evaluative instruments such as PO and SR requires training and experience

Page 24: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

What next?

• Plan for next round of PO & SR:

• Refine the PO, SR & Q-tool instruments

• Pilot a focus group of students to elicit more in-depth comments/suggestions on ASC’s ICs

• Keep an on-going record of effective IC teaching strategies

Page 25: Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena

Acknowledgements

We would like to thank our colleagues Emmaline Lear, Sharon Gibson, Jo Caffery, Ros Byrne and all the participating students for their contribution to this project.