Gordon SuddabyScholarship of Learning and Teaching

National Centre for Teaching and Learning

Researching your teaching

Rationale Academics typically find themselves questioning their own ideas

and searching out explanations for complex issues. This applies as much to teaching as to research. This module will look at why academic staff might investigate their own teaching practice, determine the possible benefits associated with doing so, consider how they might do so, how they might use the findings to further develop themselves as scholarly and research-led academic teachers, and to advance their careers.

Why should we be interested in researching our teaching?Brainstorm:1.2.3....

Universities are generally interested in teaching... to ensure quality, effectiveness, and accountability to provide recognition and reward to bring about improvement. ... ...The expression of this interest usually involves some form of

evaluation.

Evaluation of teachingThere are two overriding purposes for evaluation;

To measure and accord value to teaching for accountability

To improve the quality of teaching (and learning)

Thinking about evaluating our teaching: Poor evaluation, whether of students or of staff, renders an

unfair judgement and fails to reveal shortcomings in performance. Good evaluation on the other hand provides decision makers with the information necessary for informed choices and teachers with useful feedback for improvement.

Centre, 1993, p.1

Vocabulary Formative: Provides feedback which is used during the

[teaching] process for improvement. It is continuous, diagnostic, remedial, and low stakes.

Summative: ...used after [the teaching] process has been completed. Grading and accountability are major outcomes. It is terminal, finite, descriptive and high stakes.

After Scriven, 1967.

Activity In groups of 3, identify the university processes involving decision-making

about you that requires evaluative information about your teaching

...

What aspects of your teaching provides that information?

What (if any) further information could/should be provided?

How and by whom?

How valid and reliable do you think the information is?

Lets be a little pragmatic... One of the most important and tangible reasons for

evaluating teaching is for promotion so this is a good place to start.

We are effectively evaluating our teaching and to a degree, making that evaluation ‘public’ (at least to a selected audience)

So what information is expected in a promotion application?

5.2.1 Criteria Specific to Teaching Good teaching enhances student development in the directions specified

for particular programmes. Criteria for promotion will allow for different teaching philosophies and contexts.

Indicators of teaching effectiveness include:

(a) the utilisation of current research-based knowledge;

(b) planning and teaching strategies that present knowledge clearly and engage challenge and stimulate students to become independent learners;

(c) evaluation of one’s own teaching activities;

(d) effective communication with students;

(e) appropriate assessment procedures;

(f) evidence of self-evaluation and continuing professional development;

(g) recognition of teaching quality by peers and students;

(h) Teaching Portfolio (Refer Training & Development Unit for assistance).

(i) effective education and teaching of post-graduate students.

5.2.2 Evidence specific to teaching Applicants (except those in research-only positions) should present evidence of teaching

effectiveness, which includes both quantitative and qualitative data from a variety of sources. This should include:

(a) quantitative data on teaching loads, including data on paper numbers, contact hours, on-campus courses, student numbers, supervision loads; number of successful postgraduate student completions.

(b) a brief outline and justification of the applicant's approach to teaching;

(c) examples of paper design (including learning outcomes, curriculum outline/s and teaching and assessment strategies) highlighting aspects that reflect the applicant's approach to teaching;

(d) an overview of student feedback eg. SECAT, etc., including (where appropriate) feedback from the applicant's research students, all with summary comments from the applicant;

(e) a description of steps taken by the applicant to further develop her/his teaching effectiveness;

(f) details of prizes awarded for teaching or other forms of recognition of excellence;

(g) assessment by the Small Group Instructional Diagnosis [SGID] (contact Training & Development Unit).

(h) evidence of incorporation of relevant recent research into paper content.

(i) any other evidence demonstrating teaching effectiveness and commitment;

Something to think about...In what may as well be starkly labelled smug

satisfaction, an amazing 94% [of college instructors] rate themselves above average teachers and 68% rate themselves in the upper quartile of teaching performers.

K. Patricia Cross

Some thoughts about evaluating teaching...When evaluations are used for promotion and tenure, the accepted rule

is that no single evaluation should be considered adequate for decision making

Theall and Franklin, p.94, 1991Use multiple sources of data if you are serious about improving

teaching Cashin, p.93, 1992It is wise to be circumspect about using student ratings to make

judgments on teaching quality and to recognise their complications as well as their virtues.

Ramsden, p.229, 1992

Activity Rule a column down the middle of a piece of paper On the left hand side list things that you believe you have

done well when teaching

On the right hand side for each indicator, map the evidence you have that supports the point

What evidence do you use to know things haven’t gone well?

So the question for you as a teacher is... How do you know you are teaching well?

Or

How well you are teaching?

And

How might you improve?

Think about...A colleague suggests that he/she is a good teacher

What evidence would you need to verify that statement?(NB the evidence needs to be valid, legitimate, verifiable, reliable and ethical)

...

How might such evidence be collected? ...

Researching teaching Apart from promotion, what other reasons might we have for

researching our teaching? ... ... ...

Researching teachingPurposes; For improvement, i.e. Quality enhancement; development and

improvement of learning, teaching etc

For accountability i.e. Quality assurance; competence, assurance for stakeholders etc

For publication i.e. Development of pedagogical research

For portfolios i.e. Promotion, recognition, reward, reflection

For integration i.e. Alignment of curricula in specific disciplines

For reflection i.e. Development of critical reflective practice

Scholarship i.e. Making learning transparent and explicit

Definition: Scholarship of Teaching and Learning (SoTL) ‘The scholarship of teaching and learning is the systematic

reflection on teaching made public”. This involves supporting the development of teaching and learning which fosters ‘long-lasting learning for all students; enhances the practice and profession of teaching, and; brings to faculty members’ work as teachers the recognition and reward afforded to other forms of scholarly work”. In doing so, SoTL “renders teaching public, subject to critical evaluation, and usable by others in both the scholarly and general community”.

http://www.carnegiefoundation.org/scholarship-teaching-learning

Boyer’s Scholarships The Scholarship of Discovery.

The Scholarship of Integration.

The Scholarship of Application.

The Scholarship of Teaching.

Boyer, 1990.

Boyer’s scholarships The Scholarship of Discovery. This comes closest to what is usually meant by ‘basic research’. It is

scholarship that involves “commitment to knowledge for its own sake, to freedom of inquiry and to following, in a disciplined fashion, an investigation wherever it may lead” (Boyer, 1990, p. 17).

The Scholarship of Integration. This is “serious disciplined work that seeks to bring new insight to bear on original work” (p. 19). Giving meaning to isolated facts, putting such facts into perspective, fitting research into larger intellectual patterns, making connections across the disciplines, placing the specialties in larger contexts are all activities that Boyer associates with this mode of scholarship.

The Scholarship of Application. This work, which also requires rigor and accountability, constitutes service to others which calls for the application of special fields of knowledge and associated skills. In the course of this service, new understandings may also arise as “theory and practice vitally interact and one renews the other” (p. 23). Note: Boyer (1996) subsequently renames this type of scholarship, the Scholarship of Engagement.

The Scholarship of Teaching. Boyer viewed teaching as a scholarly enterprise because the “work of the professor becomes consequential only as it is understood by others” (p. 23) and teaching serves to both educate and entice future scholars. “It is inspired teaching that keeps the flame of scholarship alive.” (p. 24). He also observed that “pedagogical procedures must be carefully planned, continuously examined, and relate directly to the subject taught” and “good teaching means that faculty, as scholars, are also learning” (about teaching and learning).

Scholarship of Teaching is about:A. knowing the literature on teaching by collecting and reading

that literatureB. improving teaching by collecting and reading the literature on

teachingC. improving student learning by investigating the learning of

one’s own students and one’s own teachingD. improving one’s own students’ learning by knowing and

relating the literature on teaching and learning to discipline specific literature and knowledge

E. improving student learning within the discipline generally by collecting and communicating results of one’s own work on teaching and learning within the discipline.

(Trigwell et al, 2000, pg 159).

A scholarly approach to teaching involves; Clear Goals

Adequate preparation

Appropriate methods

Significant results

Effective presentation

Reflective Critique

Glassick, Huber and Maeroff (1997)

What does the literature say about good teachers? Good teachers are;Knowledgeable about their subjectAdopt an organised and systematic approach to their

teachingAre enthusiastic and interestingRespect their studentsHave high expectations of their students performance

From: Prebble, Hargraves, Leach, Naidoo, Suddaby and Zepke (2005)

Clarifying our approach:Research

– enhances our theoretical and/or conceptual understanding of teaching and learning

– is firmly situated in its relevant literature and makes a substantial contribution to that literature and or field.

Investigations and Evaluations – enhances our understanding of a local problem or issue, providing

recommendations for policy and / or action – is firmly situated in its relevant literature

Literature Reviews – collection and analysis of literature aimed at describing the various ways in

which the object of the review is thought about and recommendations for practice

Scholarship of Teaching and Learning – evidence based critical reflection on practice aimed at improving practice

Personal circumstances, interests and capabilities

Is there some aspect of teaching and learning that I am really interested in and want to investigate further?

Do I think I can change and improve my own and others’ teaching through SoTL? Am I confident about my research capabilities, including ‘doing’ SoTL? Am I confident about my teaching but ready to investigate it? Given other demands in my academic and personal life, is there space for engaging in

SoTL?

Pull Factors Do other people expect me to be able to offer evidence to support my teaching

decisions or to confirm I am teaching effectively?

Push Factors Would this be another way of fulfilling PBRF expectations? Would this be another avenue for meeting postgraduate research expectations?

(Haigh, Gossman & Jiao, 2010)

Obstacles Personal pre-conditions (as above) are not present

Would what I am interested in investigating be of interest to others?

Would SoTL be given the same status as my other disciplinary research?

Would my colleagues/department be supportive of my engaging in SoTL?

Would I be able to do SoTL when there is limited funding?

Would I be able to get some publications without too much delay?

Would I be able to find relevant, useful, good quality literature on my interests?

Would I be able to develop an acceptable proposal?

Would I be able to handle the ethics considerations and make my way through the ethics approval process?

http://akoaotearoa.ac.nz/sotl

But there are optionsWe can change the focus from; “what is the quality of your teaching” To; “how can we use evidence gathered from student feedback

and other forms of information to improve teaching?” http://akoaotearoa.ac.nz/content/guide-collecting-

evidence-learner-benefitBerk (2005) has listed 12 ways as a starter to which I will add

two more; Small Group Instructional Diagnosis and Classroom Assessment Techniques.

12 sources of evidence(Berk, 2005)1. Small Group Instructional Diagnosis2. Classroom Assessment Techniques3. Student ratings4. Peer ratings5. Self-evaluation6. Videos7. Student interviews8. Exit and Alumni ratings9. Employer ratings10. Administrator ratings11. Teaching scholarship12. Teaching awards13. Learning outcome measures14. Teaching portfolios

Small Group Instructional Diagnosis Developed by Clark in 1979 Small Group Instructional Diagnosis

(SGID) is a form of classroom research which focuses on student learning. It is a formative process enhancing learning and is a safe, non-threatening and transparent mechanism liked by students and staff. It is orally-based, involving concensus and involves little time with a quick turn-around.

http://tdu.massey.ac.nz/pdfs/sgid.pdf

http://tdu.massey.ac.nz/forms/sgidrequest.html

Teaching Awards As a source of evidence of teaching effectiveness, at best, teaching

awards provide worthwhile information only on the nominees, and, at worst, they supply inaccurate and unreliable feedback on questionable nominees who may have appeared on ‘Law and Order’. The merits of teaching awards should be evaluated in the context of an institution’s network of incentives and rewards for teaching.

http://www.massey.ac.nz/massey/about-massey/university-management/avc-academic/awards/teaching-awards-framework-overview.cfm

Student ratings Student ratings is a necessary source of evidence of teaching

effectiveness for both formative and summative decisions, but not a sufficient source for the latter. Considering all of the polemics over its value, it is still an essential component of any faculty evaluation system.

Using the Massey Online Survey Tool (MOST)

http://www.massey.ac.nz/massey/staffroom/teaching/centre-academic-development-elearning/teaching-evaluation--most/en/teaching-evaluation---most.cfm

Fiona Murray will talk to us about it.

Peer ratings Peer ratings of teaching performance and materials is the most

complementary source of evidence to student ratings. It covers those aspects of teaching that students are not in a position to evaluate. Student and peer ratings, viewed together, furnish a very comprehensive picture of teaching effectiveness for teaching improvement. Peer ratings should not be used for personnel decisions.

Using Massey’s Peer Review Processhttp://www.massey.ac.nz/massey/about-massey/university-management/avc-academic/academic-quality/peer-review-of-teaching/peer-review-of-teaching_home.cfm

Teaching scholarship Teaching scholarship is an important source of evidence to

supplement the three major direct observation sources. It can easily discriminate the “teacher scholar” and very creative faculty from all others for summative decisions.

Classroom Assessment Activities (including fast feedback tools)Classroom assessment techniques drawn from Angelo and Cross (1996).

These tools are Learner Centred, Teacher directed, Mutually beneficial, Formative, Context-specific and Ongoing.

http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/assess-1.htm

http://www.ntlf.com/html/lib/bib/assess.htm

Self Evaluation Self-evaluation is an important source of evidence to consider in

formative and summative decisions. Faculty input on their own teaching completes the triangulation of the three direct observation sources of teaching performance: students, peers, and self.

Videos If faculty are really committed to improving their teaching, a video is

one of the best sources of evidence for formative decisions, interpreted either alone or, preferably, with peer input. If the video is used in confidence for this purpose, faculty should decide whether it should be included in their self evaluation or portfolio as a “work sample” for summative decisions.

Student interviews The quality control circle is an excellent technique to provide

constant student feedback for teaching improvement. The group interview as an independent evaluation can be very informative to supplement student ratings. Exit interviews may be impractical to conduct or redundant with exit ratings, described in the next section.

Exit and Alumni interviews Although exit and alumni ratings are similar to original student

ratings on the same scale, different scale items about the quality of teaching, courses, curriculum admissions, and other topics can provide new information. Alumni ratings should be considered as another important source of evidence on teaching effectiveness.

Employer ratings Employer ratings provides an indirect source of evidence for program

evaluation decisions about teaching effectiveness and attainment of program outcomes, especially for professional schools. Job performance data may be linked to individual teaching performance, but on a very limited basis.

Administrator ratings Administrator ratings is typically based on secondary sources, not

direct observation of teaching or any other areas of performance. This source furnishes a perspective different from all other sources on merit pay and promotion decisions.

Learning outcome measures Learning outcome measures should be employed with extreme caution

as a source of evidence for faculty evaluation. It’s safer to use in conjunction with the direct data sources described previously for program improvement.

Teaching portfolio As a collection of many of the previous sources and them some, the

teaching portfolio should be reserved primarily for summative decisions to present a comprehensive picture of teaching effectiveness to complement the list of research publications.

CAT examples The Minute Paper The Muddiest Point Directed paraphrasing Application cards

The proof of the pudding... If we are teaching well, the obvious outcome would be in

student learning. One of the major challenges we face is that of effectively

‘measuring’ student learning. i.e. We are talking about effective assessment!