Upload
vui
View
27
Download
0
Embed Size (px)
DESCRIPTION
Evaluation for Learning. Paul Ramsden University of Sydney. The problem. To evaluate university teaching effectively, we should apply the principles of good practice in assessing student learning. Sounds elementary. - PowerPoint PPT Presentation
Citation preview
Evaluation for LearningEvaluation for Learning
Paul RamsdenPaul RamsdenUniversity of SydneyUniversity of Sydney
To evaluate university teachingeffectively, we should apply the principles of good practice in assessing student learning.
The problemThe problem
Sounds elementary...
“Chapter 2 has demonstrated that even the best of current practices are by and large not good practice...”
Lewis Elton and Brenda Johnston‘Assessment in universities: a critical review’
(http://www.ltsn.ac.uk/genericcentre/docs/Critical%20review%20of%20assessment %20research.rtf)
28
45
66
25
38
67
Helpful feedback
Staff available todiscuss work
Teaching is good
1999
1994
Source:McInnis et al, 2000
“There are no practice exercises – every piece of work is assessed, which tends to focus me on attaining marks rather than exploring ideas”
“Assessment and marking have not been good. It’s quite subjective, and group work assignments are sometimes unfair in their assessment criteria (for example, everyone gets the same mark even though one person may do less work than another)”
“The amount of multiple choice questions in some subjects does not provide opportunities to show how much you have learnt and understood.”
“We need more feedback during semester. It is not reasonable to get a mark at the end of the year and have no idea what it is based on. It is also not ideal to front up to an exam not knowing how well we have been going in earlier assessments.”
Bad practice remains commonBad practice remains common
Multiple purposes, same techniques No formative assessment at all No match to learning objectives Feedback too little, too late Narrow range of methods unreflectively chosen No student choice Incorrect use of group assessment Ignorance of Heisenberg! Standards assumed to depend on norm referencing
Interaction reliability x validity not understood
Unreliable assessments weighted less
Limited application of grade descriptors
Use of MCQs without professional training
Plagiarism opportunities not designed out – focus on technical fixes
Bad practice remains commonBad practice remains common
Underlying all this is...
a focus on the producer’s concerns a continuing emphasis on teaching (rather than learning) a remarkable neglect of existing evidence (people seem to prefer ‘dissemination and projects’ despite overwhelming proof that it doesn’t work) and a tendency to blame the student
We should try adapting these conclusions to the evaluation (and management) of university teaching.
THEORY 1
Teaching as telling
THEORY 2
Teaching asorganising
THEORY 3
Teaching as makinglearning possible
FOCUS Teacher andcontent
Teaching techniquesthat will result inlearning
Relation betweenstudents and subjectmatter
STRATEGY Transmitinformation
Manage teachingprocess; transmitconcepts
Engage; challenge;imagine oneself as thestudent
ACTIONS Chieflypresentation
‘Active learning’;organising activity
Systematicallyadapted to suit studentunderstanding
REFLECTION Unreflective;taken for granted
Apply skills toimprove teaching
Teaching as aresearch-like,scholarly process
Sydney 1999-2003Sydney 1999-2003
Assignment:
Design a system to evaluate teaching which leads to a better student experience and improved learning outcomes.
… and make it work.
The responseThe response
1. Break the problem into manageable parts
2. Design systems that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
The responseThe response
1. Break the problem into manageable parts
2. Design systems that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
Break the problem into manageable partsBreak the problem into manageable parts
Make goals, methods, measurement and outcomes cohere ( aka ‘alignment’)
Recognise importance of perceptions/ theories in use
Design the system around the culture (avoid one-size-fits-all solutions)
Learn from others’ mistakes
The responseThe response
1. Break the problem into manageable parts
2. Use strategies that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
PrinciplesPrinciples
• Adopt a single, but flexible, SAL perspective, derived from research evidence. Use it to inform every policy and process
• Plan for coherence between collegial and managerial strategies
• Use an evidence-based approach to change and leadership, aligned with academic values
Mechanisms to leverage strengthsMechanisms to leverage strengths
• Academic Board reviews• Rigorous, peer review-driven QA process
• Manage teaching proactively• Funding drivers aligned with research measures and
national indicators
• Plans that work
• New role for academic development unit
• Strategic projects
Management of teachingManagement of teaching
• Teaching Dividend currently $4.5M
• Scholarship Index c. $650,000
• Teaching Improvement Fund$1.3M
• Required T&L plans– Annually updated operational plans
– Assess progress against targets
– Condition of access to performance-based funds
– Interrogated in Academic Board reviews
Criterion Description of Indicators Weight
Student progress Unit of study pass rate 2
First to secondyear retention
Percentage of students still enrolled afterone year
2
Graduateexperiences
Scores on three measures: good teaching,generic skills, overall satisfaction
3 x 1
Studentexperiences
Scores on three measures: good teaching,generic skills, overall satisfaction (annualstudent course experience questionnaire)
3 x 2
Employability First destinations survey employment rate 2
Further study First destinations survey rate of further study 2
Strategic projectsStrategic projects
• Re-engineered academic development unit
• First year experience: learning community
• Expansion of training opportunities
• including mandatory 21 hour training
• Research-led teaching, including PIs
• Evaluation and QA working group
• Graduate attributes for a research university
• Research supervision initiatives
The responseThe response
1. Break the problem into manageable parts
2. Use strategies that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
Align evaluation with desired outcomesAlign evaluation with desired outcomes
• Rewards and recognition at multiple levels
• SI rewards trained staff and scholarly outputs
• Array of student-focused evaluation instruments, consistent with SAL theory
• New teaching awards
• Performance linked to funding at Faculty level
• Material support for changes
• New promotions policy
Evidence-based academic promotions criteriaEvidence-based academic promotions criteria
1. Fundamentals
2. Criteria Performance
Research-ledStudent-focusedScholarshipLeadership
3. Evidence
Fundamental things have got to be simple… we must look for simplicity in the system first.
Ernest Rutherford
FundamentalsFundamentals
FundamentalsFundamentals
Interest and explanation
Respect for students
Appropriate assessment
Clear goals and challenge
Independence: student control
Learning from students
(Learning to Teach in Higher Education, Chapter 6)(Learning to Teach in Higher Education, Chapter 6)
Even more fundamental...Even more fundamental...
Positive attitude towards students
Ability to communicate well
Lively interest in improving teaching
And plainer still...And plainer still...
The aim of teaching is simple: it is to make
student learning possible.
PerformancePerformance
A lecturer should appear easy and collected, undaunted and unconcerned, his thoughts about him and his mind clear for the contemplation and description of his subject … His whole behaviour should evince a respect for his audience
Michael Faraday
PerformancePerformance
Planning (e.g. effective subject design, clear objectives)
Process (e.g. presentation technique, WebCT design)
Assessment (e.g. use of variety of appropriate methods)
Outcomes (some evidence of link to learning)
Evaluation (some evidence of use of evaluation to improve)
Research-led teachingResearch-led teaching
This atmosphere of excitement, arising from imaginative consideration of knowledge, transforms knowledge.
A. N. Whitehead
Research-led teachingResearch-led teaching
Imagination and enthusiasm: a shared journey to understanding rather than delivery of content
Effective design of curricula to engage students in inquiry
Materials make use of primary sources, recent discoveries, progress in field
(“If you can’t explain it to the charlady, you don’t know anything about it”)
Student-focused teachingStudent-focused teaching
The two secrets of lecturing from which everything else follows: first, to believe that you have something worth telling your audience; second, to imagine yourself as one of that audience.
R.V. Jones
Student-focused teachingStudent-focused teaching
Use of evaluation evidence to redesign curriculum
Use of assessment data to modify teaching strategy
Focus on relation between students and subject matter
Choice of technique reflects level of student knowledge
(From “Did I make the goals clear?” to “Are the goals clear to the students?”)
Scholarship in teachingScholarship in teaching
What is needed is for teachers in higher education to bring to their teaching activities the same critical, doubting and creative attitude which they bring habitually to their research activities.
Lewis Elton
Scholarship in teachingScholarship in teaching
Systematic use of best available evidence to select and deploy teaching and assessment strategies
Publication of refereed journal articles on university teaching in discipline
Invitations to address international conferences on university teaching
Leadership in teachingLeadership in teaching
She successfully inspired us to transform the course and to re-focus on our students. She melded a diverse group of academics into a team of great teachers.
A lecturer
Leadership in teachingLeadership in teaching
Policy development and implementation
Successful re-design and coordination of courses; team leadership in teaching
Mentoring of junior academics as teachers
Application of teaching strategies and curriculum designs in other institutions
Coordination of benchmarking activity with other universities
Criteria are hierarchically ordered...Criteria are hierarchically ordered...
Non-negotiable basis: Performance
Second level: Research-led
Third level: Student-focused
Fourth level: Scholarship
Fifth level: Leadership
… leading to a structure that can be mapped on to promotion at different levels.
And the evidence?And the evidence?
Are the basics in place?
Use multiple sources (never rely on student evaluations alone)
Evaluate teaching like research– Use peer review if possible
– Use hard data when available (e.g. S of T publications)
Do the different sources tell a similar story?
Do the claims made by the teacher match the evidence?
Required
Recognised and encouraged
Administrative Academic
Required
Recognised and encouraged
Administrative Academic
Fundamental values:
• research intensive • academic-led• self-regulation• evidence base• international referencing• focus on student experience
Academic Board reviews(self-evaluation, visit,
report)Policies on teaching:
evaluation, assessment,
ICT QA, promotions
Surveys TPIs and performance fundingExternal QA benchmarksRequired training in teachingTeaching & Learning Plans
University teaching awardsSupervision awardsTeaching Improvement FundScholarship IndexResearch-led teaching (policy and indicators)
‘Guidelines for Good Practice’ (teaching, learning with ICT);
ITL courses and support groups: quality,
graduate attributes,first year experience,
research-led teaching
Required
Administrative
Recognised and encouraged
Academic
The responseThe response
1. Identify the problem
2. Use strategies that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
BenchmarksBenchmarks
Oxford Student surveys (SCEQ), QA policies
Lund QA policies
UCL QA policies
OU ICT evaluation and QA
ANU ICT in T&L for research universities
Monash Research-led teaching PIs
Queensland Student surveys (SCEQ)
Hong Kong Academic development standards
The responseThe response
1. Identify the problem
2. Use strategies that resonate with values and leverage strengths
3. Align evaluation with outcomes
4. Benchmark good practice
5. Test the impact against the evidence
-30
-20
-10
0
10
20
30
40
50
60
GoodTeaching
GenericSkills
Assessment Satisfactionwith Support
OverallSatisfaction
Conventional vs. Innovative
Old
New (SMP)
0
5
10
15
20
25
30
35
40
Sydney Uni 1 Uni 2 Uni 3 Others
1999
2000
2001
2002
2003
Demand indicator for high quality students(percentage of offers to students with UAIs 95 or greater
1999-2003)
Changes in the Sydney first year experience, 1999-2002Changes in the Sydney first year experience, 1999-2002
1999 2002 % change
Teaching staff give helpful feedback 39 48 +9
Teamwork skills developed 48 58 +10
Motivated to do best work 38 48 +10
More confident to tackle new problems 47 54 +7
IT supports my learning 56 64 +8
Problem solving skills developed 52 58 +6
Feel part of a learning community 39 53 +14
Satisfied with dept/faculty admin. 51 68 +17
Overall course satisfaction 66 71 +5
Overall satisfaction (services & admin) 57 65 +8
Percentage agreements, annual survey
0
10
20
30
40
50
60
70
1999 2000 2001 2002
Helpful feedback Feedback only marks All you need is memory
0
10
20
30
40
50
60
70
1999 2000 2001 2002
Satisfied with faculty admin. Overall satisfaction IT supports learning
Teamwork skills Part of a learning community
Oxford vs. Sydney, 3rd year Undergraduates Oxford vs. Sydney, 3rd year Undergraduates (broad agreement)(broad agreement)
Obstacles to an evidence-based approachObstacles to an evidence-based approach
Rationality in an audit society (Smith, J.R.Statist.Soc. 1996)
EBM is an example of a very uncommon phenomenon
Motivation – what’s in it for me? Focus on teaching and methods Antediluvian attitude to ‘staff development’
– yet more projects, ‘action research’, ‘dissemination’, ‘skills’, centralist policies …. itself not evidence-based
Management inadequacies
‘Staff development & dissemination’ is the front end only: management & structures are
required
Design for your users: focus on learning; have a bold vision; capture academic imaginations through trust and credible leadership
Celebrate diversity in the sector: be inclusive
Operate at multiple levels: use a systems perspective
Forget about “accreditation”
Make it easy to share good practice
Don’t become an arm of the audit society
Advice to the National Institute for L & TAdvice to the National Institute for L & T
Further readingFurther reading
(1)(1) Management of T&L, teaching qualityManagement of T&L, teaching qualityPaul Ramsden
Learning to Teach in Higher Education
Second Edition 2003 Foreword by Sir David Watson
London: RoutledgeFalmer
– – the classic text fully revised and updatedthe classic text fully revised and updated
(2)(2) Practical advice for headsPractical advice for headsPaul Ramsden
Learning to Lead in Higher Education 1998
London: RoutledgeFalmer
www.routledgefalmer.com