Students’ Evaluation of Educational Quality (SEEQ)

Dr. Mary BenbowAssociate Professor,Department of Environment and Geography

University of Manitoba

Resources to Examine Evaluations

• Centre for Educational Advancement at Curtin University, Australia.

• “Students Rating Teaching” by Mark Lawall

• Many, many peer-reviewed research articles– A selection is

provided in the bibliography

• My web site• http://

home.cc.umanitoba.ca/~benbow/seeq.html

Using the SEEQ

• Student evaluations – what we know• Student Evaluation of Educational Quality (SEEQ)• Using the SEEQ

Student Evaluation: What We Know

• A lot of research (some apparently contradictory and/or inconsistent)

• Difficult to compare different instruments• Many useful reviews:

– Students Rating Teaching– Other articles (e.g. Ory, 2001)

Development and Use of Student Evaluations

• Introduced in the 1960s– Students sought “a stronger voice”– Today viewed as one component of accountability

• Over 90% of institutions of higher education use student evaluations of some sort

• Comprehensive scheme for student evaluations began at the U of M in the academic year 1996-97 (Policy 425)– September 2002, Senate approved voluntary posting of

SEEQ results on U of M website

Courses: Requirements, Levels and Sizes

• Electives tend to get higher ratings than required course– majors more favourable than

minors

• Higher level courses tend to get higher ratings

• There are differences between the disciplines

• Class size has little effect overall

Instructors and Students

• Student ratings positively correlated to those of alumni– positive correlation between students and colleagues

• Lower ratings for less experienced instructors and teaching assistants– relationship between research productivity and ratings is

either positive or nil

• Higher ratings where there is prior interest• Slightly higher ratings where gender of instructor and

students is the same

Administration and Use

• Diagnostic not prescriptive– Need to follow uniform

procedures– higher evaluations if followed by

a short speech– higher also if not anonymous– higher if the instructor stays in

the room

• Most important: not the only source of information concerning teaching

Student Evaluation of Educational Quality (SEEQ)

• Widely researched– very high degree of reliability (r = 0.88 - 0.97)

• Based on psychometric analysis (teaching not content)– correlate well with a wide range of measures of learning

outcome– correlates well with instructors’ self ratings

The development of the SEEQ

• Developed Herbert Marsh of the University of Western Sydney in late 1970s– Internationally recognised expert in

psychometrics

• SEEQ has been exhaustively researched– Statistical tests over 13 years, from

approximately 50,000 courses and almost 1 million students

SEEQ Factors Questions developed from interviews with faculty and students

about what constitutes effective teaching

• Learning• Organization• Individual Rapport• Breadth• Assignments

• Enthusiasm• Group Interaction• Examinations• Overall Ratings

Teaching as a multi-dimensional activity

SEEQ Questions

• Formative questions:– Identifies specific aspects of each factor– Organization:– Instructor’s explanations were clear

• Summative overall questions:– 30. Compared with other courses I have had at the U. of M., I would

say this course is:– 31. Compared with other instructors I have had at the U. of M., I

would say this instructor is:– 32. As an overall rating, I would say this instructor is:

Student Evaluation of Educational Quality:

Using the SEEQ

• Little research on summative use• Ratings are a reliable source of data • Data must then be examined and evaluated

– Examine the data to create assessment– Followed by decisions and examination of change

Using the SEEQ

• Understanding your priorities

• Undertake a self-evaluation of teaching

• Looking at your evaluations

• Identify solutions

• Making changes and smart decisions

Your Priorities

• Related to your Teaching Philosophy– Describes the principles that direct your

teaching style and activities

• Identify which factors and elements are most important to you– Are your priorities consistent those of

your discipline, department or institution?– Can you establish a reason for your

priorities

• Prior to examining your evaluations, it is important to assess which matter most to you

Self-Rating of Teaching

• The SEEQ correlates well with self ratings and trained observer ratings

• Useful to consider our assessment of teaching– Collect evidence of teaching and assemble in a

teaching dossier or portfolio– Consider our assessment using a self-rating form

based on the SEEQ– “Most of us await student reactions to our courses

with both eagerness and trepidation” (Delucchi and Pelowski, 2000)

Strongly Disagree

Disagree Neutral Agree Strongly Agree

1. I have found the course intellectually challenging and stimulating

2. I have learned something which I consider valuable

3. My interest in the subject has increased as a consequence of this course

4. I have learned and understood the subject materials of this course

Identify the most common responses you

expect from your class

• Are there any surprising responses?

• Which surprises are most relevant?

• Do the written comments reveal any more information?

• You can also use your statistical summaries of evaluation data (but differences of <0.05 are not considered significant)

Self-Analysis: Looking at your course evaluations

Analysis framework

• What special features reflect the subject, the students or other contextual factors?

• Eliminate any factors that are not appropriate to your intentions in teaching this class. – What are the main strengths of your teaching? – Which are the main weaknesses in your teaching?

• How do these findings differ from your Self-Rating Survey

Other Useful Data on the SEEQ

• Students Enrolled and Students Responding can indicate attendance (e.g. 62%)

• Expected Grade can be compared to the actual grade distributions

• Reason for taking course can reveal how many are required to take a course

“There’s always one”

! ?

• What should we do about that one or two reponse(s) that stand

out, perhaps anomalously?

• Look at the response sheets and identify which responses are the outliers

• Look at their written comments

• Look at the Student and Course Characteristics

• Are the outliers only in one question or subject area? (e.g. use of humour)

• Are they relatable (Level of interest (37) & Learning questions)?

Using the SEEQ:Making changes

• Using your priorities and self rating decide where changes would be most useful

• Need to identify sources of information – Choose information sources to suit your teaching and

learning style– Start small (quick, easy, cheap) before investing in large,

long-term changes

Select key areas for improvement

• Which are the two or three areas that received the lowest ratings? – What strategies for improvement could

you use that would be appropriate to your particular situation?

• Could the Tips to Improve Academic Teaching help you in this?

• Would further personal guidance or investigation be needed to help you improve your knowledge and skills?

Making Changes

• Start small before investing in long-term changes

• Why go it alone?– Could there be some changes that could be

accomplished by group action?

• It takes time to create a new habit– When you try something new, it may feel

uncomfortable at first– Give yourself reminders to try some new techniques

or ideas

A good place to start are the “Tips to Improve Academic Teaching” organized in 8 sections referring to each

factor12. Instructor gave lectures that facilitated taking notesStrongly Strongly Disagree Disagree Neutral Agree Agree 9 18 3 13 7

This question comprises part of the Organization factor: choose Organization & Clarity SEEQ Factor 3

Statistical Analysis of SEEQ Results

• SEEQ data cannot be analyzed or summarized statistically by administrators

• But, individuals can analyze their own data• The Statistics Advisory Group recommends the

following techniques• Can be undertaken by hand or by using a

computer spread sheet

Averages, Medians and Grouped Questions

• Find the average response for each question• Find the median response for each question• Find the average for each subject grouping or

FACTOR (e.g. average response for all four questions under “Learning”)

• 1. I have found the course intellectually challenging and stimulating

N/A 01 Strongly Disagree 02 Disagree 03 Neutral 34 Agree 205 Strongly Agree 2

• Average = (0*1)+(0*2)+(3*3)+(20*4)+(2*5) = 99/25 = 3.96

• Median = 4

• 1. I have found the course intellectually challenging and stimulating

• 2. I have learned something which I consider valuable

• 3. My interest in the subject has increased as a consequence of this course

• 4. I have learned and understood the subject materials of this course

Av. 3.96Med. 4.0

Av. 4.16Med. 4.0

Av. 4.08Med. 4.0

Av. 4.24Med. 4.0

Learning factor: Average = 4.11, Median = 4.0

You may wish to provide summaries to indicate improvements in your teaching

‘00-’01 ‘01-’02 ‘02-‘03

1. I have found the course intellectually challenging and stimulating

3.6 3.6 3.7

2. I have learned something which I consider valuable

3.8 3.9 3.8

3. My interest in the subject has increased as a consequence of this course

3.7 3.7 3.8

4. I have learned and understood the subject materials of this course

4.0 4.0 4.3

You can also graph changes in averages (in a teacher dossier, include an explanation)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2000-01 2001-02 2002-03 2003-04

Q. 30Q. 31Q.32

Additional Questions in the SEEQ

• At Curtin University (Perth, Western Australia)• Staff engaged in the 1995 pilot of SEEQ were concerned

that it might not accommodate forms of teaching and learning other than conventional face-to-face lecturing

• Teaching staff across the disciplines have subsequently developed factors and questions to complement the SEEQ form.

SEEQ Questions Examples: The computer laboratory

1. The laboratory was an essential part of the unit.

2. Laboratory-practical exercises reinforced the main points of the lecture.

3. By applying theoretical concepts in the laboratory, my understanding of the material improved.

4. The time and effort required for computer-based laboratory work was reasonable.

Written Comments

• Type them up– Include course title,

number, year, number of students, etc.

– What additional depth do they provide for the question responses?

– e.g. if you score well in Organization the comments may bring up a clear outline, good notes, web pages, etc.

• Note: Not to be used for tenure or promotion

• Use your priorities and self rating to decide where changes would be most useful

• Innovations to improve teaching are everywhere– Use the library, NetDoc or Google Scholar to find great

information and ideas

• The SEEQs are not the only source of information about your teaching – Develop your teaching dossier– Develop your teaching philosophy and goals

Students’ Evaluation of Educational Quality (SEEQ) :

Conclusions