Student Evaluation of Teaching and Course (SETC)
On-boarding Package
Prepared by Kiran Bisra, Director, Learning Experiences Assessment & Planning (LEAP)
2
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Contents 1. Introduction:......................................................................................................................................... 3
1.1 Objective ...................................................................................................................................... 3
1.2 SETC Administration ................................................................................................................. 3
1.3 Background ................................................................................................................................. 3
2. Cascading Framework ....................................................................................................................... 4
3. Question Form .................................................................................................................................... 4
3.1 Instructor Question Selection (IQS) ......................................................................................... 5
4.1.2 Aggregated Faculty Report ............................................................................................... 5
4.1.3 Aggregated Evaluating Unit Report ................................................................................. 5
4.1.4 Course Report ..................................................................................................................... 5
4.1.5 Individual Report ................................................................................................................. 6
4.2 Instructor Narrative ..................................................................................................................... 6
4.3 Report Access ............................................................................................................................. 7
5. Semester Workflow ............................................................................................................................ 8
Step 1: Data Verification ....................................................................................................................... 8
Step 2: Instructor Question Selection (IQS) ....................................................................................... 9
Step 3: Evaluation Period ..................................................................................................................... 9
Step 4: Release of Reports ................................................................................................................... 9
6. On-boarding ...................................................................................................................................... 10
Step 1: Gather information .................................................................................................................. 10
Step 2: Data Verification Training ...................................................................................................... 10
Step 3: Implementation of Pilot Project ............................................................................................. 10
Step 4: Assessment of Pilot ................................................................................................................ 11
Step 5: Complete roll-out .................................................................................................................... 11
Appendix A: Evaluation Form for FASS ................................................................................................ 12
Appendix B: Evaluation Form for FENV ............................................................................................... 14
Appendix C: Evaluation Form for FCAT ............................................................................................... 16
Appendix C: Join Form ............................................................................................................................ 18
3
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
1. Introduction: The Student Evaluation of Teaching and Courses (SETC) initiative is a new online system that has replaced pen-and-paper course evaluations in many SFU academic units. The SETC team is currently scaling up the program to the remainder of the University.
Please visit our website to find out more: www.sfu.ca/setc
1.1 Objective
SFU has a strong commitment to teaching and learning, and the SETC initiative is a reflection of this commitment. The goal of the SETC program is to meaningfully collect and disseminate feedback from students on teaching, learning, and course delivery at SFU.
1.2 SETC Administration
SETC is in the AVP Learning and Teaching portfolio. The SETC Team is situated in the Learning Experiences Assessment & Planning (LEAP) Division of the Centre for Educational Excellence (CEE) and handles the operational aspects of the program (i.e. administering evaluations, generating reports, research and measurement).
1.3 Background
The SETC initiative has been a long time in the making. The process started with a community engagement process back in 2012. The online evaluation instrument was piloted in Summer and Fall 2015, and the following academic units are currently in the system:
Education Health Sciences Applied Science Science Arts and Social Sciences (7 departments) Communication, Arts and Tech (3 units) Environment
There are several circumstances in which a course will be excluded from the SETC system. Sections taught by Teaching Assistants are excluded due to specific considerations outline in their Collective Agreement. The SETC team is working with the TSSU to resolve this situation and on-board their members. As well, Center for Online and Distance Education (CODE) courses, CO-OP Placements, and sections with fewer than five students are not evaluated using the SETC form.
4
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
SFU has purchased third-party software called “blue by eXplorance” to administer the
evaluations. Members must login using their SFU Central Authenticating Service (CAS) to
“blue” to fill out evaluations, view reports, or write questions.
The SETC program is in a state of review. There may be changes to the structure, questions
and reports in the near future.
2. Cascading Framework It is a challenge to meet the needs of all stakeholders with a single evaluation form or report. Therefore, SETC evaluation forms and reports are customizable based on by a 4-tier cascading framework (Figure 1). An Evaluating Unit is either a Department, a School, or, for Faculties that do not have academic units, the Graduate/Undergraduate Program.
Figure 1: Cascading Framework
3. Question Form SETC evaluation forms are customizable based on the cascading framework described above. At SFU, there are 12 questions chosen by the VPA’s office which appear on all forms. Then, there are four questions chosen by the Dean’s Office which appear on all courses within that respective faulty. The Evaluating Unit Head chooses a further four more questions to appear within their unit. Lastly, each instructor has the option of selecting up to four questions per course section they taught.
Usually the questions chosen at the various levels are the result of consultation. For
example, the institution-level questions were constructed by a Senate Committee on
Teaching and Learning (SCUTL) Working Group. Some Faculties/Evaluating Units have
chosen their questions after consultations with instructors, while others task a handful of
faculty members to make this decision.
All current SETC forms are available here online.
Institution
Faculty
Evaluating Unit
Instructor
5
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
3.1 Instructor Question Selection (IQS) Instructors are emailed a link every semester to select or create up to four questions per course. This link redirect them to “blue” where they have access to an item bank. Currently, no one examines these questions before they are sent out to students.
Instructions on how to select questions are available here online.
3.2 Sessional Instructors As of right now, Sessional Instructors only receive the University-level and Instructor-selected questions. The reason for this is related to the TSSU Collective Bargaining Agreement.
4. Reporting
Reporting follows the same four-tier cascading model used to select questions. Five reports
are generated and an email notification is sent out to all Report Viewers at the end of week
1 of the subsequent semester. Report Viewers log into “blue” to access their reports. For
an overview, see Table 1.
4.1 Reports
4.1.1 SFU Report
The SFU Report is sent to the AVP Teaching and Learning and contains only the Institution-level questions. Frequencies, scores and response rates of all respondents are aggregated for this report.
4.1.2 Aggregated Faculty Report
The Aggregated Faculty Report is sent to the Dean’s Office and contains the Institution and Faculty-level questions. Frequencies, scores and response rates of students who took instruction within the faculty are aggregated for this report.
4.1.3 Aggregated Evaluating Unit Report
The Aggregated Evaluating Unit Report is sent to the Evaluating Unit’s Office and contains the Institution, Faculty, and Evaluating Unit-level questions. Frequencies, scores and response rates of students who took instruction within the Evaluating Unit are aggregated for this report.
4.1.4 Course Report
The Course Report is sent to the Evaluating Unit’s Office and contains the Institution, Faculty, and Evaluating Unit-level questions. Frequencies, responses, and response rates of
6
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
students who took instruction within the course are aggregated for this report. This recipient may receive dozens of Course Reports per semester.
4.1.5 Individual Report
The Individual Report is sent to the instructor of the section, and contains the Institution, Faculty, Evaluating Unit, and instructor selected questions. Frequencies, responses, and response rates of students who took instruction within the course are aggregated for this report. This recipient will receive the same number of reports as courses taught each semester.
4.2 Instructor Narrative
During the evaluation period, instructors are able to submit a brief narrative or make note of any circumstances or self-reflections they would like appended to their Individual and Course reports; this narrative is completely optional.
7
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Table 1: Summary of Reports
Report Name Recipients Question-level Instructor Narrative
Institution Faculty Evaluating Unit
Instructor
SFU AVP - LT Aggregated Faculty Dean Aggregated Evaluating Unit Evaluating Unit Head Course Evaluating Unit Head Individual Instructor
4.3 Report Access
Each unit is tasked with keeping an up to date Report Viewer lists. If collating and archiving reports are within one’s job description, then access is granted. Chair’s Secretary, Manager of Administration, and Program Assistant are three common personnel who have access. In order to complete their tenure, promotion or salary appeal duties, the VPA and Faculty Deans will be given access to Course Reports. Besides the instructor, access to Individual Reports are never granted to another Report Viewer.
We require that you identify a Report Viewers List steward who can login to our Report Viewers Application and make edits to the list as required. Course evaluation reports contain sensitive information about instructors. Failure to remove viewers who no longer require access could result in a serious breach of privacy.
The application can be found at: setcreportviewers.its.sfu.ca
8
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
5. Semester Workflow
There are four main steps in each semester.
Step 1: Data Verification
During the first step, information is inputted into “blue” from several sources.
Often, the questions and Report Viewers will be the same term to term, and no change will be required. If a change to the questions is required, email [email protected] at the beginning of the term.
Course lists, course end dates, and instructor information is pulled from the Student Information Management System (SIMS). This data needs to be checked by Evaluating Unit Administrative personnel. Common corrections include:
Instructor change after the term has started (i.e. emergency, sickness); Accounting for team or co-taught courses; Combining course sections when an instructor teaches several small sections and
one report is required; Combining cross-listed courses to generate one report for the course; Editing the course end date to align with course practices.
Figure 2: Fall 2018 Semester Workflow Overview
September October November December
4. Release of Reports
1. Data Verification
2. Instructor Question Selection
3. Evaluation Period
Last Day of Classes
9
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
These changes are made in a SIMS component called “SETC EDITS”. Each unit decides who is responsible for verifying this data. Commonly chosen positions include Manager of Administration, Scheduler, and Program Assistant. Once this decision has been made, please email [email protected] to schedule a training session. The session will go into detail on how to make changes in SETC EDITS and which issues to keep an eye on as the semester proceeds.
Evaluating Units are not required to provide enrollment information. Enrollment lists are updated each day from SIMS. Please note that it is possible for a student to be enrolled at the time the evaluation period starts, but then deregister before the reports are sent out. In this case, it is possible for the report to list a higher number for respondents than enrolled.
Step 2: Instructor Question Selection (IQS)
Instructors are emailed a link every semester to select or create up to four questions per course (see Section 3.1). Instructors select/create their questions right before the evaluation period.
Instructions on how to select questions are available here online.
Step 3: Evaluation Period
For most classes, the evaluation period will be the last two weeks before the end of classes. If the class ends early in the term, it may be the week before the last instruction day.
Instructors receive an email when the evaluations open up with:
1. Evaluation end dates 2. A link to access a response rate monitor which updates in real time, and 3. An invitation to submit a brief narrative (see Section 4.2).
For more information on the response rate monitor click here.
Step 4: Release of Reports
An email notification is sent out to all Report Viewers a day or two after the grade submission deadline. Report Viewers log into “blue” to access their reports.
10
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
6. On-boarding
As the on-boarding process gets underway, it will be important to regularly communicate with instructors. While the SETC Team can support these communications, they are best and most effective if sent by the Evaluating Unit Head.
Step 1: Gather information
There are three key pieces of information the SETC team needs in order to proceed with
setting-up your unit:
1. The name of 1 or 2 data verification personnel 2. Report Viewers list for the Course Reports and identification of steward 3. Unit-level questions
Your unit may choose up to four questions. The institution and faculty-level questions that your students will receive can be found in Appendix A or B. Often one or a handful of faculty members are tasked with selecting the questions. It is recommended that the questions be chosen after some level of consultation with instructors.
If your future plans are to use the same questions for both Sessional Instructors as well as
with continuing faculty, then you must consult with the Teaching Support Staff Union
(TSSU).
Questions are categorized as either a course question or a teaching question. If a course is
taught by multiple instructors, course questions appear once on the evaluation form,
whereas teaching questions are asked about each instructor.
If you would require any help with selecting or formulating your questions, please contact
Lana at [email protected].
Once you have gathered the three pieces of information, please email them to Kiran at
Step 2: Data Verification Training We will contact the data verification personnel and set-up a SETC EDITS (see Section 5.1) training session. This training is crucial for implementation success.
Step 3: Implementation of Pilot Project
Next, a pilot project will be administered. Whether all courses or just a handful of courses are in the pilot will be determined by the SETC Team. It will depend on a myriad of factors, including the complexity of evaluating unit course offerings. This small scale preliminary implementation will allow us the opportunity to evaluate processes and improve on them before the complete roll-out.
11
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Step 4: Assessment of Pilot
A meeting will be held between the SETC Team and the evaluating unit to discuss issues or concerns that arose from the pilot. Topics may include timelines, revamping the questions, changes in SIMS data processes, or alterations to the report viewers list.
Step 5: Complete roll-out
Lastly, a complete roll-out will be implemented.
12
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Appendix A: Evaluation Form for FASS
University-level Questions
General
1.
I attended class…
All the time
Most of the
time
About half of
the time
Rarely
Never
2.
How easy was the course?
Very Easy
Easy
Medium
Hard
Very Hard
3.
Overall, the quality of my learning experience in this
course was…
Very Good
Good
Fair
Poor
Very Poor
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
4. The different course parts/activities (lectures, labs,
tutorials, online forums, discussions, etc.) were
connected.
5. Course materials (textbook, readings, handouts,
assignments, etc.) improved my understanding of the
course content.
6. The assessments in this course (tests, assignments, essays,
etc.) allowed me to demonstrate my understanding of
the course content.
7. Course activities (lectures, discussions, group work, labs,
etc.) were engaging.
Teaching
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
8. The course instructor explained course concepts clearly.
9. The course instructor explained grading criteria clearly.
10. The course instructor created a respectful learning
environment.
13
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
11. The course instructor was approachable when students
asked for guidance.
Open-ended
12. Do you have any further comments?
Faculty-level Questions: Arts and Social Sciences (FASS)
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
13. We are interested in knowing to what extent courses
promote critical thinking. To think critically means to
carefully evaluate an idea or hypothesis on the basis of
the available evidence, and to try to reach a justifiable
conclusion using rational analysis within the context of a
specific discipline.
My experience in this course has encouraged me to think
critically.
14. My experience in this course has motivated me to learn
more about the subject.
Teaching
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
15. When explaining course concepts, the instructor
connected the concepts to the major themes of the
course.
16. The course instructor made it clear what students were
expected to learn in the course.
14
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Appendix B: Evaluation Form for FENV
University-level Questions
General
1.
I attended class…
All the time
Most of the
time
About half of
the time
Rarely
Never
2.
How easy was the course?
Very Easy
Easy
Medium
Hard
Very Hard
3.
Overall, the quality of my learning experience in this
course was…
Very Good
Good
Fair
Poor
Very Poor
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
4. The different course parts/activities (lectures, labs,
tutorials, online forums, discussions, etc.) were
connected.
5. Course materials (textbook, readings, handouts,
assignments, etc.) improved my understanding of the
course content.
6. The assessments in this course (tests, assignments, essays,
etc.) allowed me to demonstrate my understanding of
the course content.
7. Course activities (lectures, discussions, group work, labs,
etc.) were engaging.
Teaching
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
8. The course instructor explained course concepts clearly.
9. The course instructor explained grading criteria clearly.
10. The course instructor created a respectful learning
environment.
11. The course instructor was approachable when students
asked for guidance.
15
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Open-ended
12. Do you have any further comments?
Faculty-level Questions: Environment
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
13. This course improved my understanding of the built,
social, and/or natural environment
14. This course made me more aware of the challenges and
opportunities for creating a more just and/or sustainable
world.
15. I was able to link course concepts to current issues and
real-life situations.
16. This course inspires me to take more courses in the
Faculty of Environment.
16
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Appendix C: Evaluation Form for FCAT
Evaluation Form: University-level Questions
General
1.
I attended class…
All the time
Most of the
time
About half of
the time
Rarely
Never
2.
How easy was the course?
Very Easy
Easy
Medium
Hard
Very Hard
3.
Overall, the quality of my learning experience in this
course was…
Very Good
Good
Fair
Poor
Very Poor
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
4. The different course parts/activities (lectures, labs,
tutorials, online forums, discussions, etc.) were
connected.
5. Course materials (textbook, readings, handouts,
assignments, etc.) improved my understanding of the
course content.
6. The assessments in this course (tests, assignments, essays,
etc.) allowed me to demonstrate my understanding of
the course content.
7. Course activities (lectures, discussions, group work, labs,
etc.) were engaging.
Teaching
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
8. The course instructor explained course concepts clearly.
9. The course instructor explained grading criteria clearly.
10. The course instructor created a respectful learning
environment.
11. The course instructor was approachable when students
asked for guidance.
17
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Open-ended
12. Do you have any further comments?
Faculty-level Questions: Communication, Art and Technology (FCAT)
School of Interactive Arts and Technology
General
Very Light
Light Average
Heavy
Very Heavy
13. Compared to other courses, the workload in this course
was:
Course
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
14. The course assignment, projects, tests and/or papers
helped me to develop skills I can use in other courses.
15. The course provided opportunity for me to enhance my
problem-solving skills.
16. The course assignments, projects, tests, and/or papers
highlighted important concepts of the course.
17. The course expanded my understanding on important
issues in the subject matter.
Teaching
Strongly Agree
Agree No Opinion
Disagree Strongly Disagree
18. The instructor related course concepts to professional
practices in the field.
19. The feedback I received in this course on assignments,
projects, tests, and/or papers provided guidance on how
to improve my performance in the course.
Very Good
Good Fair Poor Very Poor
20. Overall, the quality of instruction provided by the
instructor in the course was:
18
Student Evaluation of Teaching and Course, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 778.782.3910 | www.sfu.ca/setc
Appendix C: Join Form
A) Data Verification Personnel
Course lists, course end dates, and instructor information is pulled from the Student Information
Management System (SIMS). This data needs to be checked by Evaluating Unit Administrative
personnel.
We need three key pieces of information about your data verification personnel:
- Name - Position - Email
B) Course and Aggregated Evaluating Unit Report Viewers List
We need three key pieces of information about your Report Viewers.
- Name - Position - Email
As well, we need the name of your Report Viewers List steward (the person who updates your
viewers list).
C) Evaluation Form Questions Please submit the following information about your unit’s evaluation questions.
Question
Type Course
Instructor
Response Scale Strongly Agree, Agree, No Opinion, Disagree, Strongly Disagree
Very Good, Good, Fair, Poor, Very Poor
Very Heavy, Heavy, Neither Heavy/Light, Light, Very Light
Other:
Email information to: [email protected]