Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Course Evaluations and ROSI: Preparing courses for evaluation A Step-By-Step Guide
Centre for Teaching Support & Innovation
4th Floor, Robarts Library, 130 St. George Street
Toronto, Ontario
1
Table of Contents
About Course Evaluations and this Guide .................................................................................. 2
When to Complete Data Verification .......................................................................................... 3
Verifying Your Course Diagnostics for the Term ......................................................................... 4
Data Verification from Beginning to End ..................................................................................... 5
Logging in to ROSI Express ................................................................................................... 5
Accessing the right reports ..................................................................................................... 6
The Scheduled Courses with Instructor/Coordinator Diagnostics Report ................................ 7
The Input Screen ................................................................................................................ 7
The Report .......................................................................................................................... 8
Assignments Errors and How to Fix Them .......................................................................... 9
01: No instructor assignment ........................................................................................... 9
02 Instr assignment not linked to HRIS ...........................................................................11
03: No instructor UTORid ...............................................................................................12
04: No Instructor UTORid and check appt start/end dates & 05: No Instructor UTORid
and instr appt in org does not exist .................................................................................12
06: Check employee appointment start/end dates ..........................................................12
07: Employee appointment in the selected organization does not exist ..........................12
Error 1070’: EmpTyp ‘HR’ with no UTORid .....................................................................12
An error not listed here ...................................................................................................13
Changing a Course Instructor ............................................................................................14
Managing Course Coordinators .........................................................................................15
Showing Only Missing or Problematic Assignments ...........................................................16
Exporting the Report to Excel .............................................................................................17
The Confirm Courses and Instructors for Evaluation Report ..................................................18
The Input Screen ...............................................................................................................19
The Report .........................................................................................................................20
Courses with zero students enrolled ..................................................................................21
If a change occurs after the ROSI Express Data Verification Deadline… ..................................22
Data Verification: The Essential Steps ......................................................................................23
Need more support? Contact us! ...........................................................................................24
2
About Course Evaluations and this Guide
The Course Evaluation Framework plays an important role at the University of Toronto by
providing students the opportunity to comment on their learning experience in their courses. In
turn, evaluations provide instructors and academic administrators with student feedback. The
integrity of course evaluation data is essential to the Framework’s quality assurance principles
and its successful operation.
It is the responsibility of each division/department to ensure their course-instructor-student
information in ROSI is correct by a deadline determined by the Course Evaluations (CE) Team
so that accurate information is reflected in Blue, the online course evaluation system. Blue
extracts information from two campus-wide information systems, ROSI (Repository of Student
Information) and HRIS (Human Resources Information System).
Incorrect information in ROSI can lead to errors in Blue, and incorrect evaluations being issued
to instructors and students. This can negatively impact on an instructor’s promotion, tenure, and
review process. Errors also have an effect on the release of course evaluation reports.
This guide is intended to support individuals responsible for verifying course-instructor-student
information in ROSI by providing them with a step-by-step walkthrough of the process, as well
as the tools to troubleshoot potential issues.
3
When to Complete Data Verification
At the beginning of each academic term – Fall, Winter, and Summer – a member of the CE
Team will provide you with a ROSI Express-Course Evaluations Data Verification Deadline.
For accurate information about instructors and courses to be included in course evaluations, the
two ROSI Express Reports Scheduled Courses with Instructor/Coordinator Diagnostics and
Confirm Courses and Instructors for Evaluation must be reviewed, and any necessary changes
made by that date.
Important Note: If there is a change to any course-instructor assignment information after the
ROSI Express deadline, please contact the CE Team ([email protected])
directly. Changes made in ROSI after the deadline are NOT captured by the Course
Evaluations System and will need to be adjusted manually by the Course Evaluations Team.
4
Verifying Your Course Diagnostics for the Term
The following steps are crucial and must be completed by your ROSI Express Deadline.
1. Using the Scheduled Courses with Instructor/Coordinator Diagnostics report in ROSI
Express, ensure instructors are assigned to the correct courses. Correct any course-
instructor assignments showing errors 01 through 05, as courses and instructors with
these errors will not be evaluated.
2. Using the Confirm Courses and Instructors for Evaluation report in ROSI Express,
review the Evaluation Method and Evaluation Timing for each course-instructor
assignment. If the Evaluation Method is set to “NOT” this course will not be evaluated.
3. Review your data for the following common issues:
a. “Global Eval Y/N”/Courses set to “NOT” be evaluated: On the input screen for the
Confirm Courses and Instructors for Evaluation report, there is an option that
allows you to “Show only Courses/Instructors that are NOT set up for evaluation.”
Before completing your review, make use of this option to ensure all courses with
a setting of NOT should indeed NOT be evaluated. If the course has a value of N
in the Global Eval Y/N column, this course will NEVER be evaluated. Should
there be an error in this value, contact the Course Evaluations Team, as you will
not be able to change this setting yourself.
b. Evaluation Method – PPR: Instructors are eligible to use paper based evaluation
forms only if they are pre-tenure/pre-promotion continuing appointment faculty,
hired before July 1, 2012. If instructors have requested paper evaluation forms
but do not meet the above criteria, their Evaluation Method must be set to ONL
for online.
c. Courses with more than four instructors – A maximum of four instructors
teaching a single lecture and meeting section (i.e. four instructors all assigned to
ANT100Y-Y-LEC0101) can be evaluated by the online evaluation system. Any
course with more than four instructors requiring evaluation is not currently
evaluated by the online evaluation system. Please set the Evaluation Method for
all instructors to NOT. Contact the Course Evaluations Service to seek guidance
with respect to an alternate evaluation method. Otherwise, continue historical
evaluation practices within your Division.
d. Courses with 0 students enrolled – Courses with no students enrolled should not
be included for evaluation. For these courses, be sure to change the Evaluation
Method to NOT and the Evaluation Timing to [blank].
4. Return to the Scheduled Courses with Instructor/Coordinator Diagnostics report
and generate an Excel download of this report for your unit. Review this report for
potential errors, and keep a copy for your records.
5
Data Verification from Beginning to End
Logging in to ROSI Express To log-in to ROSI Express you must have an eToken. Once your eToken is connected to your
computer, go to: https://www.sis.utoronto.ca/rosi-express/, and follow log-in instructions.
Should you require access to ROSI Express please contact ROSI Help at [email protected]
6
Accessing the right reports Both the Scheduled Courses with Instructor/Coordinator Diagnostics and Confirm Courses and
Instructors for Evaluation reports can be found on the Reports page, which can be accessed on
the menu on the left-hand side of the ROSI Express main page.
7
The Scheduled Courses with Instructor/Coordinator Diagnostics Report The Scheduled Courses with Instructor/Coordinator Diagnostics report allows you to verify the
list of your scheduled courses and their current instructor/coordinator assignments, and also to
make changes that reflect your current course and instructor assignments directly on the
report itself.
Using this report, you can:
View your scheduled courses for a session or academic year
View the instructors and coordinators assigned to these courses
Identify if there are any issues with these assignments or the HRIS records of the
assigned faculty members
Make necessary updates or changes to your instructor and coordinator assignments
The Input Screen 1. Enter the session code for the report (e.g. 20179).
2. Enter your Primary Organization Code.
3. If applicable, enter your Secondary Organization Code AND/OR Co-Secondary
Organization Code. This is the ROSI code for your unit/department.
4. Click Submit.
By default, the report does not include a row to indicate which courses are missing coordinator
records, because not all divisions make use of the Course Coordinator role. If you wish to
include missing coordinator rows in your report, check the Include Missing Coordinator box.
Individuals associated with the Coordinator role in ROSI are not evaluated by the Course
Evaluation system. If the individual is also associated with the instructor role, then they can be
configured for evaluation.
8
The Report The Scheduled Courses with Instructor/Coordinator Diagnostics report displays scheduled
courses with instructor/coordinator assignments for the session code and organization codes
you entered. Each course meeting section, with its respective instructor or coordinator
assignments, will appear as a separate row in the report. Please note that the report can only
identify data that is missing or problematic. It is unable to detect if the incorrect person is
assigned as a course instructor or coordinator.
1. The “Assignment Message” column provides information about each instructor or
coordinator assignment. If there are no detectable issues with the assignment, the
messages “99: Instructor OK” or “99: Coordinator OK” will be displayed.
2. On the right-hand side of the report, there are icons representing each of the actions you
can take to manage your course-instructor and coordinator assignments.
a. ADD: Use this option to add an instructor or coordinator assignment to this
course meeting section. You can add an instructor or coordinator to a course
that has none, or add additional instructors or coordinators when an assignment
already exists.
b. MODify: You can use this option to edit the existing instructor or coordinator
assignment, e.g. if you need to change the course instructor from one professor
to another.
c. DELete: This option deletes the instructor or coordinator assignment. The Delete
option does not delete the course meeting section. It only removes the instructor
or coordinator assignment from that meeting section.
9
Assignments Errors and How to Fix Them There are errors that prevent your courses and instructors from being evaluated during the
course evaluation period.
01: No instructor assignment
02: Instr assignment not linked to HRIS
03: No instructor UTORid
04: No Instructor UTORid and check appt start/end dates
05: No Instructor UTORid and instr appt in org does not exist
If any of these issues are present for a course instructor or coordinator, the person in that role
will not be able to access the course in any system that depends on full and accurate ROSI
assignments, including Portal (Blackboard), eMarks and Course Evaluations.
You may also encounter HRIS errors that on their own do not affect course evaluations.
06: Check employee appointment start/end dates
07: Employee appointment in the selected organization does not exist
01: No instructor assignment - If a course instructor assignment is missing, the course will
appear in the list with a message that there is no instructor assigned.
To add an instructor to a course meeting section that does not currently have one, click the
“Add” icon (green circle with a white plus sign) on the row for that meeting section. On the
screen that follows, you can add a new instructor assignment for this course meeting section.
To search the system for existing employee HRIS records, leave Employee Type set to “Human
Resources” and click Search, beside “Name*”.
10
In the window that pops up, type all or part of the employee’s first and/or last names, and click
“Search.” In the search results that appear, click on the name of the employee you wish to
assign as the course instructor.
You will return to the “manage instructor” screen, with the selected employee’s name now
appearing in the Name field.
Percentage Override: This field is optional and does not affect course evaluation operations. It
is used, when necessary, to indicate that instructors of a single meeting section are not sharing
teaching responsibility equally. If there are multiple instructors for this meeting section and they
are sharing the teaching responsibility equally, leave this field blank.
Add as Coordinator?: This field is optional and does not affect course evaluation operations. If
the instructor you are adding also needs to be added as the course coordinator, check the box.
Otherwise, leave it un-checked.
Evaluation Method: For courses that will be evaluated using the Online Course Evaluation
system, set this field to ONL, for “Online.”
11
Evaluation Timing: This also applies to courses that will be evaluated using the Online Course
Evaluation System. This example uses a half-year F course, so the Evaluation Timing field
should be populated with DEC, for “December.”
To save the new instructor assignment, click Save, then Confirm in the window that pops up.
You will be returned to the report, which shows the new instructor assignment. The “99:
Instructor OK” assignment message will be displayed If the employee record has no HRIS or
UTORid issues.
02 Instr assignment not linked to HRIS - If the instructor or coordinator assignment is not linked
to an employee’s HRIS record, it will be noted in the report, with an Employee Type of “OTH”
and a message in the Assignment Message column.
To change a non-HRIS (“OTH”) instructor assignment to one that is linked to an HRIS record
(“HR”), begin by clicking the Modify icon (a small pencil).
On the screen that follows, you will notice that the Employee Type field says “Other.”
Use the drop-down arrow to change this to “Human Resources.”
The name fields will switch into the Search link. From there, you can use the Search function to
look up the instructor and add them to the course, following the same steps as if you were doing
an “Add.”
12
03: No instructor UTORid - If an instructor or coordinator assignment is linked to an employee
whose UTORid either does not exist or is not properly linked to their record in HRIS, it will be
noted in the report with the message “No Instructor/Coordinator UTORid.”
You will need to do some troubleshooting to fix this error. It frequently means that the instructor
needs to activate their UTORid. Once the instructor notifies you that this has been completed,
you will need to delete the current instructor association, and add it again. Another option is to
connect with the Business Manager for your division or department to address any potential
issues.
04: No Instructor UTORid and check appt start/end dates & 05: No Instructor UTORid and instr
appt in org does not exist - You may also encounter errors 04 and 05. These are combinations
of error 03 and errors 06 and 07. These should be addressed to ensure the instructor can gain
access to the course evaluation system and other UTORid dependant services. Please review
error 03 for possible solutions.
The report also provides information about other issues involving the employee’s HRIS record
that may require a resolution. Error Codes 06 and 07, by themselves, are not critical errors.
These instructors will still be able to use Blackboard, and will be OK for setup in the Course
Evaluation system. As long as the instructor is assigned to the course via their HRIS record, and
the instructor has a valid UTORid, you will not need to take immediate action to resolve these
errors.
06: Check employee appointment start/end dates -This means that the employee’s HRIS
appointment in the selected organization is not active for all or part of the selected session.
07: Employee appointment in the selected organization does not exist - This means that the
employee does not currently have any HRIS appointment in the selected organization. This
type of information is not automatically updated in the instructor or coordinator assignment
record when a change is made to an employee’s HRIS record.
Error 1070’: EmpTyp ‘HR’ with no UTORid - Some users have experienced error 1070 when
trying to add instructors to a course for evaluation. This indicates an issue with the employee
record in HRIS.
If the HRIS record for the instructor was created within the last few days, wait another day or
two and then try re-adding the instructor in ROSI Express. It can take 2-3 days for a new
employee number in HRIS to link to the UTORid.
If the HRIS records were created more than a few days ago, the departmental business officer
should contact [email protected] to investigate why the employee number isn’t being
properly linked with the UTORid.
13
An error not listed here - If you encounter an error that does not correspond to one of these
codes (Errors 01-07), please contact the Course Evaluation Team at
[email protected] for assistance.
14
Changing a Course Instructor The process of changing an instructor assignment from one person to another is nearly identical
to fixing a 02 error.
Click the “Modify” icon to proceed, and follow the same steps as before. The only difference is
that the existing instructor’s name will already appear next to the search button. You can
continue as before, by clicking “Search,” looking up the new instructor’s information, and clicking
“Save” and “Confirm.” This instructor will replace the previous assignment.
15
Managing Course Coordinators You manage course coordinators the same way you manage course instructors in this report. If
you need to display and manage course coordinators, check the “Include Missing Coordinators”
box on the input screen and re-run the report.
When the report is run with this option selected, a “coordinator” row will be included for each
course meeting section.
You can then use the options to make any necessary updates to your course coordinators.
Again, individuals in the coordinator role are not evaluated by the Course Evaluation
system.
16
Showing Only Missing or Problematic Assignments On the report input screen, you have the option to check a box to “Show only courses with
missing instructors and problematic instructors/coordinators.” When you run the report with this
box checked, any instructor/coordinator assignment with a message of 99: Instructor OK or 99:
Coordinator OK will be excluded from the output.
When you run the report with this box checked, assignments will disappear from the report as
you fix them. This happens because once they are fixed, they are no longer problematic.
This option can be helpful in identifying problems with course-instructor assignments, however
for course evaluation purposes, be sure to run the report without this option selected, so that
you can verify that all course-instructor assignments are correct.
17
Exporting the Report to Excel After running the report, you have the option to export the data to be opened up in an Excel
spreadsheet. Scroll to the bottom of any Output screen to find the “Export Options” box.
To open the report as an Excel spreadsheet, click the “Excel” link.
The Excel report contains some information that is not displayed in the on-screen version.
Additional columns you may find useful:
Current Registration Quantity – the number of students currently enrolled in the
course
Course Evaluation Settings (these apply only to Instructor rows):
o Evaluate YN (Global Evaluation Setting): flag that sets courses as “Evaluated
(Y)” or “Never-Evaluated (N)”
o Evaluation Method: Online (ONL), Paper (PPR), or Not Evaluated (NOT)
o Evaluation Timing: Defaults to the last month of the course session
Faculty Member Appointment Start and End Dates
18
The Confirm Courses and Instructors for Evaluation Report
This report is used to view and edit the following settings:
Evaluation Method
Evaluation Timing
This report also identifies those courses that are set to not be evaluated. The two primary
reasons a course might be set up to “not evaluate” are:
1. The Course is Never Going to Be Evaluated
As per divisional guidelines, certain courses, because of their pedagogical configuration,
are not appropriate for evaluation in the course evaluation framework:
Independent Studies
Reading Courses
Research Seminars
Program Milestone “Marker” courses
These kinds of “Never-Evaluated” courses are designated as such in ROSI using the Global
Evaluation Setting field. The courses are always set to NOT be evaluated, saving you the step
of making this selection every year.
If you have courses that should be set in ROSI as “Never Evaluated,” please contact the Course
Evaluations Team at [email protected].
2. There is Something Wrong with the Course Instructor Assignment
If any of the following issues are present for a course instructor assignment, the report
will prevent you from making any selections and the Evaluation Method will default to
“NOT”:
01: No Instructor Assignment - There is no course instructor assignment
02: Instr Assignment Not Linked to HRIS - The course instructor assignment is
not linked to the professor’s HRIS record
03: No Instructor UTORid - The professor does not have an active UTORid
04: No Instructor UTORid and check appt start/end dates
05: No Instructor UTORid and instr appt in org does not exist
You will recall that these error messages appear within the Scheduled Courses with
Instructor/Coordinator Diagnostics report. You will need to return to this report to resolve these
issues.
19
The Input Screen 1. Enter the session code for the report.
2. Next, enter your Primary Organization Code.
3. If applicable, enter your Secondary Organization Code AND/OR Co-Secondary
Organization Code (graduate programs only). This is the ROSI code for your
unit/department.
4. Click Submit.
The option to Show only courses/instructors that are NOT set up for evaluation allows you to
review those courses that are currently not going to be evaluated. Before completing this
report, please use this option to review your courses set to NOT to confirm that there are
no courses incorrectly assigned this value. If a course is assigned the Evaluation Method
of NOT it will not be evaluated.
20
The Report The report will display scheduled courses with assigned instructors for your unit in the session
code you entered.
The current Evaluation Method and Evaluation Timing values are displayed for each course-
instructor assignment, describing how and when the instructor and course will be evaluated.
The Assignment Message column lets you know if there are any outstanding issues with the
instructor assignment. If there are no outstanding issues, the message “99: Instructor OK” will
be displayed.
As mentioned earlier, if the Assignment Message column displays errors 1 through 5, you will
not be able to update the Evaluation Method and Timing. You will need to resolve these issues
using the Scheduled Courses with Instructor/Coordinator Diagnostics report before
proceeding.
There are also two columns entitled Eval Form and Global Eval Y/N. If the Eval Form field
reads “NONE” or the Global Eval Y/N field reads “N”, this course/instructor is excluded from
evaluation and the Action column will read ‘No eval’. These settings are typically requested
when a course should never be evaluated.
The values for Evaluation Method and Evaluation Timing are:
Evaluation Method:
ONL – The course should be evaluated using the online course evaluation system.
PPR – The course should be evaluated using a paper based evaluation form. Note:
only pre-tenure/pre-promotion continuing appointment faculty hired prior to July 2012
are eligible to use paper based evaluation forms.
NOT – The course should not be evaluated. Courses with this setting are excluded
from evaluation.
Evaluation Timing:
DEC/APR – December or April, during the regular school year
JUN/AUG – June or August, during the Summer term
21
OTH – Only for courses that fall outside the regularly scheduled term
[blank] – the option without text must be chosen when you set the Evaluation Method to
“NOT”
To change either the Evaluation Method or Timing value, click the “Edit” button. Both the
Evaluation Method and Timing values will become drop down menus. After making changes,
click Save. A pop-up window will ask you to confirm the change. Click Confirm.
For courses with no outstanding critical issues, verify that the default values are correct.
Courses with zero students enrolled Courses with no students enrolled are sometimes maintained in ROSI for administrative
purposes. These courses should not be configured for evaluation, as courses with no students
assigned cause errors in the course evaluation system. These errors can lead to the delay of
course evaluation reports.
For these courses and instructors, change the Evaluation Method to “NOT”, and the Evaluation
Timing to [blank].
22
If a change occurs after the ROSI Express Data Verification
Deadline…
Changes that happen after the ROSI Express Data Verification Deadline are not captured by
the course evaluation system. If changes occur after the deadline, you must alert the Course
Evaluation Team at [email protected].
Be sure to include in your e-mail:
Your division (ie. Arts & Science, UTM, Engineering, etc.)
Your department/unit (if applicable)
The course code affected
The name of the instructor affected
23
Data Verification: The Essential Steps
Now that you’ve reviewed the entire data verification process from beginning to end, take the
time to review again these essential steps. If you do not complete the following tasks, your
courses are at risk of being improperly evaluated.
1. Use the Scheduled Courses with Instructor/Coordinator Diagnostics to ensure
instructors are assigned to the correct courses. Resolve any course-instructor
assignments showing errors 01 through 05. Contact the Course Evaluations Team
should you encounter any difficulty.
2. Use the Confirm Courses and Instructors for Evaluation report to confirm the
Evaluation Method and Evaluation Timing for each course instructor assignment.
3. Check your data for the following common issues:
a. “Global Eval YN”/Courses set to “NOT” be evaluated: If a course has a value of N
in the Global Eval Y/N column in the Confirm Courses and Instructors for
Evaluation report or an Evaluation Method of “NOT” this course will NEVER
be evaluated. Correct any errors in these settings.
b. Evaluation Method – PPR: Instructors are eligible to use paper based evaluation
forms only if they are pre-tenure/pre-promotion continuing appointment faculty,
hired before July 1, 2012. If they do not meet this criteria, change the Evaluation
Method to ONL.
c. Courses with more than four instructors – Any single lecture section with >4
instructors associated is not eligible to be evaluated by the online. Set the
Evaluation Method for all instructors to NOT for courses with >4 instructors.
d. Courses with zero students enrolled – Set any courses with zero students
enrolled to have an Evaluation Method of “NOT”, and an Evaluation Timing of
[blank].
4. Return to the Scheduled Courses with Instructor/Coordinator Diagnostics report
and generate an Excel download of this report for your unit. Review this report for
potential errors, and keep a copy for your records.
24
Need more support? Contact us!
If you need additional support while completing these reports, the Course Evaluations Team is
happy to help. Please contact us at [email protected]. Your question will be
addressed within two business days.
This version updated August 8, 2018.