25
Course Evaluations and ROSI: Preparing courses for evaluation A Step-By-Step Guide Centre for Teaching Support & Innovation 4 th Floor, Robarts Library, 130 St. George Street Toronto, Ontario [email protected]

Course Evaluations and ROSI: Preparing courses for evaluation

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Course Evaluations and ROSI: Preparing courses for evaluation

Course Evaluations and ROSI: Preparing courses for evaluation A Step-By-Step Guide

Centre for Teaching Support & Innovation

4th Floor, Robarts Library, 130 St. George Street

Toronto, Ontario

[email protected]

Page 2: Course Evaluations and ROSI: Preparing courses for evaluation

1

Table of Contents

About Course Evaluations and this Guide .................................................................................. 2

When to Complete Data Verification .......................................................................................... 3

Verifying Your Course Diagnostics for the Term ......................................................................... 4

Data Verification from Beginning to End ..................................................................................... 5

Logging in to ROSI Express ................................................................................................... 5

Accessing the right reports ..................................................................................................... 6

The Scheduled Courses with Instructor/Coordinator Diagnostics Report ................................ 7

The Input Screen ................................................................................................................ 7

The Report .......................................................................................................................... 8

Assignments Errors and How to Fix Them .......................................................................... 9

01: No instructor assignment ........................................................................................... 9

02 Instr assignment not linked to HRIS ...........................................................................11

03: No instructor UTORid ...............................................................................................12

04: No Instructor UTORid and check appt start/end dates & 05: No Instructor UTORid

and instr appt in org does not exist .................................................................................12

06: Check employee appointment start/end dates ..........................................................12

07: Employee appointment in the selected organization does not exist ..........................12

Error 1070’: EmpTyp ‘HR’ with no UTORid .....................................................................12

An error not listed here ...................................................................................................13

Changing a Course Instructor ............................................................................................14

Managing Course Coordinators .........................................................................................15

Showing Only Missing or Problematic Assignments ...........................................................16

Exporting the Report to Excel .............................................................................................17

The Confirm Courses and Instructors for Evaluation Report ..................................................18

The Input Screen ...............................................................................................................19

The Report .........................................................................................................................20

Courses with zero students enrolled ..................................................................................21

If a change occurs after the ROSI Express Data Verification Deadline… ..................................22

Data Verification: The Essential Steps ......................................................................................23

Need more support? Contact us! ...........................................................................................24

Page 3: Course Evaluations and ROSI: Preparing courses for evaluation

2

About Course Evaluations and this Guide

The Course Evaluation Framework plays an important role at the University of Toronto by

providing students the opportunity to comment on their learning experience in their courses. In

turn, evaluations provide instructors and academic administrators with student feedback. The

integrity of course evaluation data is essential to the Framework’s quality assurance principles

and its successful operation.

It is the responsibility of each division/department to ensure their course-instructor-student

information in ROSI is correct by a deadline determined by the Course Evaluations (CE) Team

so that accurate information is reflected in Blue, the online course evaluation system. Blue

extracts information from two campus-wide information systems, ROSI (Repository of Student

Information) and HRIS (Human Resources Information System).

Incorrect information in ROSI can lead to errors in Blue, and incorrect evaluations being issued

to instructors and students. This can negatively impact on an instructor’s promotion, tenure, and

review process. Errors also have an effect on the release of course evaluation reports.

This guide is intended to support individuals responsible for verifying course-instructor-student

information in ROSI by providing them with a step-by-step walkthrough of the process, as well

as the tools to troubleshoot potential issues.

Page 4: Course Evaluations and ROSI: Preparing courses for evaluation

3

When to Complete Data Verification

At the beginning of each academic term – Fall, Winter, and Summer – a member of the CE

Team will provide you with a ROSI Express-Course Evaluations Data Verification Deadline.

For accurate information about instructors and courses to be included in course evaluations, the

two ROSI Express Reports Scheduled Courses with Instructor/Coordinator Diagnostics and

Confirm Courses and Instructors for Evaluation must be reviewed, and any necessary changes

made by that date.

Important Note: If there is a change to any course-instructor assignment information after the

ROSI Express deadline, please contact the CE Team ([email protected])

directly. Changes made in ROSI after the deadline are NOT captured by the Course

Evaluations System and will need to be adjusted manually by the Course Evaluations Team.

Page 5: Course Evaluations and ROSI: Preparing courses for evaluation

4

Verifying Your Course Diagnostics for the Term

The following steps are crucial and must be completed by your ROSI Express Deadline.

1. Using the Scheduled Courses with Instructor/Coordinator Diagnostics report in ROSI

Express, ensure instructors are assigned to the correct courses. Correct any course-

instructor assignments showing errors 01 through 05, as courses and instructors with

these errors will not be evaluated.

2. Using the Confirm Courses and Instructors for Evaluation report in ROSI Express,

review the Evaluation Method and Evaluation Timing for each course-instructor

assignment. If the Evaluation Method is set to “NOT” this course will not be evaluated.

3. Review your data for the following common issues:

a. “Global Eval Y/N”/Courses set to “NOT” be evaluated: On the input screen for the

Confirm Courses and Instructors for Evaluation report, there is an option that

allows you to “Show only Courses/Instructors that are NOT set up for evaluation.”

Before completing your review, make use of this option to ensure all courses with

a setting of NOT should indeed NOT be evaluated. If the course has a value of N

in the Global Eval Y/N column, this course will NEVER be evaluated. Should

there be an error in this value, contact the Course Evaluations Team, as you will

not be able to change this setting yourself.

b. Evaluation Method – PPR: Instructors are eligible to use paper based evaluation

forms only if they are pre-tenure/pre-promotion continuing appointment faculty,

hired before July 1, 2012. If instructors have requested paper evaluation forms

but do not meet the above criteria, their Evaluation Method must be set to ONL

for online.

c. Courses with more than four instructors – A maximum of four instructors

teaching a single lecture and meeting section (i.e. four instructors all assigned to

ANT100Y-Y-LEC0101) can be evaluated by the online evaluation system. Any

course with more than four instructors requiring evaluation is not currently

evaluated by the online evaluation system. Please set the Evaluation Method for

all instructors to NOT. Contact the Course Evaluations Service to seek guidance

with respect to an alternate evaluation method. Otherwise, continue historical

evaluation practices within your Division.

d. Courses with 0 students enrolled – Courses with no students enrolled should not

be included for evaluation. For these courses, be sure to change the Evaluation

Method to NOT and the Evaluation Timing to [blank].

4. Return to the Scheduled Courses with Instructor/Coordinator Diagnostics report

and generate an Excel download of this report for your unit. Review this report for

potential errors, and keep a copy for your records.

Page 6: Course Evaluations and ROSI: Preparing courses for evaluation

5

Data Verification from Beginning to End

Logging in to ROSI Express To log-in to ROSI Express you must have an eToken. Once your eToken is connected to your

computer, go to: https://www.sis.utoronto.ca/rosi-express/, and follow log-in instructions.

Should you require access to ROSI Express please contact ROSI Help at [email protected]

Page 7: Course Evaluations and ROSI: Preparing courses for evaluation

6

Accessing the right reports Both the Scheduled Courses with Instructor/Coordinator Diagnostics and Confirm Courses and

Instructors for Evaluation reports can be found on the Reports page, which can be accessed on

the menu on the left-hand side of the ROSI Express main page.

Page 8: Course Evaluations and ROSI: Preparing courses for evaluation

7

The Scheduled Courses with Instructor/Coordinator Diagnostics Report The Scheduled Courses with Instructor/Coordinator Diagnostics report allows you to verify the

list of your scheduled courses and their current instructor/coordinator assignments, and also to

make changes that reflect your current course and instructor assignments directly on the

report itself.

Using this report, you can:

View your scheduled courses for a session or academic year

View the instructors and coordinators assigned to these courses

Identify if there are any issues with these assignments or the HRIS records of the

assigned faculty members

Make necessary updates or changes to your instructor and coordinator assignments

The Input Screen 1. Enter the session code for the report (e.g. 20179).

2. Enter your Primary Organization Code.

3. If applicable, enter your Secondary Organization Code AND/OR Co-Secondary

Organization Code. This is the ROSI code for your unit/department.

4. Click Submit.

By default, the report does not include a row to indicate which courses are missing coordinator

records, because not all divisions make use of the Course Coordinator role. If you wish to

include missing coordinator rows in your report, check the Include Missing Coordinator box.

Individuals associated with the Coordinator role in ROSI are not evaluated by the Course

Evaluation system. If the individual is also associated with the instructor role, then they can be

configured for evaluation.

Page 9: Course Evaluations and ROSI: Preparing courses for evaluation

8

The Report The Scheduled Courses with Instructor/Coordinator Diagnostics report displays scheduled

courses with instructor/coordinator assignments for the session code and organization codes

you entered. Each course meeting section, with its respective instructor or coordinator

assignments, will appear as a separate row in the report. Please note that the report can only

identify data that is missing or problematic. It is unable to detect if the incorrect person is

assigned as a course instructor or coordinator.

1. The “Assignment Message” column provides information about each instructor or

coordinator assignment. If there are no detectable issues with the assignment, the

messages “99: Instructor OK” or “99: Coordinator OK” will be displayed.

2. On the right-hand side of the report, there are icons representing each of the actions you

can take to manage your course-instructor and coordinator assignments.

a. ADD: Use this option to add an instructor or coordinator assignment to this

course meeting section. You can add an instructor or coordinator to a course

that has none, or add additional instructors or coordinators when an assignment

already exists.

b. MODify: You can use this option to edit the existing instructor or coordinator

assignment, e.g. if you need to change the course instructor from one professor

to another.

c. DELete: This option deletes the instructor or coordinator assignment. The Delete

option does not delete the course meeting section. It only removes the instructor

or coordinator assignment from that meeting section.

Page 10: Course Evaluations and ROSI: Preparing courses for evaluation

9

Assignments Errors and How to Fix Them There are errors that prevent your courses and instructors from being evaluated during the

course evaluation period.

01: No instructor assignment

02: Instr assignment not linked to HRIS

03: No instructor UTORid

04: No Instructor UTORid and check appt start/end dates

05: No Instructor UTORid and instr appt in org does not exist

If any of these issues are present for a course instructor or coordinator, the person in that role

will not be able to access the course in any system that depends on full and accurate ROSI

assignments, including Portal (Blackboard), eMarks and Course Evaluations.

You may also encounter HRIS errors that on their own do not affect course evaluations.

06: Check employee appointment start/end dates

07: Employee appointment in the selected organization does not exist

01: No instructor assignment - If a course instructor assignment is missing, the course will

appear in the list with a message that there is no instructor assigned.

To add an instructor to a course meeting section that does not currently have one, click the

“Add” icon (green circle with a white plus sign) on the row for that meeting section. On the

screen that follows, you can add a new instructor assignment for this course meeting section.

To search the system for existing employee HRIS records, leave Employee Type set to “Human

Resources” and click Search, beside “Name*”.

Page 11: Course Evaluations and ROSI: Preparing courses for evaluation

10

In the window that pops up, type all or part of the employee’s first and/or last names, and click

“Search.” In the search results that appear, click on the name of the employee you wish to

assign as the course instructor.

You will return to the “manage instructor” screen, with the selected employee’s name now

appearing in the Name field.

Percentage Override: This field is optional and does not affect course evaluation operations. It

is used, when necessary, to indicate that instructors of a single meeting section are not sharing

teaching responsibility equally. If there are multiple instructors for this meeting section and they

are sharing the teaching responsibility equally, leave this field blank.

Add as Coordinator?: This field is optional and does not affect course evaluation operations. If

the instructor you are adding also needs to be added as the course coordinator, check the box.

Otherwise, leave it un-checked.

Evaluation Method: For courses that will be evaluated using the Online Course Evaluation

system, set this field to ONL, for “Online.”

Page 12: Course Evaluations and ROSI: Preparing courses for evaluation

11

Evaluation Timing: This also applies to courses that will be evaluated using the Online Course

Evaluation System. This example uses a half-year F course, so the Evaluation Timing field

should be populated with DEC, for “December.”

To save the new instructor assignment, click Save, then Confirm in the window that pops up.

You will be returned to the report, which shows the new instructor assignment. The “99:

Instructor OK” assignment message will be displayed If the employee record has no HRIS or

UTORid issues.

02 Instr assignment not linked to HRIS - If the instructor or coordinator assignment is not linked

to an employee’s HRIS record, it will be noted in the report, with an Employee Type of “OTH”

and a message in the Assignment Message column.

To change a non-HRIS (“OTH”) instructor assignment to one that is linked to an HRIS record

(“HR”), begin by clicking the Modify icon (a small pencil).

On the screen that follows, you will notice that the Employee Type field says “Other.”

Use the drop-down arrow to change this to “Human Resources.”

The name fields will switch into the Search link. From there, you can use the Search function to

look up the instructor and add them to the course, following the same steps as if you were doing

an “Add.”

Page 13: Course Evaluations and ROSI: Preparing courses for evaluation

12

03: No instructor UTORid - If an instructor or coordinator assignment is linked to an employee

whose UTORid either does not exist or is not properly linked to their record in HRIS, it will be

noted in the report with the message “No Instructor/Coordinator UTORid.”

You will need to do some troubleshooting to fix this error. It frequently means that the instructor

needs to activate their UTORid. Once the instructor notifies you that this has been completed,

you will need to delete the current instructor association, and add it again. Another option is to

connect with the Business Manager for your division or department to address any potential

issues.

04: No Instructor UTORid and check appt start/end dates & 05: No Instructor UTORid and instr

appt in org does not exist - You may also encounter errors 04 and 05. These are combinations

of error 03 and errors 06 and 07. These should be addressed to ensure the instructor can gain

access to the course evaluation system and other UTORid dependant services. Please review

error 03 for possible solutions.

The report also provides information about other issues involving the employee’s HRIS record

that may require a resolution. Error Codes 06 and 07, by themselves, are not critical errors.

These instructors will still be able to use Blackboard, and will be OK for setup in the Course

Evaluation system. As long as the instructor is assigned to the course via their HRIS record, and

the instructor has a valid UTORid, you will not need to take immediate action to resolve these

errors.

06: Check employee appointment start/end dates -This means that the employee’s HRIS

appointment in the selected organization is not active for all or part of the selected session.

07: Employee appointment in the selected organization does not exist - This means that the

employee does not currently have any HRIS appointment in the selected organization. This

type of information is not automatically updated in the instructor or coordinator assignment

record when a change is made to an employee’s HRIS record.

Error 1070’: EmpTyp ‘HR’ with no UTORid - Some users have experienced error 1070 when

trying to add instructors to a course for evaluation. This indicates an issue with the employee

record in HRIS.

If the HRIS record for the instructor was created within the last few days, wait another day or

two and then try re-adding the instructor in ROSI Express. It can take 2-3 days for a new

employee number in HRIS to link to the UTORid.

If the HRIS records were created more than a few days ago, the departmental business officer

should contact [email protected] to investigate why the employee number isn’t being

properly linked with the UTORid.

Page 14: Course Evaluations and ROSI: Preparing courses for evaluation

13

An error not listed here - If you encounter an error that does not correspond to one of these

codes (Errors 01-07), please contact the Course Evaluation Team at

[email protected] for assistance.

Page 15: Course Evaluations and ROSI: Preparing courses for evaluation

14

Changing a Course Instructor The process of changing an instructor assignment from one person to another is nearly identical

to fixing a 02 error.

Click the “Modify” icon to proceed, and follow the same steps as before. The only difference is

that the existing instructor’s name will already appear next to the search button. You can

continue as before, by clicking “Search,” looking up the new instructor’s information, and clicking

“Save” and “Confirm.” This instructor will replace the previous assignment.

Page 16: Course Evaluations and ROSI: Preparing courses for evaluation

15

Managing Course Coordinators You manage course coordinators the same way you manage course instructors in this report. If

you need to display and manage course coordinators, check the “Include Missing Coordinators”

box on the input screen and re-run the report.

When the report is run with this option selected, a “coordinator” row will be included for each

course meeting section.

You can then use the options to make any necessary updates to your course coordinators.

Again, individuals in the coordinator role are not evaluated by the Course Evaluation

system.

Page 17: Course Evaluations and ROSI: Preparing courses for evaluation

16

Showing Only Missing or Problematic Assignments On the report input screen, you have the option to check a box to “Show only courses with

missing instructors and problematic instructors/coordinators.” When you run the report with this

box checked, any instructor/coordinator assignment with a message of 99: Instructor OK or 99:

Coordinator OK will be excluded from the output.

When you run the report with this box checked, assignments will disappear from the report as

you fix them. This happens because once they are fixed, they are no longer problematic.

This option can be helpful in identifying problems with course-instructor assignments, however

for course evaluation purposes, be sure to run the report without this option selected, so that

you can verify that all course-instructor assignments are correct.

Page 18: Course Evaluations and ROSI: Preparing courses for evaluation

17

Exporting the Report to Excel After running the report, you have the option to export the data to be opened up in an Excel

spreadsheet. Scroll to the bottom of any Output screen to find the “Export Options” box.

To open the report as an Excel spreadsheet, click the “Excel” link.

The Excel report contains some information that is not displayed in the on-screen version.

Additional columns you may find useful:

Current Registration Quantity – the number of students currently enrolled in the

course

Course Evaluation Settings (these apply only to Instructor rows):

o Evaluate YN (Global Evaluation Setting): flag that sets courses as “Evaluated

(Y)” or “Never-Evaluated (N)”

o Evaluation Method: Online (ONL), Paper (PPR), or Not Evaluated (NOT)

o Evaluation Timing: Defaults to the last month of the course session

Faculty Member Appointment Start and End Dates

Page 19: Course Evaluations and ROSI: Preparing courses for evaluation

18

The Confirm Courses and Instructors for Evaluation Report

This report is used to view and edit the following settings:

Evaluation Method

Evaluation Timing

This report also identifies those courses that are set to not be evaluated. The two primary

reasons a course might be set up to “not evaluate” are:

1. The Course is Never Going to Be Evaluated

As per divisional guidelines, certain courses, because of their pedagogical configuration,

are not appropriate for evaluation in the course evaluation framework:

Independent Studies

Reading Courses

Research Seminars

Program Milestone “Marker” courses

These kinds of “Never-Evaluated” courses are designated as such in ROSI using the Global

Evaluation Setting field. The courses are always set to NOT be evaluated, saving you the step

of making this selection every year.

If you have courses that should be set in ROSI as “Never Evaluated,” please contact the Course

Evaluations Team at [email protected].

2. There is Something Wrong with the Course Instructor Assignment

If any of the following issues are present for a course instructor assignment, the report

will prevent you from making any selections and the Evaluation Method will default to

“NOT”:

01: No Instructor Assignment - There is no course instructor assignment

02: Instr Assignment Not Linked to HRIS - The course instructor assignment is

not linked to the professor’s HRIS record

03: No Instructor UTORid - The professor does not have an active UTORid

04: No Instructor UTORid and check appt start/end dates

05: No Instructor UTORid and instr appt in org does not exist

You will recall that these error messages appear within the Scheduled Courses with

Instructor/Coordinator Diagnostics report. You will need to return to this report to resolve these

issues.

Page 20: Course Evaluations and ROSI: Preparing courses for evaluation

19

The Input Screen 1. Enter the session code for the report.

2. Next, enter your Primary Organization Code.

3. If applicable, enter your Secondary Organization Code AND/OR Co-Secondary

Organization Code (graduate programs only). This is the ROSI code for your

unit/department.

4. Click Submit.

The option to Show only courses/instructors that are NOT set up for evaluation allows you to

review those courses that are currently not going to be evaluated. Before completing this

report, please use this option to review your courses set to NOT to confirm that there are

no courses incorrectly assigned this value. If a course is assigned the Evaluation Method

of NOT it will not be evaluated.

Page 21: Course Evaluations and ROSI: Preparing courses for evaluation

20

The Report The report will display scheduled courses with assigned instructors for your unit in the session

code you entered.

The current Evaluation Method and Evaluation Timing values are displayed for each course-

instructor assignment, describing how and when the instructor and course will be evaluated.

The Assignment Message column lets you know if there are any outstanding issues with the

instructor assignment. If there are no outstanding issues, the message “99: Instructor OK” will

be displayed.

As mentioned earlier, if the Assignment Message column displays errors 1 through 5, you will

not be able to update the Evaluation Method and Timing. You will need to resolve these issues

using the Scheduled Courses with Instructor/Coordinator Diagnostics report before

proceeding.

There are also two columns entitled Eval Form and Global Eval Y/N. If the Eval Form field

reads “NONE” or the Global Eval Y/N field reads “N”, this course/instructor is excluded from

evaluation and the Action column will read ‘No eval’. These settings are typically requested

when a course should never be evaluated.

The values for Evaluation Method and Evaluation Timing are:

Evaluation Method:

ONL – The course should be evaluated using the online course evaluation system.

PPR – The course should be evaluated using a paper based evaluation form. Note:

only pre-tenure/pre-promotion continuing appointment faculty hired prior to July 2012

are eligible to use paper based evaluation forms.

NOT – The course should not be evaluated. Courses with this setting are excluded

from evaluation.

Evaluation Timing:

DEC/APR – December or April, during the regular school year

JUN/AUG – June or August, during the Summer term

Page 22: Course Evaluations and ROSI: Preparing courses for evaluation

21

OTH – Only for courses that fall outside the regularly scheduled term

[blank] – the option without text must be chosen when you set the Evaluation Method to

“NOT”

To change either the Evaluation Method or Timing value, click the “Edit” button. Both the

Evaluation Method and Timing values will become drop down menus. After making changes,

click Save. A pop-up window will ask you to confirm the change. Click Confirm.

For courses with no outstanding critical issues, verify that the default values are correct.

Courses with zero students enrolled Courses with no students enrolled are sometimes maintained in ROSI for administrative

purposes. These courses should not be configured for evaluation, as courses with no students

assigned cause errors in the course evaluation system. These errors can lead to the delay of

course evaluation reports.

For these courses and instructors, change the Evaluation Method to “NOT”, and the Evaluation

Timing to [blank].

Page 23: Course Evaluations and ROSI: Preparing courses for evaluation

22

If a change occurs after the ROSI Express Data Verification

Deadline…

Changes that happen after the ROSI Express Data Verification Deadline are not captured by

the course evaluation system. If changes occur after the deadline, you must alert the Course

Evaluation Team at [email protected].

Be sure to include in your e-mail:

Your division (ie. Arts & Science, UTM, Engineering, etc.)

Your department/unit (if applicable)

The course code affected

The name of the instructor affected

Page 24: Course Evaluations and ROSI: Preparing courses for evaluation

23

Data Verification: The Essential Steps

Now that you’ve reviewed the entire data verification process from beginning to end, take the

time to review again these essential steps. If you do not complete the following tasks, your

courses are at risk of being improperly evaluated.

1. Use the Scheduled Courses with Instructor/Coordinator Diagnostics to ensure

instructors are assigned to the correct courses. Resolve any course-instructor

assignments showing errors 01 through 05. Contact the Course Evaluations Team

should you encounter any difficulty.

2. Use the Confirm Courses and Instructors for Evaluation report to confirm the

Evaluation Method and Evaluation Timing for each course instructor assignment.

3. Check your data for the following common issues:

a. “Global Eval YN”/Courses set to “NOT” be evaluated: If a course has a value of N

in the Global Eval Y/N column in the Confirm Courses and Instructors for

Evaluation report or an Evaluation Method of “NOT” this course will NEVER

be evaluated. Correct any errors in these settings.

b. Evaluation Method – PPR: Instructors are eligible to use paper based evaluation

forms only if they are pre-tenure/pre-promotion continuing appointment faculty,

hired before July 1, 2012. If they do not meet this criteria, change the Evaluation

Method to ONL.

c. Courses with more than four instructors – Any single lecture section with >4

instructors associated is not eligible to be evaluated by the online. Set the

Evaluation Method for all instructors to NOT for courses with >4 instructors.

d. Courses with zero students enrolled – Set any courses with zero students

enrolled to have an Evaluation Method of “NOT”, and an Evaluation Timing of

[blank].

4. Return to the Scheduled Courses with Instructor/Coordinator Diagnostics report

and generate an Excel download of this report for your unit. Review this report for

potential errors, and keep a copy for your records.

Page 25: Course Evaluations and ROSI: Preparing courses for evaluation

24

Need more support? Contact us!

If you need additional support while completing these reports, the Course Evaluations Team is

happy to help. Please contact us at [email protected]. Your question will be

addressed within two business days.

This version updated August 8, 2018.