Embed Size (px)
Citation preview
UTM Fall Faculty Workshop: August 16, 2018
Faculty Evaluation ProcessMorning Session 8:30am-11:30am, Campus-Wide
Two big topics:
1. Faculty Evaluation Process--Overview and Preview of Coming Attractions
2. Bylaws--Especially changes mandated by the UT Board of Trustees
Faculty Evaluation ProcessOVERVIEW:
● FEP Process History, Overview, and Future● Who is involved
● Campus Expectations● Student Ratings
● Resources● Digital Measures
● Questions
Why are we doing this?
● Inconsistencies across campus● Concerns about the fairness of evaluations
● Board of Trustees displeasure with evaluation systems throughout our state
What was accomplished in 2017-18 (Year One)?
● Hired expert Dr. Raoul Arreola
● Arreola presented workshop on UTM Campus
● Departments selected at least two faculty experts for additional training (these became the College Committee Representatives)
● Deans served as College Committee Chairs and selected two faculty experts from each College Committee to serve on the Campus Wide Committee (cFEC)
● Departments independently proposed definitions, weights, data sources, etc.
● Merged by College Committees and college results merged by cFEC.
● Selection of Beta Departments to serve as test group for upcoming year (min of 1 from each college) - We have more than that with 10 volunteer departments!
Others involved with cFEC● Rich Helgeson, Interim Provost
○ Non-Voting Chair, Year One
● Julie Hill, Chair - Department of Music○ Non-Voting Coordinator
● Chris Caldwell, Faculty Senate President○ Non-Voting ex-officio member
● Stephanie Kolitsch, SACS Accreditation and Digital Measures Coordinator○ Added by the cFEC Chair, Voting member
● Renee LaFleur, Faculty Senate President-elect○ Non-Voting ex-officio member
This Past Summer’s Fun!● Website
● Guidebook “Finalization”
● Timeline/Calendar
● Plan for integration with Digital Measures, campus visit/weekly calls
● Development of Forms/Rubrics
● Preparation of materials needed for workshop today
Who are the Beta Groups? Meet the brave tenFamily and Consumer Sciences Health and Human Performance
Biological Sciences Computer Science
Engineering Math and Statistics
Music Accounting, Finance, Economics, and Political Science
Management, Marketing, and Information SystemsPaul Meek Library
What to expect in 2018-19 (Year Two)
● Deans/Beta Departments (more details this afternoon)○ System Implementation
○ Determining the roles and future of the college committees
○ Feedback to cFEC and College Committees for edits/changes
○ Negotiations between the departments and their colleges on ranges, requirements, weight ranges…
○ Forms, Scales, Sources for Evaluation, and restrictive ranges for each component will be created■ Separate forms should be generated for term appointments and adjuncts
○ Determine what changes will be required in department/college Bylaws and the Faculty Handbook
○ Participation in Faculty Evaluation workshop this afternoon and Digital Measures workshop(s) beginning in November
○ Pilot testing with Digital Measures
What to expect in 2018-19 (Year Two)?
● The Campus-Wide Committee (cFEC) will
○ Digital Measures involvement for this year is still TBD, but most likely the minimum of uploading a c.v.
○ Submit Form Declaring Role Weights at end of this academic year to be used in FEP for upcoming year
○ Will follow current “old” system of evaluation one more year which will be the final year
What will happen in 2019-2020 (Year 3) and beyond?● Nothing has been solidified - possibilities for cFEC agenda this year
○ FEP integration with Tenure and Promotion
■ The cFEC does NOT support a system in which tenure and promotion
decisions are solely based on the annual composite rating.
■ The cFEC DOES support a process by which the yearly annual composite
rating informs decisions on tenure and promotion.
○ Determine how FEP should impact Merit Pay
○ Continually adjust and improve
● Dr. Phil Cavalier, working with cFEC, will determine the agenda for each year
● The campus will continue to be informed at every step as we work to build community and consensus
Overview of Beta Process for Upcoming Year1.Establish the Faculty Role Model2.Determine Role Parameter Values3.Define Each Role’s Components4.Establish the Role Component Weights5.Determine Appropriate Sources6.Establish Source Weights7.Determine How to Gather Information8.Design or Select Appropriate Forms*Conduct both new and old process.Compare/Feedback/Edit FEP.
Institutional Requirements v/s College/Department DecisionsInstitutional Requirements - Set Items Campus-Wide
Weights of Components of Teaching Ranges for three roles
Teaching (50% to 90%)A. Instructional design (30%)B. Instructional delivery (30%)C. Instructional assessment (25%)D. Course management (15%)
Scholarship (5% to 40%)A. Proficiency (0% to 100%)B. Discovery/creativity (0% to 100%)C. Dissemination (0% to 100%)D. Translation (0% to 100%)
Service (5% to 40%)A. To the institution (0% to 100%)B. To the profession (0% to 100%)C. To the community (0% to 100%)D. To the students (0% to 100%)
College/Department Decisions
● Determine the role of advising in the yearly evaluation process
● Colleges and Departments may enact smaller ranges within the set campus ranges for their units
● This year only - set deadline for submission of Declaration of Role Weights Form
● Chairs should decide the timeline for their departments to conduct face to face meetings with faculty for annual review.
● Role Component Weights for Service and Scholarship
○ Fixed Amounts or Faculty Flexibility?
○ How much value or weight should be placed on the information provided by selected sources?
● Determine sources for each role and their weight
● Determine impact sources have in overall evaluation (source impact weights)
● Build the Source Impact Matrix for each role
● Determine how to gather information
○ Forms, Checklist, Questionnaire, or Other?
○ Build Data Gathering Matrix for each role
● Design/Select Appropriate Forms with numerically common scale
● Adaptation of process needed for Term Appointments (and Adjuncts)
MUST DO at either College or Department Level:
Other Items to Consider:
Preview of Intended Results of FEP ○ TWO outcomes
■ A numerical score between 1.00 and 4.00
■ One of four Board of Trustees categories (similar to the current system)
○ At this time, there is no fixed translation between the numerical score and the
Board of Trustees category. The cFEC will collect and analyze data from this
year’s beta testing (and perhaps next year’s process results) before making
any recommendations regarding how to interpret the numerical score as a
categorical rating.
New Student Rating System • Short survey (15 minutes)• Reflects questions already used by many departments on existing student rating instruments• Provides comparison data (you vs. similar institutions nationally)• Allows for student comments• Includes resources to improve teaching
For more information, seehttps://www.ets.org/sir_ii/about
Overview of Tools to Help● www.utm.edu/facultyevaluation
○ Faculty Evaluation Tab on Assessment Website■ FEP OVERVIEW
■ GUIDEBOOK
■ FAQs
■ MEETINGS and DEADLINES CALENDAR
■ PROJECT TIMELINE in progress
■ COMMITTEES and BETA GROUPS - Who is doing what?
■ RESOURCES: FORMS/TEMPLATES/FACULTY EVALUATION WORKSHOP SLIDES (today’s presentation)
Digital Measures Overview and Implementation
Stephanie Kolitsch
Digital MeasuresWhat is it?
Software to track Faculty Productivity (teaching, scholarship, and service)
What does it do?Creates faculty vitas, annual planning and
assessment reports, etc.
Who uses it?The College of Business and Global
Affairs already uses it. We will be using it campus-wide.
Is it hard to learn or use?No--we will have a guidebook and training sessions. We are planning multiple opportunities to meet in small-group settings and work.
When do I have to start using it?Spring 2019. We’ll start small (what
have you done in 2018-19) then work backwards to fill in earlier years.
Some departments are planning to have student workers help with
inputting information.
Stay tuned for more info....
Questions?
Short Break then on to Bylaws
It didn’t, but you forgot to
volunteer.
Workshop Goal: To work together to accomplish a difficult task we all have in common.Bias disclaimer: I always seek to
• Protect Tenure• Strengthen Faculty Voice• Use the System
Planned Outline•Context & Climate: Why bylaws (and why now)?
•Specific: We spend most of our time reviewing our bylaws with our colleagues, interspersed with large group discussion. The first items will take the most time, later ones are quick and easy.
Planned order: 1, 2, 3, 4, 5, (6 & 7), 8, 9, 12, 11, 10
Before We Start (3 min)
Use the 3 by 5 card to write down your biggest concern
(just one per card)
Why Bylaws?
•They are required by the BOT
[BOT Resolution for BT0006, 18 Mar 2018] 3. (b) … Departments shall submit the required procedures [for Appendix A: Tenure] for approval no later than November 30, 2018 and shall be effective beginning with the next tenure review cycle following approval.
BUT WHY?
Public Perception of Tenure
Public Perception of Tenure
Some of our Perceptions of Tenure?
Some of our Perceptions of Tenure?
Why Bylaws?
•They are required by the BOT
•They help protect the due process of tenure and academic freedom
•They can support openness, fairness and collegiality
“… having a set of bylaws to address promotion and tenure within the university framework instructs outside evaluators what the department considers important among the criteria for T&P. It was that specific issue that led to us developing bylaws back when I was dean. The performing arts people needed it the most as did several of our departments in A&S.”
“Bylaws also codify rules for selecting department chairs and for committee structures. There are good reasons departments and colleges should have their own rules in this age of over bureaucratizing things.”
What Else Belongs in Bylaws?
•Rules to make the bylaws hard to change
• Information all must know and which rarely changes
• clear solid walls
What Does Not Belong?
•Long quotes from other sources
•Web links to specific documents.
• anything that will keep changing
CC 3
(Don’t get defensive yet, please wait until the next slide.)
Suggestion: Simplify Using Hierarchy When Possible
Faculty Handbook
BOT PolicyFederal and State Law
Campus Procedures
College BylawsDepartmental
Bylaws
Just Remember…
•The deadline is close (Sept 7, 21)
•Tenure (Appendix A) is Key
•Everything can be fixed later.
Watch Pres. DiPietro’s
video from BOT
(< 5 min)
Checklist Walk Through
• Group yourselves into units (departments and disciplines)
• As we look at each item you will have group time to answer:
• Is it there? If so, place a checkmark in first box. (Extra credit: add reference or page)
• Does it seem to be a problem, maybe unclear or possibly false? If so, place a checkmark in the query box.
• We will work together and discuss each item.
List 1
•Why? •Any danger in being too precise, too imprecise?•Consistent with college / campus procedures?•Are your criteria clear enough?•Who has a good example to share?•Any advice based on experience?
(Uniqueness? Protection?)
Discussion QuestionWhat are (or should be) the differences between the standards for tenure and for promotion to associate professor?
• Is one harder to get than the other?
• Why might someone get tenured but not promoted?
• Why might someone get promoted but not tenured?
• How is this difference reflected in your bylaws?
• Did your deans address external reviews in their bylaws?• Short reference (except maybe f)?• Note defaults to the majority of the committee (not voters). • Should require that all committee members sign.
External Reviews
(In the white 25 page June BOT Excerpt)
Approved New Handbook Text:
•Anyone going to address this in their bylaws?• If so, what are you going to say?
Enhanced Tenure Track Review (ETTR)
PPC meets August 30, Senate votes September 25, …, BOT votes November 2.
All tenure-track faculty members shall submit a dossier to be reviewed in either the third or fourth year of the faculty member’s probationary period (to be determined by the department chair) for an Enhanced Tenure Track Review (ETTR) to be conducted by department faculty and the department chair following the same committee structure, voting procedures as tenure with the exception that external reviews are not required. The department faculty will record their vote and write a recommendation including suggestions for furthering the progression towards tenure, and a minority and majority report if required using the form at the Academic Affairs website. The department chair will meet with the faculty member to share the report of the department faculty, as well as his or her own written assessment, then discuss plans enhancing the faculty member’s progression toward tenure. The committee report and the chair’s assessment will be shared with the college dean.
QUICK
BREAK (?)
• Find the (Beige) template for PPPT
• Address almost all issues • Will be on Acad. Affairs
site• Due to the System October
15• Board of Trustees Nov 1-2
•How many have already done this?•Have your deans given you any guidance?•Any questions before we look at it?
•Consistent with college / campus procedures?•Are your criteria clear enough?•Who has a good example to share?•Any advice based on experience?
• Both are required by Handbook but if FEP becomes sufficient that could be adjusted.
• I am not 100% sure what salary adjustment is. Merit pay? Annual raises? Raises with promotion?
[Faculty Handbook 3.4.1] The criteria for evaluating non-tenure-track faculty for purposes of hiring and retention must be approved by the departmental faculty and published in the departmental bylaws.
• Any questions?
Music Example: Proposed amendments to these bylaws must be brought to the full faculty by the Advisory Committee, which will accept suggestions from any faculty member. Proposed changes must be distributed electronically to the full faculty at least 10 business days prior to a vote.
The bylaws may be amended by a simple majority vote at any departmental meeting; faculty members who are unable to attend may submit their vote on any amendment(s) to the Chair in advance of the meeting. Amendments must be approved by the Dean of the College of Humanities and Fine Arts and the Vice Chancellor for Academic Affairs.
Math Example: These Bylaws may be amended at any regular or special Departmental faculty meeting for which particular notice has been given. A simple majority vote is required to amend any provision. Any amendments will be forwarded to the Dean and the VCAA.
Ed Studies: These by-laws may be amended at any regular or special Departmental faculty meeting for which particular notice has been given. Amendment of any provision requires a simple majority vote.
https://www.utm.edu/departments/music/https://www.utm.edu/departments/educate/documents.php
Music Example: 2.4 Voting. Full-time tenured and tenure-track faculty are eligible to vote on all motions and matters brought before the department. Full-time lecturers may vote on all issues with the exception of those dealing with personnel. Part-time, adjunct faculty, and faculty with visiting appointments are not eligible to vote. If a faculty member cannot attend a meeting, they may submit their vote on an issue or proposal to the Department Chair.
Proposals must be vetted and approved by the appropriate committee prior to consideration by the faculty. The committee chair must distribute proposals …
A simple majority vote with a quorum will decide all issues. …
Math/Stat Example: Membership and Voting: All faculty members who are serving as full-time faculty or have budgeted part-time teaching responsibilities within the Department pursuant to an appropriate appointment by the Vice Chancellor for Academic Affairs (VCAA) are entitled to vote, subject to University guidelines, on all matters for which faculty input is sought. Administrative staff who have faculty rank in the department are considered voting members of the faculty during those semesters in which they are teaching.
Ed Studies Example: Membership and Voting: All term, tenured, or tenure-track faculty with 50% or more teaching responsibility who are serving as full-time faculty within the Department of Educational Studies pursuant to an appropriate appointment by the Vice Chancellor for Academic Affairs (VCAA) are entitled to vote, subject to University guidelines, on all matters for which faculty input is sought. All faculty members are similarly expected to participate fully in all Departmental responsibilities.
[BT0006 App B. 2 p. 23] The department head shall direct the tenured departmental faculty to consider the faculty member’s performance in teaching, research, and service and, by an anonymously cast vote taken in accordance with applicable department or college bylaws, to make a recommendation on the question of whether the faculty member’s performance constitutes Adequate Cause for termination. [Question: Does this imply bylaws must proscribe the method?]
“What important questions remain unanswered?”
Remainder of Morning Session (if any)
Built in Work Time
LUNCH 11:30am-1:30pm
Afternoon Session1:30pm-4:30pmBeta Groups, Deans
Chris K Caldwell16 Aug 2018
Developing your (beta) Faculty
Evaluation Process
CFEC (GUIDEBOOK PAGE 8)
57
The Campus-wide Faculty Evaluation Committee (cFEC) is the faculty group that will continue to oversee and adjust this evaluation process. They will:
1. Listen2. Provide Guidance3. Collect Data4. Adjust
They will continue to work with the Senate’s PPC.
DEVELOPING A COMPREHENSIVE FACULTY
EVALUATION SYSTEM WORKSHOP
Raoul A. Arreola, PhD
Professor EmeritusThe University of Tennessee Health Science Center
58
SECTIONS:1. BIG PICTURE FIRST2. DEVELOP YOUR SYSTEM
(LONG)3. WRAP-UP
59
A NON-OPTION
60
REMINDER FROM THE GUIDEBOOK (PAGE 1)
61
• two outcomes• numerical score between 1.00 and 4.00• one of four BOT categories
• numerical scores consistent across campus• no fixed translation
“Until we have used this process once or twice, there will be no fixed translation between these two types of outcomes (numeric and categorical).”
FOUR CATEGORIES (GUIDEBOOK PAGES 13-14)
62
•Exceeds Exectations•Meets Expectations•Needs Improvement•Unsatisfactory
Professional Level for RankExemplary Performance for Rank
FOUR CATEGORIES (GUIDEBOOK PAGE 14)
Professional Level for Rank
“Applies to faculty who, during this rating period, consistently met the campus, college and departments’ standards for professional performance. These faculty are the excellent and valued professionals on whom UT Martin relies for the continued success of our programs, missions, and goals; and should include the clear majority of our faculty.”
FOUR CATEGORIES (GUIDEBOOK PAGE 13)
Exemplary Performance for Rank
“Applies to faculty who, during this single rating period, consistently and substantially exceeded the campus standards of professional performance. This rating is reserved for those rare faculty members who are true models of peak academic performance and who make significant contributions to their departments, university, academic field, and society.”
PURPOSES OF FACULTY EVALUATION (1 OF 2)
“To provide meaningful feedback for individual professional growth:
• Enhance current professional performance
• Detect any emerging deficiencies and provide information for correcting them.” 65
PURPOSES OF FACULTY EVALUATION (2 OF 2)
“Provide accurate & reliable summative information based on a pattern of performance over time on which to base personnel decisions (Promotion, Tenure, Merit Pay, etc.).”
66
REMINDER FROM THE GUIDEBOOK (PAGE 6)
1.2. Tenure and Promotion
“The results from this annual faculty evaluation process will inform decisions about tenure and promotion in the same way the results from the old evaluation processes did, but they will not directly change any of the current tenure and promotion requirements or processes.”
FROM ARREOLA’S TRAINING
The development of a successful faculty evaluation system requires integrating the technical requirements of good measurement with the political process of building consensus around shared values.
68
SECTIONS:1. BIG PICTURE FIRST
2. DEVELOP YOUR SYSTEM
3. WRAP-UP
69
THE 8-STEP PROCESS - OVERVIEW
1. Establish the Faculty Role Model2. Determine Role Parameter Values3. Define Each Role’s Components4. Establish the Role Component Weights5. Determine Appropriate Sources6. Establish Source Weights7. Determine How to Gather Information8. Design or Select Appropriate Forms
70
STEP 1: ESTABLISH THE FACULTY ROLE MODEL
71
Faculty Role Model
Teaching
Scholarship
Service
STEP 2: DETERMINE ROLE PARAMETER VALUES
72
minimum weight Faculty Role Model maximum
weight
50% Teaching 90%
5% Scholarship 40%
5% Service 40%
EXAMPLE: MATH/STAT ROLE PARAMETERS
73
minimum weight Faculty Role Model maximum
weight
50% Teaching 90%
15% Scholarship 40%
15% Service 40%
REMINDER FROM THE GUIDEBOOK (PAGE 12)
74
2.3. Setting the Weights of Components“It is important to note that these weights are measures of value and are not the same as workload. The value of an activity is often independent of the amount of time it took. It is entirely possible that some faculty teaching only one-quarter time may still prefer to have half of their evaluation based on the quality of their teaching. However, it is expected that workload will be considered in the weight negotiation.”
STEP 2: ACTIVITY ONE (5 MINUTES)
75
Locate and Verify your unit’s parameters• Are they possible?• To whom do they apply?• Anyone using multiple
versions?• Explore min/max
STEP 3: DEFINE EACH ROLE’S COMPONENTS
(Definition of the roles and their components are found in the Guidebook)
Chapter Three!
76
• What is translation?• What is proficiency?• Where does mentoring students go?• Where does mentoring faculty go?
STEP 4: ESTABLISH ROLE COMPONENT WEIGHTS
cFEC fixed the components of teaching, but left others with 0 to 100%
77
STEP 4: COMPONENT WEIGHTS FOR TEACHING
78 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
InstructionalDesignInstructionalDeliveryInstructionalAssessmentCourseManagement
Component Weights
30%
30%
25%15%
fixed
STEP 4: MATH/STAT WEIGHTS FOR SCHOLARSHIP
79 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
Proficiency
Discovery / Creativity
Dissemination
Translation
Component Weights
20 – 60%20 – 60%20 – 60%–
40%0
STEP 4: POLITICAL SCIENCE WEIGHTS FOR SCHOLARSHIP
80 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
Proficiency
Discovery / Creativity
Translation
Component Weights
20%
0%
80%
0%
Dissemination
STEP 4: OTHER BSBA WEIGHTS FOR SCHOLARSHIP
81 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
Proficiency
Discovery / Creativity
DisseminationTranslation
Component Weigh
ts20 - 40%
20 - 40%
30 - 50%
0 - 5%
DEF. OF TRANSLATION FROM GUIDEBOOK (PAGE 27)
82
This component of scholarship is based on the general principle of utilizing research results within a discipline, not necessarily from your own research, and applying those results to the development of new products, services, performances, or artistic expressions of value or benefit to the professional or larger general social audience.
STEP 4: AG WEIGHTS FOR SERVICE
83 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
To the InstitutionTo theProfessionTo the CommunityTo theStudents
Component Weights
40 - 75%
20 - 50%
10 - 40%
0 - 70%
STEP 4: BIOLOGICAL SC. WEIGHTS FOR SERVICE
84 100%Total Source Impact Weights
SOURCES ??
??
??
RoleComponents
To the Institution
To the CommunityTo theStudents
Component Weights
15 - 60%
0 - 45%
0 - 45%
15 - 60%
To theProfession
STEP 4: ACTIVITY TWO (10 MINUTES)
Verify your unit’s component weights and place them on the
source matrices.
• Are each of the values possible? • Is there consensus for the value statements they make? • Did anyone change (or want to change) their weights?
85
WEIGHT DECLARATION FORM
86
Exciting features!• Version• Dean (temp)• Excel version• Based on PAR• Questions?
WEIGHT DECLARATION FORM
87
Potential features:• Color• Planning• A draft! (cFEC)
STEP 5 DETERMINE APPROPRIATE
SOURCES
Reach consensus on which source(s) should provide the information on which the evaluation of each role will be based.
88
HOW NOT TO DETERMINE APPROPRIATE SOURCES
A Bad Source Principle:
89
STEP 5: DETERMINE APPROPRIATE SOURCES
The “Best Source Principle:”✓ Obtain information from those who
have first hand experience with the performance in question.
90
STEP 5: DETERMINE APPROPRIATE SOURCES
91
Caldwell’s source principle: This is a beta year, do not bite off more than you can chew.
STEP 5: SOURCE IDENTIFICATION MATRIX FOR TEACHING
92
RoleComponents Students Peers ChairInstructionalDeliveryInstructionalDesignInstructionalAssessmentCourseManagement
YES
YES YES
YES YES
YES
NO
NO NO
NO
NONO
STEP 5: ACTIVITY THREE (10 MINUTES)
Lightly (in pencil) place “yes” for sources you will use and
“no” for those you will not. REACH CONSENSUS!
(If you group is done early, start thinking about Step 7: what will be used to collect info from these sources: rubric? form?
survey?)
93
STEP 5: ACTIVITY FOLLOW-UP
94
• Are there any of you who had trouble agreeing on these?
• Did anyone use anything other than students for instructional delivery?
• Anyone use self evaluations currently?
• How will they be shaped and used?
Quick Questions:
STEP 6 DETERMINE SOURCE
WEIGHTS
Determine how much value or weight should be placed on the information provided by the various selected sources
95
STEP 6: DETERMINE SOURCE WEIGHTS
Determine the impact the information from the various sources will have on the overall
evaluation of each role.(“source impact weights”)
96
STEP 6: SOURCE WEIGHTS FOR TEACHING
97 100%Total Source Impact Weights
PeersRole ComponentsInstructional Delivery
Instructional AssessmentCourse Management
Component Weights
30%30%
25%
15%
100 0 0
25 75 0
0 0 100
0 80 20
Students Chair
YES NO NO
YES YES NO
NO YES YES
NO NO YES
Instructional Design
STEP 6: SOURCE IMPACT MATRIX FOR TEACHING
98 100%Total Source
Impact Weights
PeersRole ComponentsInstructional Delivery
Instructional AssessmentCourse Management
Component Weights
30%30%
25%
15%
100 0 0
25 75 0
0 0 100
0 80 20
Students Chair
30% 0% 0%
8% 23% 0%
0% 20% 5%
0% 0% 15%
Instructional Design
38% 43% 20%
STEP 6: ACTIVITY FOUR (15 MIN)
Agree on source weights for your source matrices
99
• Use pencil (you might want to change in the next step)
• Do your rows add to 100%?• Change ‘Yes’s / ‘No’s if there is a new consensus
• Did anyone change any of their ‘Yes’s or ‘No’s? • Why did you change?• Does the impact of students, peers and chair seem reasonable for your unit?
STEP 7 DETERMINE HOW TO GATHER
INFORMATIONDetermine what type of form, questionnaire, checklist, or other data gathering method will be used to obtain the specified information from each source.
100
STEP 7: DATA GATHERING MATRIX FOR TEACHING
101
Students
ChairPeersRoleComponents
Sources
InstructionalDelivery
InstructionalDesign
InstructionalAssessment
CourseManagement
YES
YES YES
YES YES
YES
NO NO
NO
NO
NO NO
STEP 7: DATA GATHERING MATRIX FOR TEACHING
102
Students ChairPeers
RoleComponents
Sources
InstructionalDelivery
InstructionalDesign
InstructionalAssessment
CourseManagement
RatingForm
Peer Reviewof Materials
Interview
Checklist,Grade Report
RatingForm
Peer Analysisof Exams, etc.
STEP 7: DATA GATHERING MATRIX
103
Again, consider how much you are biting off.• Can you use what you are already doing?
• Less complicated processes at start?
• Leave more to the chair at first?
• Do you have any concerns or questions that others in the room may be able to address?
STEP 7: ACTIVITY FIVE (10 MIN)
104
In the appropriate cells on your source matrix identify the method you plan to use to collect data from that source.
• Rating Form• Rubric• Checklist• Interview• Student Rating• Other suggestions?
We will work 10 minutes, then pause
to discuss & pool our knowledge, then
• Who is done?• Who is far from done?• Which are difficult?• decide what more time
we need.
STEP 8 DESIGN AND/OR SELECT
APPROPRIATE FORMS
All forms and measurement procedures should be as valid, reliable, and as objective as possible.
105
STEP 8: DESIGN AND/OR SELECT FORMS
✓ Student Ratings (next)✓ Rubric✓ Count and scale✓ Checklist✓ Self Rating form✓ …
106
STEP 8: DESIGN AND/OR SELECT FORMS
“If possible, select and/or adapt from professionally developed forms, especially student rating forms”
Consult faculty whose base profession expertise is in psychometrics
107
STEP 8: DESIGN AND/OR SELECT FORMS
“If building your own forms consult with faculty whose
base professional expertise is in assessment or
psychometrics.” 108
DRAKE’S PROFESSIONAL ACTIVITIES RECORD (PAR)
One source for forms is Drake’s Professional
Activities Record. Find it in your papers—we will look at it after the next two slides.
109
STEP 8: OUR CHOSEN RATING FORM
110
• Short survey (15 minutes)
• Reflects questions already used by many departments on existing student rating instruments
• Questions sorted into sections that fit well within Teaching components
• Instructional Delivery: Section A. Course Organization and Planning and Section E. Supplementary Instructional Methods
• Instructional Design: Section B. Communication and Section C. Faculty/Student Interaction
• Provides comparison data (you vs. similar institutions nationally)
• Allows for student comments
• Includes resources to improve teaching
STEP 8: ACTIVITY SIX (10 MIN)
Look at the SIR II and the PAR forms with your
colleagues 1. How might you use the SIR II? (Where on your matrices?)
2. Are any of the PAR forms adaptable?
3. What are the PAR forms’ strengths and weaknesses? 111
STEP 8: DESIGN / SELECT APPROPRIATE FORMS
Arreola says: The forms used to gather data from all sources
must use a numerically common scale.
112
STEP 8: WHAT IF NOT FROM 1 TO 4?
•
113
TRANSLATIONS DO NOT NEED TO BE LINEAR
114
rating description
4.0 Rare or exceptionally high student ratings in every section taught
3.6 For more than half of the sections, scores are exceptionally high
3.2 For all class sections, student ratings are at least satisfactory
2.9 For the majority of class sections, student ratings are satisfactory
2.0 For the majority of class sections, student ratings are unsatisfactory
1.5 Clear evidence of consistently unfavorable student ratings
1.0 Rare or exceptionally low student ratings in all sections; or fails to complete any student evaluations
CREATING SCALES (GUIDEBOOK PP. 17-18)
115
“Departmental and college rating scales will be used to rate faculty in each component of each role in the faculty evaluation system, so it is essential that these scales be developed with the proper input of the faculty who will be held accountable to the expectations reflected in the rating scales.”
CREATING SCALES (GUIDEBOOK PAGE 18)
116
CREATING SCALES (GUIDEBOOK PAGE 18)
117
TABLES ARE USEFUL FOR THE SUBJECTIVE
118
For example, suppose you need to evaluate one of Dr. Caldwell’s publications
USING TABLES (A SIMPLE RUBRIC)
119
Level 3.0 Achievement of at least one of the following results:
• Submission of external research grant proposal judged as being significant by peers and departmental chairs/directors
• Presentation of peer-reviewed paper, workshop, symposia, poster-session, etc., at an acceptable academic, professional, or pedagogical conference or meeting
• Invited published papers judged by peers as requiring significant effort or having a significant impact based on quality or publication outlet
• Publication of a case or paper in peer-reviewed meeting proceedings or book
• Publication of chapter in scholarly book, professional book or textbook• Publication of editorials or research comments in professional or academic
publication.From https://advance.washington.edu/resources/docs/rubrics.pdf
INSTRUCTIONAL DESIGN (PAGE 20)
120
Required documentation:• Course syllabi that meet university and department
standards • List of course objectives and assessments with ties to
appropriate student learning outcomes • Evidence of appropriate sequencing • Evidence of use of appropriate materials and resources
• Evidence that minimal requirements as established by department course descriptions and standards have been taught
3.2.4 COURSE MANAGEMENT (PAGE 22)
121
Required documentation:
• Documentation of compliance with departmental, college, and university deadlines (including Banner grade submission reports)
• If unable to meet a class period, documentation of substitute faculty or communication with class members regarding alternative assignments/expectations
• Evidence of appropriate management of the course (e.g., ensuring that all materials and resources are available for the course, ensuring that students receive appropriate feedback, making appropriate arrangements for field trips, etc.)
INSTRUCTIONAL DELIVERY / ASSESSMENT
122
3.2.2. Instructional delivery (page 21)
Required documentation:
• Evidence of appropriate class preparation and delivery
3.2.3. Instructional assessment (page 22)
Required documentation:
• Evidence of university-level assignments
STEP 8: ACTIVITY SEVEN (15 MIN)
In each square of your source matrices write the
method you will use to collect the results.
123
REMEMBER THE GOAL (GUIDEBOOK PAGE 18)
124
TEACHING EXAMPLE FROM GUIDEBOOK (PAGE 20)
125
See also: • Scholarly Activity Example page 24• Service Example page 28
STEP 8: ACTIVITY EIGHT A (10 MIN)
Using the Posters provided list Templates, Rubrics, Forms, … that you already have or think you need
126
• Label the pages Teaching, Scholarship and Service respectively
• Write your unit name on each page• Mark each with H for have and N for need• We will collect and display these
BETA COLLABORATION AT EVERY STEP
127
STEP 8: ACTIVITY EIGHT B
Begin Collecting, Adapting and Constructing the Forms,
Rubrics, … you will use.
128
• Adapt what you already have?• Search the web• Ask other disciplines
SECTIONS:1. BIG PICTURE FIRST
2. DEVELOP YOUR SYSTEM
3. WRAP-UP
129
USING THE COMPLETED SYSTEM
Quick Reminder(Use Excel!)
130
REMINDER FROM THE GUIDEBOOK (PAGE 17)
During the beta year, units will be experimenting with ways of valuing evidence and will have no track record with which to standardize scoring. It may be that “professional” faculty members score below 3 (or an “unacceptable” faculty member scores above 3) because of an error or unexpected consequence of the draft system. For this reason cFEC recommends that chairs of beta units assign the categorical level “as usual.” For the beta units that use both the old and new systems during the beta year, this will likely mean using the old system for the categorical rating and the new for a numerical rating.
USING THE COMPLETED SYSTEM: RATINGS
✓ The evaluation of a faculty member within a role is called the Composite Role Rating (CRR).
✓ The overall evaluation of a faculty member combining all role evaluations into one is called the Overall Composite Rating (OCR)
132
EXAMPLE: COMPUTING THE COMPONENT RATINGS
133
3.93.3 2.9
3.9 3.42.4
100% 3.90
25%75%
20%
100%
3.203.802.40
InstructionalDelivery
InstructionalDesign
InstructionalAssessment
CourseManagement
Students Peers Chair
80%
+
+
EXAMPLE: COMPUTING THE OCR
E = EXEMPLARY ► 4ish
P = PROFESSIONAL ► 3ish
I = NEEDS IMPROVEMENT ► 2ish
U = UNACCEPTABLE ► 1ish
UT Martin’s Chosen Scale
135
Recommended Overall System Evaluation Scale
Warning math rant: E, P, I, U are categorical data. They are ordinal, but not interval….
The numbers 4, 3, 2, 1 are numerical data that are both interval and ratio. We make a significant value statement with significant consequences when we equate the two (think reverse).
KEEP THE CONVERSATION GOING
✓ More info from cFEC coming soon✓ Share info among beta groups✓ Provide feedback:
• Beta Group Department Chairs, [email protected]• Campus Wide Committee, [email protected]• College Committee, [email protected]
Look for updates on Office 365 Beta Team & join the conversation!
THE END
