IDEA Student Ratings of Instruction

Created by:Shelley A. Chapman, PhDSenior Educational Consultant

InsightImprovementImpact®

Texas A & M UniversityFebruary/March 2013

Plan for this Session• What Makes IDEA Unique• Conditions for Good Use• Underlying Philosophy of IDEA• Faculty Information Form• Interpreting Reports• Questions and Answers

IndividualDevelopmentEducationalAssessment

Teaching Improvement

Faculty EvaluationCurriculum ReviewProgram AssessmentAccreditation

• Kellogg Grant in 1975• Separate Organization and Non-profit status 2001

• Mission

InsightImprovementImpact®

To help colleges and universities as they seek to improve teaching, learning, and leadership

A Non-Profit Organization

What makes IDEA unique?

1. Focus on Student Learning

2. Focus on Instructor’s Purpose

3. Adjustments for Extraneous Influences

4. Validity and Reliability5. Comparison Data6. Flexibility

Conditions for Good Use

The instrument • Targets learning • Provides suggested action steps for teaching improvement

Conditions for Good Use

The Faculty• Trust the process• Value student feedback

• Are motivated to make improvements

Conditions for Good Use

Campus Culture• Teaching excellence - high

priority• Resources to improve -

provided• Student ratings -

appropriate weight

Conditions for Good Use The Evaluation Process

• 30-50% of evaluation of teaching• 6-8 classes, more if small (<10)• Not over-interpreted

(3-5 performance categories)

Reflective Practice using Individual Reports

Collect Feedback

Interpret Results

Read & Learn

Reflect & Discuss

Improve

IDEA resources that are keyed to reports

Talk with colleagues

Try new ideasOnline, Paper

What the reports sayand what they mean

Underlying Philosophy of IDEA

Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.

Faculty Information Form

Student Learning ModelSpecific teaching behaviors are associated with certain types of student progress under certain circumstances.

Student LearningTeaching Behaviors

Circumstances

Student Learning Model: Diagnostic Form

Student LearningItems 21-32

Teaching Behaviors

Items 1-20

CircumstancesStudents: Items 36-39, 43

Course: Items 33-35

Summary Items: 40-42Research Items: 44-47Up to 20 extra items

FIF: Selecting Objectives

• 3-5 as “Essential” or “Important”

• Is it a significant part of the course?

• Do you do something specific to help students accomplish the objective?

• Does the student’s progress on the objective influence his or her grade?

Be true to your course.

Common Misconception #1

Students are expected to make significant progress on all 12 learning objectives in a given course.

Common Misconception #2

Effective instructors need to successfully employ all 20 teaching methods in a given course.

Relationship of Learning

Objectives to Teaching Methods

Common Misconception #3

The 20 teaching methods items should be used to make an overall judgment about teaching effectiveness.

Faculty Evaluation

Course Description Items (FIF)• Used for research• Best answered toward end of term• Do NOT influence your results

Bottom of Page 1

Top of page 2

IDEA Online

IDEA Online: Student Survey Delivery

• Email/Course embedded URL

• Blackboard Building Block• Email reminders• Start/end dates Determined by Institution

• Submission is confidential and restricted to one

Online Response Rates – Best Practices• Create value for student feedback

• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other

• Prepare Students• Talk about it• Syllabus

Example: Course Syllabus

Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)

Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomena

Students will think more critically about the earth and environment

Objective 8: Developing skill in expressing myself orally or in writing

Students will be able to present scientific results in written and oral forms

IDEA Center Learning Objective

Course Learning Outcomes

IDEA Online: FIF Delivery

• Email delivery/reminders• Start/end dates determined by Institution

• Access is unlimited while available

• Questions can be added to student survey

• Objectives can be copied from previously completed FIFs

Copying Objectives

Reflective Practice with IDEA

Collect Feedback

Interpret Results

Read & Learn

Reflect & Discuss

ImproveIDEA Student Ratings of Instruction

Reports

Diagnostic Report Overview1. How did students rate their

learning experience?

2. What contextual factors impacted those ratings?

3. How do my scores compare to: IDEA, discipline, and institution?

4. What might I do to facilitate better learning for my students next time?

Your Average (5-point Scale)

Raw Adj.

A. Progress on Relevant Objectives1

Four objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.

1. How did Students Rate their Learning?

ProgressOnRelevantObjectives

4

4.3 + 4.34.14.23.6

5

Summary Evaluation: Five-Point Scale Report Page 1

Your Average Score

(5-point scale)

Raw Adj.

A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)

4.1 4.3

Overall Ratings B. Excellent Teacher 4.7 4.9

C. Excellent Course 4.1 4.4

D. Average of B & C 4.4 4.7

Summary Evaluation(Average of A & D) 4.3 4.5

50%

25%25%

2. What contextual factors impacted those scores?

Adjusted Scores• Student Work Habits (#43DF)• Student Motivation (#39DF)• Class Size (Enrollment, FIF)• Student Effort (multiple items)• Course Difficulty (multiple items)

2. What contextual factors impacted those ratings?

3. How do my scores compare to: IDEA, Discipline, Institution?

Comparisons (Norms): Converted Averages

• Able to compare scores on the same scale T Scores

• Average = 50• Standard Deviation = 10

• They are not percentiles or percentages

Comparisons (Norms): Converted Averages

Comparison Scores Distribution

40%

Similar

Lower20%

10%Much Lower

10%Much Higher

20%Higher

Gray Area

Comparison Scores

4. What might I do to facilitate better learning next time?

Page 2: What did students learn?

Page 3: Suggested Action Steps

#16#18#19

POD-IDEA NotesIDEA Website

POD-IDEA Notes

• Background• Helpful Hints• Application for online learning

• Assessment Issues• References and Resources

References and Linksto Helpful Resourcesare Provided

IDEA Papers

Resources for • Faculty Evaluation• Faculty Development

Reflective Practice

Collect Feedback

Interpret Results

Read & Learn

Reflect & Discuss

Improve

POD-IDEA Notes IDEA Papers

Meet with colleaguesto reflect

Interpret Reports

Paper or Online

Try something new

Questions ?ideasri.tamu.edu

www.theideacenter.org

Visit our IDEA Help Community!

Teaching Goals Inventory