HCCS - Evaluation for Greater Learning Student System

1

EGLS3 - Evaluation for Greater Learning Student Survey System Implementation at Houston Community College

EGLS3

Dr. Alan AinsworthFormer HCC Faculty

Senate President

Mario HerediaDirector Research &

Support Services

February 23, 2012

Dr. Mac AdkinsPresident

SmarterServices LLC

Jeff WorfordLead Developer

SmarterSurveys™ Product

2

Houston Community College

● A comprehensive community college serving the Houston area

● Over 75,000 students credit, non credit in Fall 2011 ● One of highest number of international students of all community colleges in the country

EGLS3

3

Student Evaluation of Instruction – History

● Paper-Based Student Evaluation of Instruction (SEOI) survey has been in place for 27 years, revised incrementally.

● For Face to face classes, the SEOI was the HCC’s paper method to administer the student evaluation

● For online classes Distance Education used their own online survey

● Structure. Four part-time staff plus a supervisor working for two months.

● Large work room

● Given the current technology, Institutional Research sought

-Online administration methodologies-New Instrument

EGLS3

4

HCC’s SEOI Survey Paper Distribution System

● IR manually prepared and routed more than 5,000 packets and 100,000 SEOI instruments each semester. Each packet contains the SEOI instrument and instructions for faculty and proctors. ● The packets were sent to their respective campuses administrators.

● Campuses administrators were responsible for distributing the materials to the Instructors and collecting from the proctors (who administer the survey to the students).

● Once IR received the packets, entered them manually in the system to track the return rate. ● IR cleaned the packets, scanned the surveys and generated data file. The report was run in SPPS.

● The final report was saved on disks and sent to the College Administrators for manual distribution. Students’ hand written comments were sent back to faculty.

● Process is cumbersome, labor intensive and costly.

● SEOI’s means report results were not used systematically in the evaluation process. ● Return rates have been an issue, but in Spring 2010 the rate of return increased to 95% and the

response rate was of 73%.

EGLS3

5

Student Evaluation of Instruction (SEOI)– Paper Base Instrument

EGLS3

6

EGLS3 Student Evaluation Survey Instrument

EGLS3

Houston Community College EGLS3 – Evaluation for Greater Learning Student Survey System

Organization and Explanation of MaterialsStrongly Agree

AgreeDisagr

eeStrongly Disagree

1. My Instructor explains diffi cult material clearly.

1. My Instructor communicates at a level that I can understand.

1. My Instructor makes requirements clear. 1. My Instructor identifies relationships between

and among topics.

Learning Environment

Strongly Agree

AgreeDisagr

eeStrongly Disagree

1. My Instructor establishes a climate of respect.

1. My Instructor is available to me on matters pertaining to the course.

1. My Instructor respects diverse talents.

1. My Instructor creates an atmosphere in which ideas can be exchanged freely.

Self-Regulated LearningStrongly Agree

AgreeDisagr

eeStrongly Disagree

1. My Instructor gives assignments that are stimulating to me.

1. My Instructor encourages me to develop new viewpoints.

1. My instructor arouses my curiosity.

1. My Instructor stimulates my creativity.

Overall OpinionsStrongly Agree

AgreeDisagr

eeStrongly Disagree

1. I like this instructor.

1. I am interested in this subject. 1. I think the classroom was appropriate for this

class.

1. I would recommend a course taught by this instructor.

General Information1. This class is:

A Required Course An Elective I am not sure

1. What grade do you expect to earn in this course? A B C D F

7

Important EGLS3 Features from UNT’s SETE

● These are the factors or elements of teaching effectiveness that are being measured:

Factor 1: Organization and explanation of materialsFactor 2: Learning EnvironmentFactor 3: Self-regulated Learning

● Four questions are used in the EGLS3 validity analysis. Scale scores (the weighted responses) are reported for each of the three domains. An overall score is also computed

The following are sample scale ranges for EGLS3

EGLS3

8

SEOI-EGLS3 Survey Instrument Comparison

Student Evaluation of Instruction(SEOI)

• Paper-Based• Home Grown Survey• Included non-instructional Questions• Library and course content questions• No Validated• Means report• IR cleaned the packets, scanned the

surveys and generated data file. The report was run in SPPS.

• The final report was saved in a disk and sent to the College Administrators. Hand-written comments returned to faculty.

• Process was cumbersome, labor intensive and costly.

• The results were not used in a systematic evaluation process.

EGLS3 - Survey Instrument

• Is implemented online to allow flexibility, transparency, and accountability

• Provides a measure of teaching effectiveness as perceived by students.

• Allows college-wide comparison in key areas

• Facilitates student evaluations of their instructors (not the course or the content)

• The EGLS3 scale scores can be used to make assessments for continual improvement of the institution

• Results available on line for individual faculty.

• User built their own reports

EGLS3

9

Early Stages

● Vice Chancellor of Planning and Institutional Effectiveness has Faculty and OIR representatives visit University of North Texas to discuss SETE.

● HCC Administration and Faculty Senate decide to move forward with assessing SETE for possible online evaluations.

EGLS3

10

Faculty Governance and Endorsement

EGLS3

● Faculty Senate President requests the evaluation of SETE for replacement of SEOI by Educational Affairs Committee

● Faculty Senate sees reports of SETE and votes unanimously to support pilot program for Spring 2011

11

Seeking Additional Endorsement

EGLS3

● Approval of Pilot from Vice Chancellor of Instruction, Deans’ Councils.

● Forums held for information and feedback

12

Student’s evaluation process, the transition from paper to online

● IR Partnership with Faculty Senate

● IR Responsibility: Develop and implement the Operational plan ● Faculty Senate President: Navigating Faculty Governance and Endorsement

● Pilot Implementation Spring 2011 . Two Locations were selected

● Outcome: Response Rate 20%

● Follow up: Student Focus Groups

● Findings: Login process cumbersome

EGLS3

13

Solutions for Log-In Process found with PS Integration

● HCC - SmarterServices People Soft login Integration of EGLS3 - Student Center

EGLS3

14

Student’s evaluation process, the transition from paper to online

● HCC - SmarterServices People Soft login Integration of EGLS3 - Faculty Center

EGLS3

15

Integration and Responsibility of IR in EGLS3 Implementation

● Institutional Research liaison between SmarterServices and the HCC departments participating in the EGLS3.

● Develop timeline and monitor progress

● Develop and review documentation with SmarterServices, IT/Costumer Support

● Develop and review communication plan with Communications Department.

● Several media channels used to notify the institution: meetings, email, paper, social media, web page, plasma screens, posters, etc.

EGLS3

16

Fall 2011 - Full Implementation Facts

● Only Regular Term and Second Start Classes surveyed, including Distance Education and Dual Credit

● Data submitted: 6,460 classes with 140,800 enrollment.

● System runs smoothly

● 23,005 responses with a Response Rate of 16%

Technical Issues● Pop –ups blocked in College Computer Labs and other computers.● System Slow due to lack of services running on the web server.

Reporting● Available on our Web Site ● Faculty, Administrator access ● Administrators can Build their Own Reports

Follow up● Faculty Focus Groups

EGLS3

17

FALL 2011, Implementation

Full Implementation Fall 2011. District-Wide

● Steering Committee in place

● Teams and strategy are developed

● Operations Strategy : How do we implement? What processes and procedures need to be developed

● Communication Strategy: How do we communicate? What and where? What are the benefits? How is it different? How does it relate to our vision?

● Feedback Strategy : How will this improve student learning? How will it improve teaching effectiveness?

EGLS3

18

Spring 2012-Next Steps

● Spring 2012 Next Steps

● Immediate: Increase response rate

● As survey reports became available, faculty and administrators will be more involved in the process

● Develop committees to work to integrate this students evaluation with HCC Faculty/Staff evaluation system PEP.

● Redevelop PEP

● Research Project

EGLS3

About SmarterServices

● Incorporated in 2002.

● Served over 500 educational institutions including K-12, technical colleges, career colleges, community colleges, private colleges, universities, and corporations.

● Some of the other solutions provided by SmarterServices include SmarterMeasure, SmarterFaculty, and SmarterProctors.

EGLS3

About UNT’s SETE™ Instrument

● Benefit from extensive research at the University of North Texas.

● Control for bias of 24-variables● Student grades● Time of course● Delivery system

● Track improvement with a standardized effectiveness score (1-1000 scale)

● Useful for benchmarking and longitudinal analysis of faculty effectiveness

● Meet state reporting guidelines

EGLS3

Student’s Perspective within SmarterSurveys™

EGLS3

● 18 Standard SETE Questions

● 16 Likert scale questions (4 per Category)

● 2 General Information questions


EGLS3

● Custom Question can be added to SETE.

● Custom questions can be Likert, open ended, multiple choice, etc.

● HCC choose to add 5 open ended questions to their evaluation


EGLS3

● Confirmation pop-up is presented to Student’s for verification upon submission of a survey.


EGLS3

● After completing a survey, the course is marked as “Completed”

● Any remaining surveys can be taken immediately or at another time


EGLS3

● HCC choose to suppress results until the end of the survey period.

● Faculty can still see their completion percentages, survey status, and results release date.

Faculty’s Perspective within SmarterSurveys™

EGLS3

SETE™ Data Processing Flow

Survey Responses

Faculty and Student Attributes + Grades

Data ProcessedEGLS3 Scores Generated

Faculty ID, Course ID & Scores

EGLS3

● Once survey results open, faculty can view course responses and also pull aggregate reports for the term.

Faculty’s Perspective within SmarterSurveys™

EGLS3

Individual Course Response Distribution Report – Per CourseHeader and Likert Responses

Faculty Reports within SmarterSurveys™

EGLS3

Individual Course Response Distribution Report – Per CourseOpen Ended Responses


EGLS3

EGLS3 Score Report – Per Term


EGLS3

Dean Reports within SmarterSurveys™

EGLS3

Dean Reports within SmarterSurveys™

EGLS3

34

EGLS3

35

Questions….

Contact Information:

Alan Ainsworth 713.718.8709; [email protected]

Mario Heredia 713.718.8627; [email protected]

SmarterSurveys™ Info 877.499.SMARTER (7627); [email protected]

EGLS3

Documents

HCCS - Evaluation for Greater Learning Student System