Embed Size (px)
DESCRIPTION
The Development and Usability Evaluation of a Standards-Based Grading Tool for Faculty
Citation preview
The Development and Usability Evaluation of a Standards-Based Grading Tool for Faculty
Alaa Sadik, Sultan Qaboos University (OMAN)
Introduction
Grades are the common method for reporting student performance across universities.
Today, universities tend to use standards-based assessment approach to assess student performance.
Introduction
Sultan Qaboos University(SQU): a cross-cultural organization, brings together hundreds of instructors
from around the world. the largest academic community,
consists of 9 colleges, a wide variance with regard to grading
practices.
Problem of the Study
Many instructors believe that: Preparing, scoring, grading, and
reporting student academic performance are each extremely exhausting and time-consuming tasks.
It is difficult to perform standards-based assessment according to the university quality assurance policy and requirements.
The Need
Therefore,the need was emerged to: Use/develop a standards-based assessment tool that allows for more accurate and relevant grading and reporting.
Research Questions
1. How effective (useful) is the assessment tool in documenting, grading and reporting student performance?
2. How efficient (easy to use) is the assessment tool as perceived by users?
3. What is the overall satisfaction of users toward the use of the assessment tool?
4. What individual difference variables influence instructors’ perceived usability of the assessment tool?
Purpose of the Study
The main purpose of this study was to design, develop and evaluate a standards-based assessment tool for instructors at SQU.
Method
Development of the assessment tool A review of the literature A series of interviews and focus groups
were conducted, to determine the gap between the existing grading practices of faculty and those that are needed for the assessment tool.
Method
The assessment tool should Support the University grading scale and generate
students’ grades automatically; Enable qualitative assessment of student performance; Attach content and performance standards to grade
sheets; Have a built-in e-mail function for communicating
grades with students; Facilitate total point and percent weight of scores and
assignments; and Generate course statistics for distributions,
correlations and variances.
Method
Method
Method
Method
Method
Sample116 faculty members from 4 colleges at SQU.
Demographics %
Gender
Male 79.9
Female 20.1
College
Education 56.7
Arts & Social Sciences 17.5
Engineering 14.3
Commerce & Economics 11.5
Perceived computer experience
Low 14.6
Moderate 65.2
High 20.2
Years of teaching experience
Less than 5 years 24.6
5-10 years 57.8
More than 10 years 17.6
Method
ProceduresParticipants were asked to perform a series of specific tasks in RealGrade, each of which had several subtasks.
Method
Instruments Grading tool usability questionnaire
To solicit participants’ perceptions toward the effectiveness, efficiency and acceptance of the tool.
Instructor interviewsTo gain a thorough understanding of
the use of RealGrade and to provide rich detail and insights into instructors’ experiences.
Results: Questionnaire
EffectivenessParticipants strongly agreed or agreed (m=3.93) that RealGrade : Is effective in facilitating the process of standards-based assessment, Has the tools needed to assess student performance and communicate grades with students, and Increases their productivity.
Results: Questionnaire
EfficiencyParticipants indicated that (m=3.34): Learning to use RealGrade takes a short time, The user interface menus and dialogue boxes are favorable, Weighting, assessing, and managing student assignments based on performance standards are very easy tasks.
Results: Questionnaire
SatisfactionThe majority of participants (m=4.19) reported that they liked and felt comfortable with RealGrade as a tool to assess student performance.
Results: Questionnaire
Scale/sub-scale Possible range Mean Std. DeviationEffectiveness 2.56-5 3.93 .6112Efficiency 2.34-5 4.34 .4635Satisfaction 2.21-5 4.19 .4521Overall scale 2.48-5 4.22 .5356
Paired t tests for the means
Mean differences
S.D. t
Effectiveness-efficiency -.3300 .5347 -11.29*Effectiveness-satisfaction -.3500 .5439 -11.77Efficiency-satisfaction -.0023 .5667 1.87
Results: Questionnaire
Participants with greater degrees of computer experience had higher perceptions of effectiveness, efficiency, and satisfaction with RealGrade.
Participants who had less than five years of teaching experience found RealGrade more effective, efficient and satisfactory than those who had more than five or ten years of experience.
Results: Interview
Results General patterns (frequency) Distinctive viewpoints (frequency)
1. Effectiveness (usefulness)
1.1. Avoid grading errors (2)1.2. Error-free and accurate (2)1.3. Helpful in grading and reporting (2) 1.4. Important for instructors (1)1.5. Improve instructor performance (1)1.6. Increase instructor productivity (2)1.7. Multiple output formats (1)1.8. Save time (3)
- Customized grading scale (1)- Professional way for grading (2)- Use well-defined rubric (1)- Collaborative (1)
2. Efficiency (ease of use)
2.1. Appropriate documentation (2)2.2. Compatible with my system (4)2.3. Easy to learn (5)2.4. Easy to setup (3)2.5. Flexible (2)2.6. Simple and attractive (3)2.7. User-friendly (2)
- Keep students updated about their performance (1)- Essential for quality assurance and accreditation (2)- Easier than Excel grade sheets (1)- Mobility (1)
3. Satisfaction 3.1. Like to use (5)3.2. Interesting (3)
- Recommended for official use (4)
Discussion and Conclusions
Differences among instructors are vital to the eventual acceptance and implementation of the standards-based assessment tool.
Further research is needed to develop report card to help instructors and students understand the standards-based assessment information included and to make it more comprehensible.
