8/14/2019 Angela Yarnell MEMT 898 Daugherty October 1, 2009 Research Analysis

1/2

Angela YarnellMEMT 898DaughertyOctober 1, 2009

Research Analysis 2

Ciorba, C. (2009). Measurement of instrumental and vocal undergraduate performance juriesusing a multidimensional assessment rubric. Journal of Research in Music Education .Vol. 57 (1), p. 5-15.

Purpose of the study: The purpose of this study was to investigate the effectiveness of amultidimensional assessment rubric when administered across all university-level instrumentaland vocal performance juries during one semester.

Research questions posed: Is there reliability between judges of a multidimensional assessment rubric used in all the

various performance areas of brass, woodwinds, strings, voice, piano, guitar, and percussion? How effective is a multidimensional assessment rubric at rating disparities in performances

of students at different levels of music study? What similarities are there between points awarded on a multidimensional assessment rubric

and through typical single score grading of juries?

Primary methodology of this study: quantitative (non-experimental)

For quantitative (non-experimental) studies:Participants N=359, later reduced to 325Judges N=37Number of Groups: 28, later reduced to 20

Type(s) of Data Collection:Statistical Analyses (list the tests): Cronbachs alpha, reliability rating and reliabilitycoefficients, means and standard deviations, one-way multivariate analysis of variance(MANOVA), Analyses of variance (ANOVA) using the Bonferroni method at the .025 level,Pearson correlations

Methodology Summary:A panel of experienced university faculty developed an assessment rubric that could be appliedto all vocal and instrumental performance areas for use in the study. The resulting rubric waspiloted over the course of two semesters of jury performances and adjusted prior to use in thisstudy.

Faculty panels (judges) of 2, 3, and 4 members listened to undergraduate performance juries andrated student achievement on the rubric without discussion between faculty members. Therewere 20 different panels of judges and 325 participants in the study in one semester of juries at asmall university.

Conclusions:

8/14/2019 Angela Yarnell MEMT 898 Daugherty October 1, 2009 Research Analysis

2/2

Of all the panels, 18 out of 20 showed a significant level of agreement among judges. Two judges on separate panels showed a low level of reliability and were subsequently removed.When these two judges were removed, the reliability coefficients for subcategories on the rubricwere above .70. Reliability coefficients for the overall score from the rubric ranged from .66 to.99, while the reliability coefficients for holistic grading ranged from .56 to 1.0. MANOVArevealed that there were significant differences in scores by year in school. The ANOVA testshowed significance on scale scores, but not on grades by year. Correlations amongsubcategories and overall scores were high (.81 - .89), but when correlations among scaledimensions and grades were tested, they only had moderate (.64 - .72) agreement.

The above tests revealed that rubric scores were both significantly and positively related to yearin school, while typical holistic grades assigned by the same judges panels were not related toschool year. A multidimensional assessment rubric may show student achievement gain eachyear and help track a students real progress in target areas. Single letter grades are nearly equalto rubrics in their ability to sufficiently describe student achievement in a single performance.However, rubrics provide more feedback than a single letter grade can and may encouragestudents to attend to lower ranked subcategories.

Recommendations for future research:A longitudinal study of undergraduate jury performances of the same students over four years isneeded to verify the validity of this rubric. This tool also needs to be applied at other universitiesto determine content validity. The authors further recommend examination of relationshipsbetween scores from this assessment tool and other extant records of student achievement,including standardized test scores.

Evaluation/assessment of this research study:Authors of this study were interested in a single rubric that could be used with reliability andvalidity in all performance areas. While they were able to demonstrate a high degree of interjudge reliability, some performance media may benefit from separate categories specific tothat area. This studys greatest contribution is the significance of rubric scores as they pertain toyear in school, which demonstrates that achievement can be tracked from year to year andspecific feedback can be given to the student on their growth. The fact that holistic grades werenot significantly different than rubric composites is not unexpected. The additional advantages amultidimensional assessment tool can provide are worth the effort in administering such a rubric.

Generalization/transfer possibilities:Specific feedback is the goal of every teaching cycle in a private lesson or ensemble rehearsal.Rubrics developed for use in Kansas choir environments include a state honor choir auditionrubric, a large group music festivals rubric, daily rehearsal rubrics, and others. In Kansas, choirdirectors have advocated for slightly different rubrics for use in vocal situations than theinstrumental rubrics first developed for our use. We have found the extra areas address itemsthat are unique to singing and have been verified for their interjudge reliability. Regardless of similarity between holistic grading and rubric composite scores, students and directors gainmuch from the increased specific feedback that rubrics provide. They help communicateconsistency in grading and provide data for later comparisons. Performance rubrics can also beused in the development of a student portfolio to monitor growth and achievement.