Rubric for Evaluating a Psychology Research Report

7/31/2019 Rubric for Evaluating a Psychology Research Report

1/14

A rubric for evaluating a psychology research reportGail M. Gottfried, Developmental Science (2007 Instructional Resource Award recipient)

Jordan R. VosmikDrew University

Kathy E. Johnson

Indiana University Purdue University Indianapolis

The APA Guidelines for the Undergraduate Psychology Major (American Psychological

Association, 2007) recommends that students who complete a baccalaureate degree be able to

understand and apply basic research methods in psychology, including research design, data

analysis, and interpretation (Goal 2). In addition, the discipline of psychology has always

valued effective writing practice, and psychologists have worked hard to elucidate standards for

writing through an evolving editorial style manual (American Psychological Association, 2005).

Thus, Goal 7 of the APA Guidelines, Communication Skills, includes the following learning

objectives: a) students should demonstrate professional writing conventions (e.g., grammar,

audience awareness, formality) appropriate to purpose and context (b) students should use


2/14

writing an empirical report can vary dramatically. On many campuses, introductory writing

classes fall exclusively within the domain of the English Department, and instructors of

psychology may consider some aspects of student writing (e.g., grammar, spelling) to fall under

the jurisdiction of campus writing centers. As a result, students often arrive in upper-division

psychology classes with minimal experience with scientific, discipline-specific writing,

including little experience reading and critically evaluating published research in psychology.

These students struggle to learn the culture and conventions of writing for psychology.

Even in classes with direct instruction on APA-style writing, faculty may find the grading

of empirical reports to be extremely time consuming, particularly when they provide students

with detailed feedback on the strengths and weaknesses of their writing (see Willingham, 1990,

for discussion of the importance of such feedback). Additionally, faculty may feel hampered by

the lack of objective criteria for evaluating the content of the report, especially in an area in

which they may not be well versed in the literature (e.g., senior theses or other independent

projects).

We address some of these concerns with the creation of an analytical scoring rubric for

an empirical report. A rubric is an explicit expression of the performance criteria for a particular

product, in combination with detailed criteria for levels of mastery, expressed along a continuum.

It describes what excellent performance should be like, and also shows what developing and

poor performance will look likewith as many gradations as seem useful or appropriate.

Analytic scoring rubrics are especially helpful when the product or behavior being assessed is


3/14

Rubric development

This project began with request for grading systems/rubrics that faculty use for scoring

students experimental projects (complete APA style paper, proposal, oral presentation, or full

literature review with hypothesis)sent by GMG to the Cognitive Development Society Listserv.

This request yielded responses from 21 faculty and graduate student instructors from the United

States and Canada. Most included a checklist of evaluation criteria or items to be included, with

a total point value (e.g., literature review thorough and logically organized? out of 10 pts; see

Mertler, 2001, for a discussion of the distinction between checklists and rubrics), but did not

include any information about the types of answers that merited any given point value (e.g., how

does a literature review that earns 10 pts differ from one that earns 9 pts, and what would be

included or omitted in a literature review that earns 1 or 2 pts?). From these samples (i.e.,

content-related evidence; see Moskal & Leydens, 2000) and from discussion at the

Developmental Science Teaching Workshop (Gottfried, 2007), we compiled an overall list of

key content issues to be included in our rubric.

One faculty member, JRV (second author), responded to the initial email request with a

fully developed rubric for an empirical paper. Her rubric included eight graded sections (i.e.,

Abstract; Introduction Context and Literature Review; Introduction Literature

Advancement; Introduction Hypothesis; Method; Results; Discussion; and Scientific Writing

Style), which could be scored as meeting one of three well-defined achievement levels:

Exceptional (4 points); Effective to Less Effective (3 or 2 points); and Unsatisfactory (1 or 0


4/14

Working together and assisted by the participants in the 2007 Developmental Science

Teaching Workshop, GMG, JRV, and KEJ then modified this rubric, adding content from the

other examples and emphasizing the proficiency levels articulated in Halonen et al.s (2003)

paper on assessing scientific inquiry in psychology. First, we identified three distinct areas of

focus: research skills (i.e., descriptive, conceptualization, and problem solving skills domains,

including the presentation of a theoretically or methodologically interesting hypothesis, a clean

design, and appropriate analytical tools); communication skills (i.e., the content and clarity of the

written report, including selection of relevant and current empirical articles presented in an

organized and clear theoretical framework), and the use of APA style and formatting. We then

expanded the topic list to address 16 distinct components of an empirical paper. Divisions

beyond the original 8 were primarily in the method section (e.g., separately evaluating

participants, materials, procedure, and design) and regarding writing style and APA formatting.

Next, to differentiate among Effective and Less Effective responses, we added a

fourth proficiency level. Our new rubric thus included Advanced, Effective/developing, Less

effective/introductory, and Poor categories. Importantly, we used a criterion-referenced, rather

than a norm-referenced, approach to assessment: Our Advanced level of performance was not

equated with earning an A, but rather with the highest standard of the field of psychology. The

criterion is authentic, in that it is similar to the standards used by editors in evaluating a

submission for publication. In an introductory level class, for example, an instructor is not likely

to expect students to reach the advanced level of performance.


5/14

that faculty using the rubric should assign weighted point values to each section when grading,

based on their own course expectations and goals.

Reliability

We then tested the reliability of the rubric. We collected eight research papers from our

undergraduate students, who agreed to let us use their papers for this project. We included senior

theses, senior capstone projects, and empirical papers for a junior-level research methods class.

At least two of us scored each paper using the rubric; we then computed pairwise percent

agreement as measure of interrater reliability. To our surprise, our reliability ranged from 37%

to 56%. Our follow-up discussions and reflection on content validity led to another revision of

the rubric to include 18 topical categories and more clearly operationalized research skills. We

also more clearly differentiated among research skills and communication skills. GMG and KEJ

then scored two additional papers, but our inter-rater reliability was still under 70%.

Though our attempts to establish high inter-rater reliability using our revised criteria and

more detailed rubric ultimately failed, we believe that the rubric itself is an important asset to

student learning and more effective teaching. We also believe that our challenges introduce a

number of interesting research questions. Thus, rather than revise the rubric again, we have

chosen to present it in its current version, with several suggestions for its use.

Using the rubric: Advice to fellow faculty

First, we strongly believe that a carefully constructed, richly detailed rubric is probably as

effective in helping students to write better research papers as are textbook chapters devoted to


6/14

of proficiency, and with how they might change their writing to attain the next level of

competency. The framework for the rubric provides a checklist of sorts to ensure that students

include the content that is expected, and the anchors aligned with varying levels of

competency enable students to understand exactly what is expected of them in the particular

course that they are taking. Thus we believe that the rubric should be shared with students early

in the semester, and that time should be invested in making sure that the content is well

understood. A good strategy for facilitating this comprehension is peer evaluation we

recommend that students be granted the opportunity to read sections of their peers papers in

class, and to provide feedback to each other using relevant sections of the rubric. One of us (KEJ)

engaged in this activity with groups of students writing honors theses and students commented

that the peer evaluations had very much helped them to reflect upon their own writing and the

ways that it could be improved. Other possible learning experiences include using the rubric to

evaluate published journal articles, student-written articles published by Psi Chi, or departmental

honors theses from previous years.

Second, the rubric (particularly after its been used for a semester or two) cuts down

significantly on the amount of time that instructors must spend providing feedback to students on

their writing. At the same time, the quality of the feedback almost certainly improves, as students

can readily see areas that are relative strengths and weaknesses. Ultimately using the rubric for

evaluation may allow instructors to take the time to provide feedback on multiple drafts of the

same paper, a practice that almost certainly will lead to greater improvement in student writing


7/14

The most challenging aspect of using a rubric effectively is calibrating the ratings of

multiple instructors (or a single instructor and multiple teaching assistants) so that there is

sufficiently high inter-grader reliability. We suggest that such agreement may be far easier to

obtain within departments than across departments from different universities. Our efforts to

achieve similar ratings during our development of the rubric were generally unsuccessful. Yet we

were reading papers that we had not assigned, by students that we had not met and who attended

universities that we had never visited. It is likely that departmental cultures and the missions of

the institutions in which they reside (whether urban comprehensive institutions, research-

extensive universities, liberal arts colleges, or community colleges) shape the criteria that faculty

use to gauge levels of proficiency with writing. Additionally, individual faculty come to the task

with their own writing experiences, proficiency levels, and preferences (for example, JRV still

prefers her original 3-item evaluation scale, rather than the modified 4-item scale we created).

Instructor ratings can also be affected by the idiosyncratic histories of individual students. We

may be a bit more lenient, for example, when a student who has struggled mightily produces a

reasonably coherent piece of written work. An honors thesis mentee who is headed for graduate

school may be evaluated more negatively because we are attempting to instill writing habits that

may ultimately make the process of writing a thesis or dissertation more manageable. We view

the bases for inter-grader reliability (or the lack thereof) to be a highly interesting topic for future

investigations, ideally involving collaborations among instructors from a wide variety of

institutions. We hope that the rubric we have presented here will provide a helpful tool for


8/14

References

American Psychological Association. (2007).APA guidelines for the undergraduate psychology

major. Washington, DC: Author. Retrieved from www.apa.org/ed/resources.html

American Psychological Association. (2005). Publication manual of the American Psychological

Association. Washington, DC: Author.

Brookhart, S. M. (1999). The art and science of classroom assessment: The missing part of

pedagogy. ASHE-ERIC Higher Education Report, 27(1). Washington, DC: The George

Washington University, Graduate School of Education and Human Development.

Gottfried, G.M. (2007, June). Teaching workshop: Teaching developmental science. Santa

Monica, CA.

Halonen, J., Bosack, T., Clay, S., McCarthy, M., Dunn, D. S., Hill, G. W., et al. (2003). A rubric

for learning, teaching, and assessing scientific inquiry in psychology. Teaching of

Psychology, 30, 196208.

Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment,

Research & Evaluation, 7(25). Retrieved July 14, 2008 from

http://PAREonline.net/getvn.asp?v=7&n=25 .

Moskal, B. M., & Leydens, J.A. (2000). Scoring rubric development: validity and reliability.

Practical Assessment, Research & Evaluation, 7(10). Retrieved November 28, 2007 from

http://PAREonline.net/getvn.asp?v=7&n=10 .

Willingham D B (1990) Effective feedback on written assignments Teaching of Psychology
http://www.apa.org/ed/resources.htmlhttp://www.apa.org/ed/resources.htmlhttp://www.apa.org/ed/resources.html


9/14

EVALUATING A PSYCHOLOGY RESEARCH PROJECT

A scoring system

Gottfried, Vosmik, & Johnson (2008)

Original framework provided by Dr. Jordan Vosmik; revised by G. Gottfried, J. Vosmik, and K. Johnson.

Changes and additions based on scoring systems provided by 21 grad student and faculty member volunteers

contacted through the Cognitive Development Society electronic mailing list and Developmental Science pilot

testers. Support for the development of the rubric was provided by an Instructional Research Award from OTRP.

Notes on rubric use:

The system is designed for projects that have an APA-style empirical paper as an assessmentof student learning but can be adjusted for posters as needed. We recommend evaluating thetitle page and the abstract after reading and evaluating the rest of the paper/poster.

Content and form are generally scored separately. For example, inclusion of tables is scoredwith Results; formatting of tables is scored along with APA style. An exception is for

scoring the title page.

Content can be reviewed even in the absence of a formal subsection. For example, a reviewof the study design does not require a distinct Design subsection; rather, information about

the overall design of the study should be apparent from the hypothesis and the Methodsection. Data reduction may be included in Procedure, or in Results. Relatedly, students

may include relevant information but in the wrong section (e.g., discuss materials in the

procedure section). Score the content as if it were in the proper subsection (i.e., description

of the materials), and note the organizational problem when scoring form (i.e., APA styleand/or scientific writing style, as appropriate).

Not all sections will be required or emphasized for all projects; certain sections can beomitted or weighted to reflect the particulars of the class.

The Advanced criteria in this system should not be equated with earning an A, but ratherwith the highest standard of the field of psychology. Some classes may have (and may expect)

no students producing advanced work in a particular category (e.g., design a study in IntroPsych).

It is assumed that students who submit something will earn at least one point, with zeropoints being reserved for students who do not submit that particular element, if required.


10/14


11/14


12/14


13/14


14/14

Documents

Rubric for Evaluating a Psychology Research Report