A Faculty-Student Dialogue on Assessment: Rubrics as Vehicles
for Communicating Learning Expectations in the Field Eric J. Pyle
James Madison University Slide 2 Assessment vs. Evaluation
Assessment involves comparing information gathered from subjects
relative to some established goal or objective (Kizlik, 2009).
These goals, objectives, or outcomes are set in advance, and should
be clear to both instructors and students. Evaluation allows the
establishment and communication of the worth of an activity
(Kizlik, 2009). This worth can be determined by the extent to which
decisions of instructional approaches, arrangements, organization,
etc., are effective in aiding students into reaching the desired
outcomes. Slide 3 Goals vs. Objectives Goals reflect the overall
outcome expected of an learning activity, such a capacity to
generate geologic descriptions of areas; Objectives are the
specific means by which students can demonstrate that they have met
the goal expected of them, such as generating a geologic map,
cross-section, and lithologic descriptions for a given area. Slide
4 Objectives Objectives consist of three parts: Condition refers to
the context of the learning experience, the setting, the materials,
available or to be used; Performance is a statement of what
students are supposed to do, a description of the task or
demonstration; Criterion is the indication that the particular
objective has been met, and to what degree it has been met.
Example: Given a base map and aerial photograph of an area, the
student will produce a geologic map and cross- section that
describes the relative sizes and orientations of the lithologies
and structures for the region. Chiappetta & Koballa, 2006 Slide
5 Assessments and Objectives Knowledge specific knowledge, recall,
terminology, etc.; Skills specific performances, equipment,
tool-use, etc. Dispositions scientific habits of mind, persistence,
question-posing, etc. The struggle is to ensure that the entire
range of learning described in the goals is expressed through
appropriate objectives and related assessment tasks validity. Slide
6 Types of Assessment Objective/Knowledge items - multiple choice,
fill-in- the-blank; uses binary, nominal criterion
Prescribed/Lexical tasks - problems, short answers; procedures as
well as the task are of value; uses ordinal level criterion
measure; Open-Ended/Prototype tasks - essays, artifacts; uses
interval level criterion measure on 0-100 scale. The struggle in
each form is to provide a clear and consistent determination of
mastery of the related objectives reliability Slide 7 The special
case of field learning Expectations are not just limited
expressions of knowledge but the application of that knowledge;
Expectations include acquisition, demonstration, and mastery of
skill sets; Expectations include consistency, persistence,
attention to detail, and synthesis of working hypotheses. Slide 8
The special case of field learning: Knowledge Cognitive levelSample
VerbsEarth Science concept Knowledge Define, describe, identify
rock texture, RFM identification Comprehension Interpolate,
estimate, predict Draw contour lines from elevation data
Application Compute, modify, relate, use Graph a topographic
cross-section AnalysisDiagram, divide, inferPlot fold axis on a map
Synthesis Arrange, generate, design Construct a geologic map from
field data Evaluation Contrast, interpret, appraise Assess
landslide hazards from map data Based on Bloom (1956) Slide 9 The
special case of field learning: Skills Psychomotor level Sample
IndicatorsEarth Science action Imitation Crude reproduction of
action based on observation and minimal practice Determination of
mineral sample physical properties, such as hardness, streak, or
observing cleavage Manipulation Performance from instruction with
attention to form Measurement and data encoding using a Brunton
compass or Jacobs staff Precision Accuracy, proportion, and
exactness in performance, with minimal error Collection of physical
and chemical data at several points along a stream Articulation
Coordinating a series of acts with harmony and consistency
Generating a map from a series of station measurements, plotted on
a base map Naturalization Smooth, natural performance with minimum
of psychic energy Generation of finished maps that reflect multiple
layers of data collection and procedures and coordinate well with
field notes and diagrams. Based on Dave (1975) Slide 10 The special
case of field learning: Dispositions Dispositional Level Sample
IndicatorsEarth Science action Receiving Follow directions, locate,
identify Following along with a field trip guidebook as a part of a
field trip Responding Completes assigned tasks at or above level
required, or for self-satisfaction Once a local geologic map has
been studied, a student seeks out a regional geologic map to see
larger context Valuing Accepts, prefers, and commits to scientific
values More than one measurement of a particular parameter is
sought in each location Organizing Personal values are brought into
line with scientific values Each field investigation is approached
with a set of questions framed on methodologies and possible
outcomes Characterizing Lifestyle adoption indicative of a
preference for scientific values Active seeking of communications
with other students and faculty on geological issues Based on
Krathwohl, Bloom, & Masia (1973) Slide 11 Combining Domains in
Assessments StructureSample TaskDomain-Based Rationales
Objective/Kno wledge Based on observations of current stream
conditions and local erosion and sedimentation patterns, make a
prediction of how the stream changes when flow reaches flood
conditions. Knowledge beliefs from prior knowledge, applied to
novel situation; Skills measurements and observational descriptions
of setting; Dispositions use of more than one parameter in making
the prediction. Prescribed/Le xical Correctly applying terminology
in a lithologic description using texture, and mineralogy, and
internal features or structures. Knowledge recall and appropriate
application of terminology; Skills communicating descriptions in
written or oral form; Dispositions Use of a variety of descriptive
terms in a manner that reflects possible contexts. Open-
Ended/Prototy pe Constructing an accurate cross- section from a
map, or distinguishing the correct cross- section from distracters,
stating reasons for rejection. Knowledge synthesizing an analogy
representing the distribution and orientation of materials; Skills
drafting the cross-section with consistency of measurement, to
scale, from the map; Dispositions Cross-section contains all
necessary detail, drafted in a manner that communicates clearly the
interpretations drawn Based on Smith (1995), Lawrence &
Margolis (1999), Murphy (2002), and Sibley (2005) Slide 12
Application: Field Notes Rubric Condition: note-taking of field
situations, with general expectations of description and supporting
other documentation, such as maps or lithologic descriptions Slide
13 Where is the Dialogue? Exchange begins with sharing the rubrics
ahead of time Criterion can be stated a priori by instructors,
which allows reliable measurement; or Negotiated with students,
which allows for closer validity of measurement to particular or
unique contexts. Student first (and continuing) response is usually
a request for clarification Reply is finer grained not what do you
want, but what do you mean by, allowing more specific examples to
be generated. Next exchange is the assignment of scores Scores are
broken down by the level of mastery for each dimension or
objective; Narrative feedback becomes focused on stated criteria.
Next exchange is student formative response what do they do next
time to improve, or in some cases, revision of prior submission;
Less prominent is the student of response of why did I get this
grade, when subscores relative to descriptors are provided. Slide
14 Other Applications in Field Courses: Maps and Cross-Sections
Slide 15 Other Applications in Field Courses: Lithologic
Descriptions, Memoirs Slide 16 Uses and Extensions These rubrics
have been used with two successive JMU field courses; In general,
their use allows for quicker and more consistent turn-around of
student assignments; But continued discussion among participating
faculty is needed to fine-tune language relative to expectations;
The Knowledge-Skills-Dispositions format is also employed in
student evaluations of instruction, specific to the individual
assignments in the field course. Data analysis is on-going; Rubrics
are frequently uses on assignments in BA-Earth Science classes at
JMU; A general rubric is in use with the JMU program evaluation
Students participate in a performance task, with differential
expectations based on student standing in program. Slide 17
References Bloom, B. S. (1956). Taxonomy of educational objectives,
handbook I: The cognitive domain. New York: David McKay Co., Inc.
Chiappetta, E. L. & Koballa, T. R. (2006). Science instruction
in the middle and secondary schools: Developing fundamental
knowledge and skills for teaching, 6 th ed. New York: Allyn &
Bacon. Dave, R. H. (1975). Developing and writing behavioral
objectives. R. J. Armstrong, ed. Educational Innovators Press.
Kizlik, B. (2009) Measurement, Assessment, and Evaluation in
Education. Website: http://www.adprima.com/measurement.htm,
retrieved 3/10/09. http://www.adprima.com/measurement.htm
Krathwohl, D. R., Bloom, B. S., and Masia, B. B. (1973). Taxonomy
of education objectives, the classification of educational goals,
handbook II: Affective domain. New York: David McKay Co. Inc.
Lawrence, S., and Margolis, E. (1999). Concepts and cognitive
science. In E. Margolis and S. Lawrence (eds.), 3-83. Concepts.
Cambridge, MA: MIT Press. Murphy, G. L. (2002). The big book of
concepts. Cambridge, MA: MIT Press. Pyle, E. J. (2009). The
evaluation of field course experiences: A framework for
development, improvement, and reporting. In S. J. Whitmeyer, D.
Mogk, & E. J. Pyle (Eds.) Field Geology Education: Historical
Perspectives and Modern Approaches. GSA Special Paper 461. Boulder,
CO: Geological Society of America. Pyle, E. J., & Brunkhorst,
B. (2009). Developing and applying the knowledge, skills, and
dispositions needed for effective Earth science teaching. In A.
Collins & N. Gillespie (Eds.) The Continuum of Secondary
Science Teacher Preparation: Knowledge, Questions, and Research
Recommendations. Rotterdam: Sense Publishers. Sibley, D. F. (2005).
Visual abilities and misconceptions about plate tectonics. Journal
of Geoscience Education, 53(4), 471-477. Smith, E. (1995). Concepts
and categorization. In E. Smith and D. Osherson (eds.), Thinking:
An invitation to cognitive science, vol. 3., 3-33. Cambridge, MA:
MIT Press.
