False Dichotomies: Are Conceptual Questions Easier

than Numeric Questions? Faculty Research & Creativity ForumOctober 30, 2008

D. MacIsaac, PhysicsJ. Zawicki, Earth Sciences and Science EducationL. Gomez, PhysicsSUNY Buffalo State CollegeT. Johnson, Erie 1 BOCES, WNYRIC

False Dichotomies 2

• Frameworks• Syllabi• Guides• Blueprints• Benchmarks

• Objective tests• Performance assessments• Portfolios• Teacher Observations• Group Activities• Program Evaluations

Curriculum Standards

Assessment/Evaluation System Instructional Program

alignment

validity correlation

• Instructional styles• Print materials• Equipment• Facilities• Technology• Community

Conventional◦ Difficulty (Facility)◦ Discrimination◦ Response pattern

Rasch Item format Difficulties analyzed in the context of

issues:1

◦ Student◦ Instructional (Teacher, School) ◦ Testing

1NYS Biology Mentor NetworkFalse Dichotomies 3

Types of Analysis

False Dichotomies 4

Test Validity

Student Instruction

Testing

Validity

• You didn’t teach the associated core major understandings.

• You didn’t reinforce the core understandings enough.

• You taught the core content wrong

• Content Knowledge?• Literacy/Reading

Comprehension? • Question interpretation Skills?• Misconception?

– From previous instruction?– From culture contexts?– Insufficient reinforcement?

• Effort?

• Difficulty (Facility) Level?• Discrimination?• Placement on exam?• Visual distraction by nearby

(graphic) items?• Style of Question?• Flawed item?

Students do not “see” action/reaction pairs.Modified Benjamin Bloom:

knowingusing

integrating

False Dichotomies 5

Claim #1 - Newton’s Third Law

10 Earth’s mass is approximately 81 times the mass of the

Moon. If Earth exerts a gravitational force of magnitude F on the Moon, the magnitude of the gravitational force of the Moon on Earth is: (1) F (3) 9F (2) F/81 (4) 81F

False Dichotomies 6

Gravitational interaction

Item

Difficulty

1 2 3 4 NR

3 0.34 1072

1718

66

317

0

Students have a difficult time differentiating between mass and weight.

Claim #2 - Mass/Weight

False Dichotomies 7

A 2.00 kilogram object weighs 19.6-newtons on Earth. If the acceleration due to gravity on Mars is 3.71 meters per second2, what is the object’s mass on Mars?1) 2.64 Kg 2) 2.00 Kg 3) 19.6 N 4) 7.42 N

False Dichotomies 8

Mass/Weight

Item Difficulty 1 2 3 4 NR3 0.38 153 122

0129 1597 74

A horizontal force of 8.0 newtons is used to pull a 20.-newton wooden box moving toward the right along a horizontal, wood surface, as shown.

61. Starting at point P on the diagram in your answer booklet, use a metric ruler and a scale of 1.0 cm = 4.0 N to draw a vector representing the normal force acting on the box. Label the vector FN. [1] D = 0.49

62. Calculate the magnitude of the fricitional force acting on the box. (Show all work.) [2]

D = 0.7263. Determine the magnitude of the net force acting on the box.

[1]

D = 0. 4764. Determine the mass of the box. [1] D =

0.7665. Calculate the magnitude of the acceleration of the box.

[Show all work.] [2]D = 0.80

False Dichotomies 9

Mass/Weight

Sample items 2004-2007 Range of item difficulties…

False Dichotomies 10

Claim #3 - Inverse Square Law

False Dichotomies 11

June 2004 (D=0.59)

False Dichotomies 12

June 2007 (D=0.78)

False Dichotomies 13

June 2004 (D=0.67)

False Dichotomies 14

June 2005 (D=0.67)

False Dichotomies 15

June 2006 (D=0.82)

False Dichotomies 16

June 2005 (D=0.41)

Teachers often underestimate the critical role played by conceptual physics reasoning which is quite difficult to teach and learn, and focus exclusively on numeric problem solving which is often already taught to students in other classes – like in chemistry and mathematics courses. The conceptual questions on NYSED Regents’ Physics exams are routinely amongst the most difficult items for HS physics students, and a common reaction from students and less successful teachers is that these are “trick questions.” However, these conceptual questions lie at the heart of the physics, and calculating physical quantities that cannot be reliably interpreted by students shows both a weakness in physics teaching and physics learning. Calculation emerges from, is validated and interpreted by conceptual understanding, and conceptual learning is not useful in physics unless it can be used to numerically predict natural phenomena.

Additional information: J. Zawicki <zawickjl@buffalostate.edu>

False Dichotomies

Conclusions

We have observed amongst pre-service and new HS Physics Teachers a common belief that physics problems that do not involve mathematical calculations (aka “conceptual problems”) are easier than those involving mathematical calculations (aka “numeric problems”). Teachers are very likely to allocate student instruction and learning resources in physics according upon their beliefs. NY HS Physics teachers are driven by the NYSED Regents’ Physics Exam. Hence, we have selected and analyzed several elementary mechanics questions from about 1500 student scores on various NYSED Regents Physics Exams that challenge teacher perceptions of difficulty and numeric vs conceptual content.

False Dichotomies 18

  The June 2003 Regent’s Physics exams juxtaposed several numeric and conceptual problems including (on the same page) items #12 and #15.

  Item #15, a conceptual problem requiring simple definition (of inertia or mass, part of the definition of momentum) that presented extraneous numeric data was later determined to be the most difficult item on that exam offering – only about thirty eight percent of about a thousand students were able to answer it correctly. Item #12, a standard “cookbook-like” problem applying these same definitions and principles (calculating and conserving momentum in a collision) via multiple simple calculations turned out to be in the easiest third of the exam that year, answered correctly by over seventy percent of the students that year.

Example 1: Momentum, inertia and mass.

False Dichotomies 19

  A significant fraction of the NYSED Regents physics exam items make use of mass and weight calculations. Students have asked to calculate weight (a force) from mass and apply this information in different problems on every recent exam, often in complex multi-step problems as shown in the example. Students do relatively well on these items.Periodically the exam directly challenges the student to differentiate between mass and weight. These items are usually the most difficult items on that offering of the exam.

 

Example 2: Mass and weight.

False Dichotomies 20

NYSED Regents Physics exams prevalently include calculation of inverse square law forces (electrostatic and gravitation), often combined with proportional reasoning and often requiring interpretations of graphic representations of these forces.

  When presented as a direct calculational exercise as seen on item 14 from the June 2007 exam, students perform well, despite the fact that this calculation requires significant expertise with scientific notation and calculators.

  When asked for simple restatements of these gravitational interactions (the definition of any force), students find these questions about twice as difficult, amongst the most difficult on the exam. See item 12 from the June 2005 exam.

Example 3: Gravitation interactions.

False Dichotomies 21