Teaching and Assessing Discipline-Independent and Discipline-Specific Metacognitive Strategies

Laura Wenk

Assistant Professor of Cognition and Education

Hampshire College

Goals of this talk

Examples of learning challenges and metacognitive strategies Discipline-independent examples Discipline-specific examples

Teaching example for each Assessment

Discipline-independent examples

Intention (goal setting) Reading comprehension (Reciprocal Teaching, PQ4R) Writing to transform ideas (rather than knowledge telling)

e.g. Building explanation Building argument Genre

Checking for confirmation bias and pre-mature closure

Discipline-specific examples

Scientific inquiry, including: Explanatory reasoning in science

Theories, models, hypotheses Thinking with them

Empirical confirmation Interpreting evidence to distinguish among knowledge claims Research design

The helix of inquiry Where from & where to?

Primary research literature skills Reading and understanding Writing about

My courses this semester

CS 122T: Inquiring Minds: Find out what other students think and do (social research and psychology) Understand social research Use primary research literature Conduct a study, manage and analyze data, write it up

CS 208: How People Learn: Introduction to cognition and education Write an argument using the literature (and own

research data)

Why use primary research literature? Engages students’ schemas about:

What makes for a well-designed study What qualifies as evidence Distinction between data and interpretation How data are interpreted

Epistemological change Theory-based explanations are inherently uncertain Results hinge on the details of the research process

Preparation for student’s own research

Challenges to using primary literature

Students can’t find it/evaluate it Aren’t interested in it (at first) Can’t understand it

New genre, vocabulary, assumptions, etc. Difficult to draw the larger lessons from it

It’s about more than just understanding this study

Some ideas to meet the challenges

Going slowly Breaking primary literature skills into

components Metacognitive explicitness

Rubric Using the project to maintain motivation

Questions assigned with a research article:

1) What question is addressed? Explain the relevant past research and ideas that led to it

2) What hypothesis was investigated? Explain how it is related to the research question you discussed in #1 above.

3) How was the study set up? Explain why it was set up this way.

4) What data were collected? Explain why the authors chose these particular data to collect.

Questions assigned with a research article:

5) What were the results? 6) Explain how well the results do (or do not) support the

hypothesis. 7) Explain any alternative explanations for the findings 8) What further research does this study suggest?

Explain why it should be conducted.

In-class activity

Students compare answers in expert groups by question

Groups present “best” responses All discuss what makes for strong answers,

what is appropriate level of detail Meta-conceptual conversations on the

nature of science, design issues, underlying assumptions, interpretation, etc.

QuestionQuestion Articulate and Articulate and explain explain conceptual conceptual issues (well issues (well elaborated)elaborated)

Articulate Articulate conceptual conceptual issues (no issues (no elaboration)elaboration)

Miss important Miss important conceptual conceptual issues and/or issues and/or confused about confused about the sciencethe science

1 through1 through88

Students receive feedback via rubric

Subsequent assignments

Additional common articles Answer questions Student self-assesses with rubric I assess with rubric We compare assessments

Multiple opportunities for modeling and practice with feedback

Sample Student Responses and Assessments

Question Sample responses Assessment and explanation What data were collected? Explain why the authors chose these particular data to collect.

The data collected were the BDI and HOME scores. They chose these tests because they would show not only cognitive development, but also the way in which the environment was affecting them. The BDI tests various levels, thus it gives results in many areas of development…It was also used because it was standardized on 800 children from a wide spectrum of socioeconomic backgrounds, has high correlation coefficients for test-retest reliability, and acceptable content validity. Test developers of the BDI intended it to be used to identify children at risk for developmental handicaps, meaning this test was targeting exactly what the study wanted to see. (Cocaine article, paper #3106)

Elaborated—explains data discussing design considerations. In this response, the student goes beyond reporting simply what data were collected. This student articulates what the different tests are measuring, why these tests are the correct tests to use (sound psychometric properties such as reliability, validity, and standardization and they measure multiple facets of development), and also notes the importance of the environmental measure to compare its effect versus the effect of cocaine exposure.

At 6 to 30 months, children were tested under the Bayley Scales of Infant development. At 30 months, the Preschool Language Scale was used. At 4 years, the Wechsler Preschool and Primary Scale of Intelligence-Revised provided intelligence quotients. At 3 and 5 years, the Battelle Developmental Inventory was used for evaluation of the subjects. It is broken up in five categories: Personal-Social, Adaptive, Motor, Communication, and Cognitive. At 4 years, the home environment was assessed using the Home Observation for Measurement of the Environment (HOME) test. (Cocaine article, paper #3004)

Unelaborated—tells data without elaborating on design considerations. In this response, the student mentions all of the important data that were collected. There is no explanation of the significance of using these particular measures.

At the three-year test the control children weighed more than the cocaine exposed children, and at the 5 year test the control group was slightly younger. Also, at both intervals less of the cocaine exposed kids were in the care of their biological mothers. Both the cocaine-exposed and non-exposed children scored similarly on the Bayley Scales of Infant Development, the Preschool Language Scale, and the Wechsler Preschool and Primary Scale of Intelligence Revised…. (Cocaine article, paper # 3037)

Misunderstood—confuses data with results. This response calls into question whether the student understands the distinction between the data that were collected and the analyses performed on those data.

Question Level of Answer

Pre-score

Post-score

Pre-post Difference

Research Question/Importance Explanatory 24 29 5 Mentions 107 117 10

Misconception 22 20 -2

Hypothesis Explanatory 7 10 2 Mentions 76 115 39*

Misconception 34 12 -22*

Set-up (design) Explanatory 5 24 20* Mentions 120 85 -34*

Misconception 15 12 -2

Data collected Explanatory 7 27 20* Mentions 78 66 -12

Misconception 29 10 -20*

Results supporting hypothesis Explanatory 0 29 29** Mentions 105 56 -49**

Misconception 56 34 22

Alternative explanations Explanatory 5 10 5 Mentions 39 41 2

Misconception 76 54 -22

Future research suggestions Explanatory 2 5 2 Mentions 59 51 -7 Misconception 61 61 0 Totals Explanatory 7 19 12* Mentions 83 76 -7 Misconception 42 29 -13*

N=42 pre-post matched pairs

Study Results - 100 Level Natural Science Classes

First year students can

Read primary research Improve understanding of research design Improve understanding of data

interpretation Improve in distinction between data and

interpretation

My courses this semester CS 122T: Inquiring Minds: Find out what other

students think and do (social research and psychology) Understand social research Use primary research literature Conduct a study and write it up

CS 208: How People Learn: Introduction to cognition and education Write an argument using the literature (and own

research data)

Expectations for student writing

Hampshire courses are writing intensive All students complete a senior thesis

(Division III) Most course projects and the Div III are on

negotiated topics Often requires interdisciplinary arguments

Some challenges to writing an argument

Students don’t know what an argument is Why do we have to argue :-)

Early college students’ writing tends to be descriptive rather than analytical It lacks transitions It lacks explanation of why they are citing someone It ignores complexity The main point is often reached in the concluding

paragraph

Some challenges to writing an argument

Students fear redundancy I already wrote what they found

Students don’t feel qualified to have an opinion Students’ strategies support descriptive writing

Strategies students tend to use Reading articles/chapters and outlining them

Independent judgments about importance of each fact or idea

Not transformative Reading everything before writing Sitting down to write, going back to things they

had read before, and extracting the part they thought was interesting

Stringing ideas together in an order suggested by an outline of topics

Leaving little time for revision

Some strategies to meet the challenge

Write AS you read (micro-writing) About specific ideas as they occur to you Use your own words Include important details (elaboration)

Do a number of these; have an epiphany Write across the shorter pieces (macro-writing) Read out loud (maybe to a friend)

Stop when you find you need to explain something/why it was there and write that explanation down

Hand in for feedback Keep revising with feedback (peer and professor)

Assignments to support new strategies Critical response papers (articles I assign)

Develop a thesis Engage with the article and details of the

points made (of interest) Consider questions raised by the reading

Portfolio of response papers Periodically engage in:

Selection of best piece Assessing its strengths and weaknesses Revising

Assignments to support new strategies

Receive feedback from me (on portfolio and self-assessment)

For final paper Students must write critical response papers for 5

articles they find and select on their topic

No study yet - but…

I’m happier with the writing They know what I mean when I ask for

elaboration or transitions, etc. Seem more able to make connections

across articles

Assessment - Both examples

Formative feedback Explicit criteria (discussed in class/rubric) Timely feedback - adjust Both teacher and self-assessment (helps

students internalize criteria) Use same criteria on multiple assignments

Use the same criteria to judge their projects Success on project requires use of target skills