22
A Quantitative Literacy Assessment Rubric Development & Lessons Learned - Stuart Boersma, Central Washington Univ. - Caren Diefenderfer, Hollins University - Shannon Dingman, Univ. of Arkansas - Bernie Madison, Univ. of Arkansas Supported by the National Science Foundation DUE-0715039 Numeracy Article: Quantitative Reasoning in the Contemporary World 3: Assessing Student Learning http://services.bepress.com/ numeracy/

A Quantitative Literacy Assessment Rubric

  • Upload
    merle

  • View
    44

  • Download
    0

Embed Size (px)

DESCRIPTION

A Quantitative Literacy Assessment Rubric. - Stuart Boersma, Central Washington Univ. - Caren Diefenderfer , Hollins University - Shannon Dingman , Univ. of Arkansas - Bernie Madison, Univ. of Arkansas. Development & Lessons Learned. - PowerPoint PPT Presentation

Citation preview

Page 1: A Quantitative Literacy Assessment Rubric

A Quantitative Literacy Assessment Rubric

Development & Lessons Learned

- Stuart Boersma, Central Washington Univ.- Caren Diefenderfer, Hollins University- Shannon Dingman, Univ. of Arkansas- Bernie Madison, Univ. of Arkansas

Supported by the National Science Foundation DUE-0715039

Numeracy Article: Quantitative Reasoning in the Contemporary World 3: Assessing Student Learninghttp://services.bepress.com/numeracy/

Page 2: A Quantitative Literacy Assessment Rubric

Our Course

• Uses case studies centered on one or more media articles

• Each case study contains study questions focusing on the quantitative content of the article

• Student responses often involve critiquing, analyzing, decision making, and explaining.

Making Assessment Difficult!!QRCW Casebook: Case Studies for Quantitative Reasoning: A Casebook of Media Articles, 2nd edition, Pearson Custom Publishing. ISBN13: 978-0-558-19880-0.

Page 3: A Quantitative Literacy Assessment Rubric

Need: Instrument that we could apply to these non-standard student responses.

AAC&U VALUE Project:•Developed for institutional level assessment (not for grading student work)•15 areas, 10 of which are termed Intellectual and Practical skills:• Inquiry and analysis, critical thinking, written

communication, oral communication, reading, quantitative literacy, information literacy, teamwork, problem solving

A Little History

Page 4: A Quantitative Literacy Assessment Rubric

InterpretationRepresentationCalculation

Application/AnalysisAssumptionsCommunication

Page 5: A Quantitative Literacy Assessment Rubric

Alignment of QL VALUE Rubric w/ QRCW course

Missing Competencies?:– Critical Reading– Number sense– Synthesis

Scale– No “zero”– “4” designated capstone

achievement; not a consideration for student work in a QL course

Folded into “Interpretation”

Changed “application/analysis” to “analysis/synthesis”

New Scale: 0-3

Page 6: A Quantitative Literacy Assessment Rubric

  Achievement Level  

Quantitative Literacy Core Competency

3 2 1 0

Interpretation Ability to glean and explain

mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)

Correctly identifies all relevant information.

Correctly identifies some, but not all, relevant

information.

Some relevant information is identified, but none is

correct.

No relevant information identified.

Representation Ability to convert information from

one mathematical form (e.g. equations, graphs, diagrams, tables,

words) into another.

All relevant conversions are present and correct.

Some correct and relevant conversions are present but

some conversions are incorrect or not present.

Some information is converted, but it is irrelevant

or incorrect.No conversion is attempted.

Calculation Ability to perform arithmetical and

mathematical calculations.

Calculations related to the problem are correct and lead to a successful completion

of the problem.

Calculations related to the problem are attempted but either contain errors or are

not complete enough to solve the problem.

Calculations related to the problem are attempted but contain errors and are not complete enough to solve

the problem.

Calculations given are not related to the problem, or no

work is present.

Analysis/Synthesis Ability to make and draw

conclusions based on quantitative analysis.

Uses correct and complete quantitative analysis to

make relevant and correct conclusions.

Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or

partially complete (e.g. there are logical errors or

unsubstantiated claims).

An incorrect quantitative analysis is given to support a

conclusion.

Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.

Assumptions Ability to make and evaluate

important assumptions in estimation, modeling, and data

analysis.

All assumptions needed are present and justified when

necessary.

At least one correct and relevant assumption is given

(perhaps coupled with erroneous assumptions), yet some important assumptions

are not present.

Attempts to describe assumptions, but none of the assumptions described are

relevant.

No assumptions present.

Communication Ability to explain thoughts and

processes in terms of what evidence is used, how it is organized,

presented, and contextualized.

A correct and complete explanation is clearly

presented.

A partially correct relevant explanation is present, but

incomplete or poorly presented.

A relevant explanation is present, but is illogical, incorrect, illegible, or

incoherent.

No relevant explanation is provided.

Page 7: A Quantitative Literacy Assessment Rubric

Use of QL VALUE Rubric in QRCW course

Descriptions of achievement levels not objective enough for our uses:

Uses quantitative analysis of data as the basis for competent judgments, drawing reasonable conclusions…

Uses quantitative analysis of data as the basis for workmanlike judgments, drawing plausible conclusions…

v

3 2

v

Skillfully converts relevant information into insightful mathematical portrayal…

Competently converts relevant information into an appropriate and desired mathematical portrayal

4 3

Page 8: A Quantitative Literacy Assessment Rubric

  Achievement Level  

Quantitative Literacy Core Competency

3 2 1 0

Interpretation Ability to glean and explain

mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)

Correctly identifies all relevant information.

Correctly identifies some, but not all, relevant

information.

Some relevant information is identified, but none is

correct.

No relevant information identified.

Representation Ability to convert information from

one mathematical form (e.g. equations, graphs, diagrams, tables,

words) into another.

All relevant conversions are present and correct.

Some correct and relevant conversions are present but

some conversions are incorrect or not present.

Some information is converted, but it is irrelevant

or incorrect.No conversion is attempted.

Calculation Ability to perform arithmetical and

mathematical calculations.

Calculations related to the problem are correct and lead to a successful completion

of the problem.

Calculations related to the problem are attempted but either contain errors or are

not complete enough to solve the problem.

Calculations related to the problem are attempted but contain errors and are not complete enough to solve

the problem.

Calculations given are not related to the problem, or no

work is present.

Analysis/Synthesis Ability to make and draw

conclusions based on quantitative analysis.

Uses correct and complete quantitative analysis to

make relevant and correct conclusions.

Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or

partially complete (e.g. there are logical errors or

unsubstantiated claims).

An incorrect quantitative analysis is given to support a

conclusion.

Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.

Assumptions Ability to make and evaluate

important assumptions in estimation, modeling, and data

analysis.

All assumptions needed are present and justified when

necessary.

At least one correct and relevant assumption is given

(perhaps coupled with erroneous assumptions), yet some important assumptions

are not present.

Attempts to describe assumptions, but none of the assumptions described are

relevant.

No assumptions present.

Communication Ability to explain thoughts and

processes in terms of what evidence is used, how it is organized,

presented, and contextualized.

A correct and complete explanation is clearly

presented.

A partially correct relevant explanation is present, but

incomplete or poorly presented.

A relevant explanation is present, but is illogical, incorrect, illegible, or

incoherent.

No relevant explanation is provided.

Page 9: A Quantitative Literacy Assessment Rubric

CompetencyNumber

of questions

Percent of

questions

Number of case studies (N=24) which

have at least one question mapped to

competency

Number of case studies which have 50% or

more questions mapped to

competency

Interpretation 152 65% 24 18

Representation68 29% 19 6

Calculation 101 43% 21 9

Analysis/Synthesis

79 34% 20 7

Assumptions 14 6% 10 0

Communication

53 23% 18 4

Frequency and Prevalence of Core Competencies in QRCW Casebook

Page 10: A Quantitative Literacy Assessment Rubric

Reliability of QLAR

• Focus on Case Study 4.3• Readers A and B scored questions 3a, 3b, 3c,

and 3d on 24 student papers (section 1)• Readers C and D scored 2b, 2c, 2d, 3a, and 3d

from 24 papers (section 2)

Page 11: A Quantitative Literacy Assessment Rubric

Question Competencies 2b Assumptions 2c Assumptions 2d Assumptions 3a Representation, calculation,

analysis/synthesis, communication3b Interpretation, analysis/synthesis 3c Interpretation, analysis/synthesis 3d Representation, calculation

Core Competency for Study Questions

Page 12: A Quantitative Literacy Assessment Rubric

Reliability Test 1: Readers A and B

QL Competency # of disagreements

N % disagreement

Interpretation 1 48 2.08%Representation 1 48 2.08%Calculation 3 46 6.25%Analysis/Synthesis 2 72 2.78%

Communication 0 24 0%

Disagreement Level Per Core Competency

Overall: •Readers A and B disagreed on 7/238 = 2.94% of assessments.•Average of 7 differences: 1.23

Page 13: A Quantitative Literacy Assessment Rubric

Reliability Test 2: Readers C and DDisagreement Level Per Core Competency

QL Competency # of disagreements

N % disagreement

Assumptions 2 42 4.76%

Representation 2 28 10.71%

Calculation 6 28 21.43%

Analysis/Synthesis

0 14 0%

Communication 5 14 35.7%

Overall: •Readers C and D disagreed on 15/126 = 11.9% of assessments.•Average of 7 differences: (just over) 1

Page 14: A Quantitative Literacy Assessment Rubric

Added Value of QLAR

1. Allows one to align student prompts with an organized thought process as represented by the rubric

To solve a complex problem a student needs to read and glean information, convert the information into a usable form, calculate, make or draw a conclusion, and then communicate the result via explanation.

Scaffold student learning by sorting case studies by competency: Start with Interpretation & Representation and build up to more complex prompts.

Students often do not recognize these as components of a strong response. Guiding students through mastery of these competencies is crucial!

Page 15: A Quantitative Literacy Assessment Rubric

Added Value of QLAR

2. Improved student prompts.

Find out how the Standard & Poor’s 500 stock index is computed.

Describe how the Standard & Poor’s 500 stock index is computed.

Explain your reasoning!

Be sure to differentiate between quantitative information taken directly from the article and extra assumptions you made.

Justify any extra assumptions.

Use a quantitative argument to…[answer some question]

Page 16: A Quantitative Literacy Assessment Rubric

QLAR   Achievement Level  

Quantitative Literacy Core Competency

3 2 1 0

Interpretation Ability to glean and explain

mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words)

Correctly identifies all relevant information.

Correctly identifies some, but not all, relevant

information.

Some relevant information is identified, but none is

correct.

No relevant information identified.

Representation Ability to convert information from

one mathematical form (e.g. equations, graphs, diagrams, tables,

words) into another.

All relevant conversions are present and correct.

Some correct and relevant conversions are present but

some conversions are incorrect or not present.

Some information is converted, but it is irrelevant

or incorrect.No conversion is attempted.

Calculation Ability to perform arithmetical and

mathematical calculations.

Calculations related to the problem are correct and lead to a successful completion

of the problem.

Calculations related to the problem are attempted but either contain errors or are

not complete enough to solve the problem.

Calculations related to the problem are attempted but contain errors and are not complete enough to solve

the problem.

Calculations given are not related to the problem, or no

work is present.

Analysis/Synthesis Ability to make and draw

conclusions based on quantitative analysis.

Uses correct and complete quantitative analysis to

make relevant and correct conclusions.

Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or

partially complete (e.g. there are logical errors or

unsubstantiated claims).

An incorrect quantitative analysis is given to support a

conclusion.

Either no reasonable conclusion is made or, if present, is not based on quantitative analysis.

Assumptions Ability to make and evaluate

important assumptions in estimation, modeling, and data

analysis.

All assumptions needed are present and justified when

necessary.

At least one correct and relevant assumption is given

(perhaps coupled with erroneous assumptions), yet some important assumptions

are not present.

Attempts to describe assumptions, but none of the assumptions described are

relevant.

No assumptions present.

Communication Ability to explain thoughts and

processes in terms of what evidence is used, how it is organized,

presented, and contextualized.

A correct and complete explanation is clearly

presented.

A partially correct relevant explanation is present, but

incomplete or poorly presented.

A relevant explanation is present, but is illogical, incorrect, illegible, or

incoherent.

No relevant explanation is provided.

Page 17: A Quantitative Literacy Assessment Rubric

Interpretation Ability to glean and explain mathematical information presented in various forms

(e.g. equations, graphs, diagrams, tables, words)

3Correctly identifies all relevant information.

2Correctly identifies some, but not all, relevant information.

1Some relevant information is identified, but none is correct.

0No relevant information identified.

Page 18: A Quantitative Literacy Assessment Rubric

Representation Ability to convert

information from one mathematical form (e.g.

equations, graphs, diagrams, tables, words)

into another.

3All relevant

conversions are present and correct.

2Some correct and

relevant conversions are

present but some conversions are incorrect or not

present.

1Some information is converted, but it

is irrelevant or incorrect.

0No conversion is attempted

Page 19: A Quantitative Literacy Assessment Rubric

Calculation Ability to perform arithmetical and

mathematical calculations.

3Calculations related to the problem are correct and lead to a successful

completion of the problem.

2Calculations related to the problem are

attempted but either contain errors or are not complete enough

to solve the problem.

1Calculations related to

the problem are attempted but contain

errors and are not complete enough to solve the problem.

0Calculations given are not related to

the problem, or no work is present.

Page 20: A Quantitative Literacy Assessment Rubric

Analysis/Synthesis Ability to make and draw

conclusions based on quantitative analysis.

3Uses correct and complete

quantitative analysis to make relevant and correct conclusions.

2Quantitative

analysis is given to support a relevant conclusion but it is either only partially correct or partially complete (e.g. there are logical errors or

unsubstantiated claims).

1An incorrect

quantitative analysis is given to support a

conclusion.

0Either no

reasonable conclusion is made or, if present, is not

based on quantitative

analysis.

Page 21: A Quantitative Literacy Assessment Rubric

Assumptions Ability to make and evaluate important

assumptions in estimation, modeling, and data

analysis.

3All assumptions needed are present

and justified when necessary.

2At least one correct

and relevant assumption is given

(perhaps coupled with erroneous

assumptions), yet some important

assumptions are not present.

1Attempts to describe

assumptions, but none of the assumptions

described are relevant.

0No assumptions

present.

Page 22: A Quantitative Literacy Assessment Rubric

Communication Ability to explain

thoughts and processes in terms of what evidence is used, how it is organized,

presented, and contextualized.

3A correct and complete explanation

is clearly presented.

2A partially correct

relevant explanation is present, but

incomplete or poorly presented.

1A relevant explanation

is present, but is illogical, incorrect,

illegible, or incoherent.

0No relevant

explanation is provided.