Dimensional Standard Alignment in K-12 Digital

Libraries

Assessment of Self-found vs.

Recommended Curriculum

Byron MarshallRené Reitsma

Oregon State UniversityMalinda Zarske

University of Colorado

http://www.teachengineering.org/ Oregon State University

Zzz...

The Goal: Lessons For Standards

AAAS

NC

MA

60,000+ constantly changing US Science and Math Standards!

Just Call me “Doc”

TX

Edu Stds Oregon

COProviders of educational resources use experts to “align” educational resources (docs) to educational standards. These expert mappings are either used directly or used to train retrieval systems.

CO

MA

NC

http://www.teachengineering.org/ Oregon State University

The Problem: How Do We Know a Resource Aligns with a Standard?

Alignment Considerations:Concepts!Grade LevelAffect (Kids like it!)Extras (exercises,

worksheets, examples, links)

Cost

CO Std.

?

No

Yes

Do they agree? No. But are they thinking the same thing?

http://www.teachengineering.org/ Oregon State University

Collecting Nuanced Training Data

Binary (Yes/No) and singular (one dimensional) judgments produce low IRR. Instead, we asked:

Questions (paraphrased) MatchDimension

MatchCategory

motivational or stimulating (interesting, appealing, engaging) Motivation Affective

concepts, keywords, terms, and definitions from the std ConceptsContent

interesting/important background material related to the std Background

grade level is task-appropriate or I can easily adapt it Grade Level Object

I can use non-text extras(s); figures, tables, images, videos Non-textuals

Situational

I can use the real-world examples Examples

I can use the hands-on, active engineering activities Hands-on

I can use attachments; score sheets, rubrics, questions, etc. Attachments

I can use references or Internet links to other materials References

Overall, this document is relevant to this teaching assignment Overall Alignment

See Saracevic (2007) in JASIST for more on these match categories

strongly disagreedisagree N/A

somewhat disagree

somewhat agree

agreestrongly agree

http://www.teachengineering.org/ Oregon State University

Differentiating Judgments by Task

Self-FoundPlease Rate the Relevance of a Self-Found Doc/Std Pair

Recommended What do you think of a given Doc/Std Pair?

http://www.teachengineering.org/ Oregon State University

Question: Do People Evaluate Self-Found Items Differently than Recommended ones?

Experiment:◦43 Teachers, 934 standard/doc pairs◦71 educational standards

Two tasks:◦Find docs for a std and rate them (Self-Found)◦Rate doc/std pairs suggested by others

(Recommended)

http://www.teachengineering.org/ Oregon State University

Yes, Self-found Pairs are Scored Differently than Recommend Ones

People judged differently when asked to find docs for stds and rate the alignment of as opposed to when asked to rate provided doc/std pairs◦Two solid but different linear regression models

emerged◦User behavior as observed in the Web logs was

different

http://www.teachengineering.org/ Oregon State University

Different Models of Relevance

Self-Found Concepts Background Grade Activities Attachments Examples

Coefficient .227 .346 .295 .199 .183

Cumulative R2 .338 .564 .653 .705 .788

Dependent variable: “Overall Alignment” Independent variables are our dimensions

Recommended Concepts Background Grade Activities Attachments Examples

Coefficient .343 .474 .159 .123

Cumulative R2 .608 .703 .733 .734

Situational Match

Object Match

Content Match

Both Models Have ‘Good’ Explanatory

Power

All coefficients statistically significant at p<.01

Activities and Attachments did not Influence the

Decision for Recommended Pairs

Content Still Dominates

http://www.teachengineering.org/ Oregon State University

User Behavior When Searching and Evaluating vs. Evaluating only

DL Activity Log indications:◦ Searchers opened 1.97 .pdfs, .docs, & worksheets when

rating self-found document/std pairs◦ Raters viewed 1.37 when deciding on a provided pair

Scores on one dimension “bled-over” more to other dimensions for recommended pairs (that is, we saw lower correlation between dimensions on self-found items)

http://www.teachengineering.org/ Oregon State University

So What?

Self-Found Vs. Recommended Resources

Adjust user interfaces?Incorporate training data differently

http://www.teachengineering.org/ Oregon State University

Thanks! Questions?