Developing a multiple-document-processing performance assessment

for epistemic literacy

Simon Knight

Karen Littleton

http://sjgknight.com

@sjgknight

Developing a multiple-document-processing performance assessment

for epistemic literacy

Simon KnightKaren Littleton

Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah, Simon Buckingham Shum, Bart Rienties, Fridolin Wild

http://sjgknight.com

@sjgknight

Introduction

• New technologies:

– Foreground issues in literacy – finding & comprehending multi-media

– Afford new opportunity to explore & tackle these issues

What is [Digital] literacy?

• [Digital] literacy –

– Comprehension in context of selecting & processing rich online multimedia content with diverse sources (OECD, 2013)

Tasks

• Two collaborative tasks facilitated by a browser addon

• One group provided with documents; the second group searches on the web

• “A review of the best supported claims around the risks” of a substance (herbicide or food supplement)

7

Friends of the Earth:

PressRelease(Urine

presence)

FoE

Commissioned report

(‘scientific’)(-ve)

Science-Literacy website:

Refutation(+ve)

Farmer’s Weekly

Reprints(+ve)

Related peer-review publication

(Limited risk)

Peer-review publication

Health danger

Reuters

Reprints main claims

Blogger

Critiques journal &

author

Peer-review publication

(Limited health risks)

Peer-review review of literature

(Limited risk to health or

plants)

Peer-review of lit

(Limited risk; control suggestions)

Urine

Health

Agriculture

8

Statins

Citrinincontamination

Concentration

Skills of literacyRouet [39] – students should be taught:• Skill of integration: the ability to

connect prior and new information, including across documents, and including where claims are inconsistent or contradictory

• Skill of sourcing: the ability to identify parameters that characterize the author and conditions of production of the information

• Skill of corroboration: the ability to check information against multiple sources for its accuracy

Skills of literacy

• Communication – particularly meta-discourse involving exploratory discussion of credentials, sources, etc. – is key to collectively authored written outputs

(Goldman and Scardamalia [17, p.260]).

Coagmento Tool

• Chat

• Foreground searches

• Share ‘snippets’

• Etherpad

• Tracks pageviews & copy (/ctrl+c)

Epistemic Commitments

we should try looking on Wikipedia for that

13

A

CB

¬A

B

…………………

14

A

CBy x (2002)

…………………

B

¬ABy y (2014)

BBy Gov (nd)

15

Source features

• Participants given 2 texts of varying quality are more likely to favor high quality sources,

• BUT they make little reference to source features (on average only 1.85 out of 10 features);

• and rarely (<6% of the 154 participants) explicitly use source information for justifying their evaluation of the text’s explanation

Kobayashi [26](2014)

Epistemic commitments

• Multiple Documents—Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet/Rouet & Britt)

• 5 Steps:1. Model construction

2. information need assessment;

3. document processing; - relevance, update needs, seek new information

4. task product creation;

5. task product assessment.

“exploring students’ thought processes during online searching allows examination of personal epistemology not as a decontextualized set of beliefs, but as an activated, situated aspect of cognition that influences the knowledge construction process” (Hofer, 2004, p. 43).

The Lens of Epistemic Beliefs Commitments

Certainty

19

A

C

By x (2002)

…………………

Source

A B CA

BC

Simplicity

Justification

1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?2.The justification for knowledge – does knowledge come from observation, or from rules of

enquiry?3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in

interaction with others?4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute

Known variables (GPA, etc.)• ISEQ

• GPA

• Self-report prior knowledge

• Self-report search skill

• Self-report partner familiarity

Trace constructs• n Pages visited

• n Pages used

• Page-dwell >= 30 seconds

• n Pages visited/N pages visited

Pages visited

Pages used

Collaborative symmetry

Query diversity

Chat data

Exploratory/epistemic

Meta-discourse(authorshipPub dates, etc.)

Topics

Other chat

• n Pages used/N pages used• Query distances• Coded-chat instances (topic-terms,

source-terms, exploratory-terms)• Collaborative-symmetry on <<&^^^

Outcome constructs

• 1-3 score on:

– Topic coverage

– Source diversity (largely ‘3’)

– Source quality/evaluation

– Synthesis

• 1-12 total score

22

Source Diversity

Source Quality/Evaluation

Synthesis Topic coverageOutcome:

(Peer assess?)

ProjectIDISEQPairedGPAPairSearchSkillPair

PriorKnowlPair

PartnerFamilPair

SynthScoreTopicScoreSourDivScoreSourQualScoreTotatlScore

n-pages visited

n-pages usedn-dwelledChat-n-messagesChat-n-explorChat-n-topicChat-n-sourcequeryDists (CIS ONLY)Partner Symmetry

Outcome vars

Process vars

Control vars

24

Predictors?

GPA, topic knowledge, search skill

Source Diversity

Source Quality/Evaluation

Synthesis Topic coverageOutcome:

(Peer assess?)

Pages visited

Pages used

Collaborative symmetry

Query diversity

Chat data

Exploratory/epistemic

Meta-discourse(authorshipPub dates, etc.)

Topics

Other chat

?

• ISEQ - psychometrics, etc.

• Outcome data (IRR)

• Trust

• Predictive value of summary data (e.g. pages->Topics)

• Sequences

• Deep dive – key patterns, proof of concept of those patterns. Critical incidents.

Thank youSimon Knight

http://sjgknight.com

@sjgknight

Knight, S., Littleton, K., (2015). Developing a multiple-document-processing performance assessment for epistemic

literacy. In The 5th International Learning Analytics & Knowledge Conference (LAK15): Scaling Up: Big Data to Big

Impact, Poughkeepsie, NY. http://oro.open.ac.uk/41711/

27

Predictors?

GPA, topic knowledge, search skill

Source Diversity

Source Quality/Evaluation

Synthesis Topic coverageOutcome:

(Peer assess?)

Pages visited

Pages used

Collaborative symmetry

Query diversity

Chat data

Exploratory/epistemic

Meta-data(authorshipPub dates, etc.)

Topics

Sources(http://)

Other chat

@sjgknight www.sjgknight.com

Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search,

Presented at the 2nd Society of the Query Conference, Amsterdam• Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K.,

(2014). Epistemic Networks for Epistemic Commitments. Presented at the International Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39254/

• Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of Learning Analytics. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39226 (HEA presentation version Blog/presentation)

• Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education. Available via The Open University Eprint Archive: http://oro.open.ac.uk/39181/

• Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R. Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research on Teaching Thinking. Routledge.

• In draft publications on computational approaches to discourse analysis + social account of epistemic cognition

ReferencesAnmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemologicalbeliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media Institute: The Open University, UK. Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdfGoldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources: Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Information Age Publishing (IAP).Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning. Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. BehaviorResearch Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centereddesign. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17). Rowman & Littlefield.Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information. Metacognition and Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdfRouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge.Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information? Retrieved from http://dspace.ou.nl/handle/1820/4390Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030

Task1 (warmup)

Please type the answers to the following three prompts in your Task Pad (click in the bar at the top of the browser). You may use the internet to find the answers.

• In 2010 what was the educational expenditure per primary student in [XXX] as a % of GDP?

• In 2010 what was the total health expenditure as a % of GDP in [XXX]?• How much (in US dollars) does a big mac cost in [ZZZ]?

Warmup – variables, – XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana,

Guatemala, El Salvador, Ecuador (list of countries not too far from top of alphabet, with data for 2010)

– Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada, Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index)

If you find the warmup taking too long (over 10 minutes) but you feel you’re now comfortable with using Coagmento, you should move on to Task 1.• Task Submission Procedure [Slide 9]

MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate.Your task is to act as an advisor to an official within the science ministry. You are advising an official on the issues below. The official is not an expert in the area, but you can assume they are a generally informed reader. They are interested in the best supported claims in the documents. Produce a summary of the best supported claims you find and explain why you think they are. Note you are not being asked to “create your own argument” or “summarise everything you find” but rather, make a judgement about which claims have the strongest support.A colleague has already found a number of documents for you to process with your partner, you should use these to extract the best supported claims (without using the internet to find further material).You should:Read the questions/topic areas provided, these will require you to find information and arguments in the documents to present the best supported of these, you should decide with your partner which are best as you read.Group information together by using headings in the EditorYou should work with your partner to explain why the claims you’ve found are the best availableYou should spend about 45 minutes on this taskA review is coming up for the license of Glyphosate, the official would like to know the best supported claims around its risks.A colleague has collected some documents, available from the