Embed Size (px)
Citation preview
Christine Sälzer & Fabian Zehner
PISA 2012 in Germany: Exploring Gender Differences in
ReadingConference: PISA and its meaning for policy and practice around the Baltic Sea, Tallinn, Estonia
Presentation based on: Zehner, F., Goldhammer, F & Sälzer, C. (under review). Further Explaining the Reading Gender Gap: An Automatic Analysis of Student Text Responses in the PISA Assessment.
Gender gap: A classic in Reading Performance
PISA 2009
Different means = gender gap?
• Comparing means in Reading proficiency disguises usefulinformation
• In fact: big overlap of distributionsdespite large mean differences(Stanat & Kunter, 2002)
• Mean scores are not representative of their entiredistribution
à Is the „gap“ really a gap?
Gender gap in Germany
• Over the past 6 PISA cycles, the gap in Reading performancebetween boys and girls was substantial
- 2000: 34 points (OECD, 2002; Stanat & Kunter, 2002)
- 2003: 42 points (OECD, 2004; Schaffner, Schiefele, Drechsel & Artelt, 2004)
- 2006: 42 points (OECD, 2007; Drechsel & Artelt, 2007)
- 2009: 40 points (OECD, 2010; Naumann et al., 2010)
- 2012: 44 points (OECD, 2014; Hohn, Schiepe-Tiska, Sälzer & Artelt, 2013)
- 2015: 21 points (OECD, 2016; Weis, Zehner, Sälzer, Strohmaier, Artelt & Pfost, 2016)
Explanations?
• Our study attempts to explain what may be behind this constantlyfound gap
• Can students‘ open-ended responses to PISA Reading items beused to explain the gender gap?
• Some earlier studies suggest that girls‘ and boys‘ readingproficiency could be an effect of item format x motivation; but effectsizes are very small (Schwabe, McElvany & Trendtel, 2015)
• Differential reading motivation may be due to differential interests(OECD, 2015): Girls prefer fiction (+ 19%) and magazines (+ 14%)Boys prefer newspapers (+ 7%) and comics (+ 10%)
• Strong predictors of reading literacy: Reading engagement andstrategies
Our study
• Taking into account that the score distributions ofgirls and boys in Reading overlap
• A closer look into the responses to open-endedquestions: Do girls‘ and boys‘ responses differ?Can we identify response types?
à Types are then described as „girls‘ type“ and „boys‘ type“ à prototypes
• Using automatic coding (natural languageprocessing, NLP) for analyzing the students‘ open-ended responses
• Identifying similarities and differences
• Assumptions about cognitive processes behind a response
Theory: Cognitive model
• Propositional information from the stimulus text…
…is processed to situation model (Kintsch & van Dijk, 1978; van Dijk & Kintsch, 1983) à holistic model describing which cognitiveprocesses are involved while readers try to understand a text
…distinguishing micro- and macropropositions:Micropropositions carry information on text entities like personsor their relationsMacropropositions contain higher-order information such as a paragraph‘s gist
• Analyzing the students‘ responses, we determine which and howmany propositions they selected and how response features can bemapped to cognitive processes (Graesser & Franklin, 1990; Graesser & Murachver, 1985; Graesser & Clark, 1985)
Automatic system used
• Approach recently published (Zehner, Sälzer & Goldhammer, 2016)
Automatic system used
Material and data
• PISA 2012 sample of Germany (continued with the PISA 2015 data)
• 8 Items assessing Reading literacy
• PISA coding guides to decide which parts of a response arerelevant or irrelevant
Main task
Process in focusaccording toassessmentframework
Total n ofstudent
responses
Analyses (condensed)
1. Proposition Entity Count (PEC): Number of PEs
- How many micro- and macropropositions has thestudent used in the response?
- How many of those propositions were relevant, howmany were irrelevant for a correct response?
Analyses (condensed)
1. Gender type: Responses were semantically grouped (clusters) and distinguished in responses of the „girl type“ and responsesof the „boy type“à Gender type instead of gender split
à Response is taken as an indicator of a response type(„girl“, „boy“) independent from the students‘ gender
à number of clusters determined by human-computer agreement
à for each cluster, ratio boys : girls determined plus 95%-CI (Oranje, 2006; Wilson, 1927)
à types with a significantly dominating gender included
Analyses: Summary
• Contrasting two extreme groups of student responses
• Distinguishing types according to the gender majority in eachcluster (i. e., a cluster dominated by girls will be a „girl-type“ response cluster)
• Student responses are used as a database for drawingconclusions about cognitive processes lying behind theseresponses à principle of IRT and many large-scale studentassessments
Results (selection) IProposition Entity Count for Gender Type and Correctness
• Responses of the boy-type contain less PEs than responses ofthe girl-type
Results (selection) IIProposition Entity Count for Gender Type and Correctness
Results (selection) IIIMicro and Relevance Measures
Results (selection) IVMicro and Relevance Measures
Results: Summary
• Different cognitive types dominated by genders instead ofhomogeneously separated genders
Boy Type- parsimonious selection of PEs
(controlled for correctness)- if response is correct, efficacy
is worthwile, but:- type is relatively consistent
across correct and incorrectresponses à less stablesituation model
- flawed reconstruction (Kintsch & van Dijk, 1978)
Girl Type- Elements in the situation model
are juggled with quite high flexibility (PEC & relevance)- micro level where required- macro level where required
- Question focus and categoryare identified more easily
cognitive types could be an effect of different readingstrategies (Artelt, Naumann & Schneider, 2010)
Limitations
• data limited to Germany only at this pointà Joint research project with appr. 12 other PISA NCs usingfurther languages
• relevance measures are based on coding guides which are not empirically exhaustive (Zehner, Goldhammer & Sälzer, 2015) à relevance is partly underestimated
• operationalizations yield only rough implementation of thesituation modelà improvements through progress in NLPà further features are possible
• expand theory and operationalizations to allow text responsesto be a new source of information for further content-relatedresearch questions
References
Artelt, C., Naumann, J., & Schneider, W. (2010). Lesemotivation und Lernstrategien. In E. Klieme et al. (Eds.), PISA 2009 (pp. 73–112). Münster u.a.: Waxmann.
Graesser, A. C., & Clark, L. F. (1985). Structures and procedures of implicit knowledge (Vol. 17). Norwood, NJ: Ablex.
Graesser, A. C., & Franklin, S. P. (1990). QUEST: A cognitive model of question answering. Discourse Processes, 13(3), 279–303.
Graesser, A. C., & Murachver, T. (1985). Symbolic procedures of question answering. In A. C. Graesser & J. Black (Eds.), The psychology of questions (pp. 15–88). Hillsdale, N. J.: Erlbaum.
Hohn, K., Schiepe-Tiska, A., Sälzer, C., & Artelt, C. (2013). Lesekompetenz ins PISA 2012: Veränderungen und Perspektiven. In M. Prenzel, C. Sälzer, E. Klieme, & O. Köller (Eds.), PISA 2012: Fortschritte und Herausforderungen in Deutschland (pp. 217–244). Münster: Waxmann.
Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363.
Naumann, J., Artelt, C., Schneider, W., & Stanat, P. (2010). Lesekompetenz von PISA 2000 bis PISA 2009. In E. Klieme et al. (Eds.), PISA 2009 (pp. 23–71). Münster u.a.: Waxmann.
OECD. (2002). Reading for change: Performance and engagement across countries. Paris: OECD Publishing.
OECD. (2004). Learning for tomorrow’s world: First results from PISA 2003. Paris: OECD Publishing.
OECD. (2007). PISA 2006: Science competencies for tomorrow’s world: Volume 1: Analysis. Paris: OECD Publishing.
OECD. (2010). PISA 2009 Results: What students know and can do: Student performance in reading, mathematics and science (Volume I). Paris: OECD Publishing.
OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing.
OECD. (2014). PISA 2012 results: What students know and can do (volume I, revised edition, February 2014). Paris: OECD Publishing.
OECD. (2015). The ABC of gender equality in education: Aptitude, behaviour, confidence. Paris: OECD Publishing.
Oranje, A. (2006). Confidence intervals for proportion estimates in complex samples (Vol. RR-06-21). Princeton, N.J.: ETS.
Schwabe, F., McElvany, N., & Trendtel, M. (2015). The school age gender gap in reading achievement: Examining the influences of item format and intrinsic reading motivation. Reading Research Quarterly, 50(2), 219–232.
Schaffner, E., Schiefele, U., Drechsel, B., & Artelt, C. (2004). Lesekompetenz. In M. Prenzel et al. (Eds.), PISA 2003 (pp. 93–110). Münster: Waxmann.
Stanat, P., & Kunter, M. (2002). Geschlechterspezifische Leistungsunterschiede bei Fünfzehnjährigen im internationalen Vergleich. Zeitschrift für Erziehungswissenschaft, 4(1), 28–48.
van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
Weis, M., Zehner, F., Sälzer, C., Strohmaier, A., Artelt, C. & Pfost, M. (2016). Lesekompetenz in PISA 2015: Ergebnisse, Veränderungen und Perspektiven. In Reiss, K., Sälzer, C., Schiepe-Tiska, A., Klieme, E. & Köller, O. (Hrsg.). PISA 2015—Eine Studie zwischen Kontinuität und Innovation (249-284). Münster: Waxmann.
Zehner, F., Goldhammer, F., & Sälzer, C. (2015). Using and improving coding guides for and by automatic coding of PISA short text responses. In Proceedings of the IEEE ICDM Workshop on Data Mining for Educational Assessment and Feedback (ASSESS 2015).
Zehner, F., Sälzer, C., & Goldhammer, F. (2016). Automatic coding of short text responses via clustering in educational assessment. Educational and Psychological Measurement, 76(2), 280–303.
