Running Head: DBQ ASSESSMENTS
Generating Student Performance Data with Document-Based Question (DBQ) Assessments:
Results from Two Urban Schools
by
Scott M. Petri
Los Angeles Unified School District
Submitted for TSU International Conference on Teaching and Learning
December 13, 2013
Track 1: Learning /Teaching Methodologies & Assessment
Presentation: This presentation and paper will describe how educators can incorporate the Common Core Standards for Writing into History-Social Science instruction with Document-
based Question Assessments (DBQs). This process will also generate new metrics and evaluation techniques that can improve the study of student writing.
Format: Interactive multi-media with slides displaying data, audio clips of student perceptions,
video clips of student & teacher reflections, samples of student work, and an activity for participants to evaluate an online peer-review rubric with their mobile devices.
Author Note:
Scott M. Petri, Ed. D.
6050 Colfax Avenue, North Hollywood, CA 91606.
[email protected] 818-319-2581
DBQ ASSESSMENTS
2
Abstract
This pilot was conducted at two PHABO secondary schools in Los Angeles. Both schools had
API scores in the lowest decile of public schools in the state of California. A series of Document
Based Question Assessments (DBQs) were administered to a sample of students from each
school (N=315). The resulting performance data, additional factors, and implications for
Common Core writing instruction in Social Studies suggest that educators will be successful in
implementing common core writing tasks if they are supported by instructional leaders who
explicitly teach writing scaffolds across the curriculum, endorse the use of peer review grading
policies and incorporate revision memos into the writing process, and allow teachers to
collaboratively develop, design, and assess document based question assessments into all social
studies subjects.
DBQ ASSESSMENTS
3
Generating Student Performance Data with Document-Based Question (DBQ) Assessments: Results from Two Urban Schools
The Common Core State Standards call for teachers to emphasize argumentative and
informative writing into all subjects. Many teachers across the content areas are unsure how to
respond to these new standards. Should secondary social studies teachers stop delivery of subject
content to explicitly teach spelling, vocabulary, and sentence construction? Should professional
learning communities devote a specific amount of time to writing instruction in each subject?
How many writing projects should be delivered in each subject? Educators will struggle with
these questions over the next few years as they implement the Common Core writing and literacy
standards, however, this article will present a method for how teachers can implement the
Common Core Standards for writing into History-Social Science instruction immediately.
After a brief review of the literature on writing and formative assessments, it will report
the results of a pilot program that was conducted at two Predominantly Hispanic, Asian, Black
and Other (PHABO) secondary schools in California. Both schools had API scores in the lowest
decile of public schools in the state of California. A series of Document-Based Question (DBQ)
assessments were administered to a sample of 315 students at the two different high schools. The
resulting performance data and implications for Common Core writing instruction in Social
Studies will be discussed in this article.
Literature Review
Under Common Core, all teachers need to be writing teachers (Rothman, 2011). Social
Studies teachers can increase student literacy skills by incorporating document-based questions
as formal assessments. DBQ word production may be used as a goal setting strategy to help
communicate a key measure of student engagement to parents and students. In short, we can use
DBQ ASSESSMENTS
4
our students writing to show them where they are and use data from a national sample to show
them where they need to be.
There is concern that a majority of adolescents do not develop the competence in writing
they need to be successful in school, the workplace, or their personal lives (Graham & Perin,
2007). Blanton (1986) found English language learners (ELL) students often respond to each act
of writing as if it were a test, denying themselves space to practice with written language. ELL
students want to get it right the first time, a perception that often prevents them from becoming
proficient writers. Rogers & Graham (2008) found goal setting for productivity effective in a
meta-analysis of single subject writing interventions. Other researchers (Bissex & Bullock, 1987;
Calkins, 1994; Graves, 1983;) have noted a connection between increased reading and writing
and higher levels of academic achievement. Hence, Common Core and an increasing number of
assessments including the ACT, CRWA, and SAT employ writing-from-sources tasks that
integrate reading and writing.
De La Paz (2005) compared 8th-grade students (N=70) in an integrated social studies and
language arts unit designed to promote historical understanding and argumentative writing to a
control group of students (N=62) who did not receive writing intervention or instruction. Results
indicated the students who demonstrated mastery of the target strategies during instruction wrote
historically more accurate and more persuasive essays regardless of their initial learning profile.
Similarly, De La Paz & Felton (2010) compared 11th grade students who learned a pre-
writing strategy (N=81) for composing argumentative essays related to historical events to a
control group (N=79) that read the same primary and secondary source document sets. They
found that the essays written by students who received pre-writing instruction were longer, were
rated as having significantly greater historical accuracy, were significantly more persuasive, and
DBQ ASSESSMENTS
5
claims and rebuttals within each argument became more elaborated. The word count for the pre-
writing instruction group increased from 195.32 to 327.86 on average, an increase of 132.54
words. Yet for the control group, word production only increased by 14.45 words.
This research suggests that writing instruction that is focused on goal setting strategies,
argumentative claims and rebuttals, and historical accuracy may be effective when introducing
common core writing tasks to students. Hence, instructional leaders should encourage teachers
to design, develop, and analyze DBQs as formative assessments in common planning time, or
department professional development.
Complex writing assignments, or DBQs, are essential for improving adolescent literacy
(Fisher & Frey, 2007). DBQ units align with plans for increasing writing proficiency, critical
thinking, and creating a college-going culture. DBQs can be designed to give students a preview
of Advanced Placement curriculum. Increased use of DBQs should lead to greater English
proficiency and help students avoid costly and demoralizing remedial coursework that has an
adverse effect on college completion rates. DBQs should be jointly developed and graded by
History and English teachers to ensure that students will meet the new Common Core standards
for Writing. Students may be more motivated when they get credit in both classes for the same
assignment. This credit can be given in multiple stages for planning, writing, and revising
DBQs.
The College and Career Readiness Anchor Standards in Writing (See Figs. 1 & 2)
demand that students are able to write arguments on discipline-specific content, developing
claims and counterclaims, while establishing a formal tone and objective style citation. Using
controversial, or debatable content, teaching students to write in the third person, and using their
historical content knowledge to qualify them as an expert may provide motivation when
DBQ ASSESSMENTS
6
teaching this type of writing.
Figure 1
64
| 6
-12
| HIS
TOR
Y/SO
CIA
L ST
UD
IES
, SC
IEN
CE
, AN
D T
EC
HN
ICA
L S
UB
JEC
TS |
WR
ITIN
G
WHST
COMMON CORE STATE STANDARDS FOR ENGLISH LANGUAGE ARTS & LITERACY IN HISTORY/SOCIAL STUDIES, SCIENCE, AND TECHNICAL SUBJECTS
Writing Standards for Literacy in History/Social Studies, Science, and Technical Subjects 612The standards below begin at grade 6; standards for K5 writing in history/social studies, science, and technical subjects are integrated into the K5 Writing standards. The CCR anchor standards and high school standards in literacy work in tandem to define college and career readiness expectationsthe former providing broad standards, the latter providing additional specificity.
Grades 68 students: Grades 910 students: Grades 1112 students:Text Types and Purposes1. Write arguments focused on discipline-specific
content.a. Introduce claim(s) about a topic or issue,
acknowledge and distinguish the claim(s) from alternate or opposing claims, and organize the reasons and evidence logically.
b. Support claim(s) with logical reasoning and relevant, accurate data and evidence that demonstrate an understanding of the topic or text, using credible sources.
c. Use words, phrases, and clauses to create cohesion and clarify the relationships among claim(s), counterclaims, reasons, and evidence.
d. Establish and maintain a formal style.e. Provide a concluding statement or section
that follows from and supports the argument presented.
1. Write arguments focused on discipline-specific content.a. Introduce precise claim(s), distinguish the
claim(s) from alternate or opposing claims, and create an organization that establishes clear relationships among the claim(s), counterclaims, reasons, and evidence.
b. Develop claim(s) and counterclaims fairly, supplying data and evidence for each while pointing out the strengths and limitations of both claim(s) and counterclaims in a discipline-appropriate form and in a manner that anticipates the audiences knowledge level and concerns.
c. Use words, phrases, and clauses to link the major sections of the text, create cohesion, and clarify the relationships between claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims.
d. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.
e. Provide a concluding statement or section that follows from or supports the argument presented.
1. Write arguments focused on discipline-specific content.a. Introduce precise, knowledgeable claim(s),
establish the significance of the claim(s), distinguish the claim(s) from alternate or opposing claims, and create an organization that logically sequences the claim(s), counterclaims, reasons, and evidence.
b. Develop claim(s) and counterclaims fairly and thoroughly, supplying the most relevant data and evidence for each while pointing out the strengths and limitations of both claim(s) and counterclaims in a discipline-appropriate form that anticipates the audiences knowledge level, concerns, values, and possible biases.
c. Use words, phrases, and clauses as well as varied syntax to link the major sections of the text, create cohesion, and clarify the relationships between claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims.
d. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.
e. Provide a concluding statement or section that follows from or supports the argument presented.
Figure 2
65
| 6
-12
| HIS
TOR
Y/S
OC
IAL
STU
DIE
S, S
CIE
NC
E, A
ND
TE
CH
NIC
AL
SU
BJE
CTS
| W
RIT
ING
WHST
COMMON CORE STATE STANDARDS FOR ENGLISH LANGUAGE ARTS & LITERACY IN HISTORY/SOCIAL STUDIES, SCIENCE, AND TECHNICAL SUBJECTS
Writing Standards for Literacy in History/Social Studies, Science, and Technical Subjects 612
Grades 68 students: Grades 910 students: Grades 1112 students:Text Types and Purposes (continued)2. Write informative/explanatory texts, including
the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic clearly, previewing what
is to follow; organize ideas, concepts, and information into broader categories as appropriate to achieving purpose; include formatting (e.g., headings), graphics (e.g., charts, tables), and multimedia when useful to aiding comprehension.
b. Develop the topic with relevant, well-chosen facts, definitions, concrete details, quotations, or other information and examples.
c. Use appropriate and varied transitions to create cohesion and clarify the relationships among ideas and concepts.
d. Use precise language and domain-specific vocabulary to inform about or explain the topic.
e. Establish and maintain a formal style and objective tone.
f. Provide a concluding statement or section that follows from and supports the information or explanation presented.
2. Write informative/explanatory texts, including the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic and organize ideas,
concepts, and information to make important connections and distinctions; include formatting (e.g., headings), graphics (e.g., figures, tables), and multimedia when useful to aiding comprehension.
b. Develop the topic with well-chosen, relevant, and su!cient facts, extended definitions, concrete details, quotations, or other information and examples appropriate to the audiences knowledge of the topic.
c. Use varied transitions and sentence structures to link the major sections of the text, create cohesion, and clarify the relationships among ideas and concepts.
d. Use precise language and domain-specific vocabulary to manage the complexity of the topic and convey a style appropriate to the discipline and context as well as to the expertise of likely readers.
e. Establish and maintain a formal style and objective tone while attending to the norms and conventions of the discipline in which they are writing.
f. Provide a concluding statement or section that follows from and supports the information or explanation presented (e.g., articulating implications or the significance of the topic).
2. Write informative/explanatory texts, including the narration of historical events, scientific procedures/ experiments, or technical processes.a. Introduce a topic and organize complex ideas,
concepts, and information so that each new element builds on that which precedes it to create a unified whole; include formatting (e.g., headings), graphics (e.g., figures, tables), and multimedia when useful to aiding comprehension.
b. Develop the topic thoroughly by selecting the most significant and relevant facts, extended definitions, concrete details, quotations, or other information and examples appropriate to the audiences knowledge of the topic.
c. Use varied transitions and sentence structures to link the major sections of the text, create cohesion, and clarify the relationships among complex ideas and concepts.
d. Use precise language, domain-specific vocabulary and techniques such as metaphor, simile, and analogy to manage the complexity of the topic; convey a knowledgeable stance in a style that responds to the discipline and context as well as to the expertise of likely readers.
e. Provide a concluding statement or section that follows from and supports the information or explanation provided (e.g., articulating implications or the significance of the topic).
3. (See note; not applicable as a separate requirement)
3. (See note; not applicable as a separate requirement)
3. (See note; not applicable as a separate requirement)
Note: Students narrative skills continue to grow in these grades. The Standards require that students be able to incorporate narrative elements e"ectively into arguments and informative/explanatory texts. In history/social studies, students must be able to incorporate narrative accounts into their analyses of individuals or events of historical import. In science and technical subjects, students must be able to write precise enough descriptions of the step-by-step procedures they use in their investigations or technical work that others can replicate them and (possibly) reach the same results.
Developing the Pilot Program
In order to strengthen social studies teachers abilities to support writing across the
curriculum, several professional development seminars were developed to implement a
Document Based Questions (DBQ) assessment program. Teachers were skeptical about
assigning more frequent writing tasks and creating more work for themselves. To address these
concerns, revision memos and peer review rubrics were employed, so that teacher workloads
would not significantly increase. Mean word counts were charted for each class period. Essays
were skimmed by teachers to make sure students were on topic. After administering five DBQs,
students significantly increased their word production (See Figs. 3 & 4). Student performance
DBQ ASSESSMENTS
7
was charted for each writing task so parents could see where their student performed in relation
to their peers (See Fig 5). At the conclusion of this exercise, all of the teachers agreed to
implement DBQs into their classroom assessment practices the following semester.
Figure 3
!"#$%&'#"$()*$&'*#&+"(#&,-&."+"#/&0&1/($*2/%&&
3456789&:*,;&*$C?2@ &6BD&
7&
677&
DBQ ASSESSMENTS
8
Tenth grade World History students were given a series of complex writing tasks or
DBQs. In order to measure student effort, words produced in a 60 minute time period were
counted and analyzed on a per class basis. The results from the final writing task were compared
to the totals from the first writing task to determine how much student effort improved. It is
important to note that there were a total of 106 members of the tenth grade cohort, but only 73
writing tasks were completed on the first and last administrations of in-class DBQs. Student
absences were problematic and student motivation tended to be low, particularly during complex
writing tasks. DBQ teachers may reverse these factors by constantly displaying student effort
data in class, scaffolding pre-writing organizational tasks, and by increasing the number of DBQ
assessments given each semester.
Figure 4
!"#$%&'%()"(*+$,(-$+,.#/+"(
0(
100(
200(
300(
400(
5000(
5100(
5( 6( 7( 8( 9( 55( 56( 57( 58( 59( 15( 16( 17( 18( 19( 65( 66( 67( 68( 69( 25( 26( 27( 28( 29( 75( 76( 77( 78( 79( 35( 36( 37( 38( 39( 85( 86(
:;?(
:%&"@ ( (573(
:%,)&"@ (571(
A+B@ (031(
C)DE@ (686(
By May 2012, the word production generated by this cohort of students had increased to a low of 76 words per hour and a high of 990 words per hour.
The median was 315 words per hour
and the cohort mean was 342 words per hour.
This level of production was 68 words
per hour below the national mean. These results indicate that additional
practice has benefitted students and further practice could lift word production levels over the national mean.
Complicating Factors
At the end of the Spring 2012 semester, 105 students completed the class with 29 of these
students failing, yielding a course pass rate of 72% and a course failure rate of 28%. These
students had an average of 9.69 absences and 7.87 tardies. Assuming a loss of 82 minutes for
DBQ ASSESSMENTS
9
each class period and 15 minutes lost due to each tardy, this resulted in the loss of over 912
minutes, or 15 hours of instruction in Social Studies.
Cohort II
This study was replicated at a second high school location with a larger cohort of 216
students, 121 were enrolled in 10th grade World History, and 95 were enrolled in 11th grade US
History. Word production was tracked for three DBQ assignments. The remaining DBQs were
peer graded by students using rubrics designed for evaluating thesis statements and use of textual
evidence. Students who demonstrated increases in word production were given bonus points in
addition to the points awarded for the assignment. Also, students were able to submit revisions
of their DBQ essays for extra-credit during the semester.
Figure 5
!"#$
%$ %&&$%%'$
%("$ %("$%'%$
()"$($
(%"$ ((*$ (*#$ (*"$ (')$ ('!$ ('+$ ('"$ (+"$*#&$
*%"$
*+"$ '))$ ')!$'#"$ '&*$
'%+$ '%"$''#$ ''"$
,-.//0$12341$
-5674880$-9648:42;$
827.2150$;;/770$,26429$
82;-9/?0$82,
>/7$
:4/66/?0$/;64D>/$
72.5B0$,2642;;/$
4.7/84280$E6/11B$
E4;72B85;0$=4727$
:>-@/60$89/,26$
A/6/?0$-9648:5A9/6$
F47742,
80$12;1622615$
957D>4;0$2;:95;B$
DBQ ASSESSMENTS
10
period and 15 minutes lost due to each tardy, this resulted in the loss of approximately 648
minutes, or 11 hours of instruction in Social Studies. See (Figs. 7 & 8) for a comparison of the
final grades from both cohorts.
Conclusion
This article has described a Common Core writing instruction program that requires the
students to work harder than the teachers. The teacher delivers controversial social studies
content from multiple sources to students. The teacher demonstrates strategies for detecting bias,
evaluating rhetoric, and determining validity in a series of texts. The students then write an
argument in response to a complex question, where there is no correct answer. They utilize
sections of the texts to strengthen the claims in their argument. Then, the student writing is
assessed by an elbow partner. By utilizing peer-reviewed feedback, teachers are free to use their
instruction time encourage student debate and discussion on the topic, to demonstrate pre-writing
strategies, and to provide examples of effective student writing.
Figure 7
!"#"$%&'&()*+,&-$+./&0)1%$)234"*&
567898:&!"#$%&' (' )' *' +' ,'
8& 9& ;& & 9& ?&
DBQ ASSESSMENTS
11
that increase student engagement, lower tardy and absence rates, and increase student literacy
skills. DBQ instruction may boost student literacy rates and student achievement overall,
however, unless carefully scaffolded, this may result in increased number of incomplete
assignments and increased course failure rates, particularly among male students (See Figs. 9 &
10). Additional research may shed light on how to ameliorate this problem.
Figure 9
!"#"$%&''&()*+*,-&.%/01,%2&&
34&51,01$&67899:&
40! 15 !
Figure 10
!"#$%&'()*!"+"&,-*
!"#$%&'!()&'
*+,'
-..(/0#&01'
2#$%13'4"1&'
567'
809:&".&'
2;/3'?';@'
A(..(0/'
B"C.'
->/3'?';@'
D":E(&.'
A(..(0/'
80.1:910%'
A(0
DBQ ASSESSMENTS
12
References
Bardine, B., & Fulton, A. (2008). Analyzing the benefits of revision memos during the writing
and revision process. The Clearing House, 81(4), 149-154.
Bissex, G. L., & Bullock, R. (1987). Seeing for ourselves: Case-study research by teachers of
writing. Portsmouth, N.H.: Heinemann.
Blanton, L.L. (1986). Reshaping ESL students perceptions of writing. ELT Journal. 41(2), 112-
118. DOI: 10.1093/elt/41.2.112
Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and grading.
Teacher Librarian, 40(4), 52.
Calkins, L. (1994). The art of teaching writing. Portsmouth, N.H.: Heinemann.
De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in
culturally and academically diverse middle school classrooms. Journal of Educational
Psychology, 97(2), 139-156.
De La Paz, S., & Felton, M. (2010). Reading and writing from multiple source documents in
history: Effects of strategy instruction with low to average high school writers. Journal of
Contemporary Educational Psychology, 35, 174-192.
Graves, D. (1983). Writing: Teachers and children at work. Exeter, N.H.: Heinemann
Educational Books.
Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of
adolescents in middle and high schools - A report to Carnegie Corporation of New York.
Washington, DC: Alliance for Excellent Education.
DBQ ASSESSMENTS
13
Grant, S. G., Gradwell, J. M., & Cimbricz, S. K. (2004). A question of authenticity: The
document-based question as an assessment of students knowledge of history. Journal of
Curriculum and Supervision, 19, 309-337.
Fisher, D., & Frey, N. (2007). Checking for understanding: Formative assessment techniques for
your classroom. Alexandria, VA: Association for Supervision and Curriculum
Development.
O'Toole, R. (2013) Pedagogical strategies and technologies for peer assessment in Massively
Open Online Courses (MOOCs). Discussion Paper. University of Warwick, Coventry,
UK: University of Warwick. (Unpublished)
Rogers, L., & Graham, S. (2008). A meta-analysis of single subject design writing intervention
research. Journal of Educational Psychology, 100(4), 879-906.
Rothman, R. (2011). Something in common: The common core standards and the next chapter in
American education. Cambridge, MA: Harvard Education Publishing.
DBQ ASSESSMENTS
14
Author Bio
Scott Petri has worked as a social studies teacher since 2003; five of those years in
middle school and the rest in high school. He has served as a teacher, coordinator, and small
school principal in the Los Angeles Unified School District. He earned a Bachelor of Arts in
Political Science from the University of San Diego, a M. A. in Education Administration, and a
Doctorate in Educational Leadership from California State University Northridge, where he
developed an Entrepreneurial Orientation instrument for educators that evaluated 729 California
charter, private, and traditional public school teachers along domains of innovativeness,
proactiveness, and risk taking.