Embed Size (px)
Citation preview
Presented at the American Educational Research Association Meeting, Chicago, Ill. 4/21/04
Barbara Perushek, Ph.D.
Cloquet Senior High School
218-879-3393, ext. 3102
Daniel Naslund
Cloquet Senior High School
218-879-3393, ext. 4102
William Fleischman, Ph.D.
University of Minnesota Duluth
218-726-7557Use of this information prohibited with out written consent of the authors ©
A Longitudinal Study of Students’ Basic Skills: Development and Use of an
Assessment Methodology
Abstract
Current practices in the teaching of English do not often employ empirical data to help assess student achievement. Rather, teachers are left to rely on subjective measures—e.g., portfolios, essays, and anecdotal assessments. At the same time, however, graduation standards and school accountability point to the need for objective, concrete measurement of English study and learning. Moreover, the data generated from state tests are far too global to benefit the specific student in a specific classroom.
This four-year, longitudinal study connects empirical results with performance assessment and achievement in the English classroom. The project originated from within our department as a way for teachers to measure student skills against standards we set for ourselves and our students. Thus far, three years of pretest/post test data indicate overall student improvement and confirm our current teaching strategies.
This study demonstrates how classroom teachers can use a locally-developed, pre/post, “English Skills Measurement Test” to objectively measure basic skills including grammar, punctuation, mechanics, spelling, and usage to identify strengths and weaknesses in student performance. The data generated can help students to identify problems and improve performance and, equally important to this project’s aim, can assist teachers in using the empirical data to assess the practice and focus of their instructional program. The data make clear to students and teachers the department’s expectations and illustrate progress being made in meeting those outcomes.
This project does not suggest replacing qualitative/subjective assessments inherent to the English classroom. Rather, the data generated by this study provide an objective, longitudinal view of student performance in our high school program and address a heretofore unfocused piece in our assessment picture.
Research Problem
• Lack of objective data in measuring student achievement in acquiring basic skills in Standard Formal English
• Sole reliance on subjective assessment in the English classroom
• Inability to track student/class longitudinal progress • State-mandated tests too global to assist the specific student,
the specific classroom • Current data inadequate for assessing our curricula and
teaching strategies
• Instituted as part of Communication Department Goals
• Developed a collaborative research partnership with University of Minnesota Duluth Professor William Fleischman
• Received a grant from UMD’s Center for Community and Regional Research
Project Background
• Measure students’ knowledge of standard formal English
• Provide a measure of student progress over academic years
• Provide a tool to assess curricula and teaching strategies
Project Goals
Test Development
• 100-question test (adapted from English simplified, l999), designed to measure basic skills including: Grammar, punctuation, mechanics, spelling, usage
• Test dimensions
Sentence Errors
Subject/Verb/Pronoun Agreement
Punctuation
Mechanics
Numbers
Spelling
Usage Usage
Context
• 9 – 12 high school in northeastern Minnesota
• Four years of English required
• Extensive course offeringsincluding Journalism, Creative Writing,
Technical Writing, Pre-College WritingCollege Composition, Humanities and Theater (all for college credit), as well as literature courses at all four grade levels
• Average class size =24
• Eight teachers in the department(six with advanced degrees)
• Pre and Post Tests given each September and May
• Computer-based analysis system provides for “feedback sheets” for each teacher/each class
• SPSS - used for additional data analyses
Methodology
Computer Scoring Output
• Student feedback sheet (Table 1)
• Correct score distribution graph (Table 2)
• Students by correct answer (Table 3)
• Number correct for each student across dimensions (Table 4)
• Student scores ranked by number correct for class (Table 5)
• Discrimination index (Table 6)
• Answer choices by question (Table 7)
TABLE 1: STUDENT FEED BACK SHEET
TABLE 2: DISTRIBUTION OF THE NUMBER OF STUDENTS BY NUMBER OF CORRECT ANSWERS TO QUESTIONS IN ONE
PART(4) OF THE TEST
TABLE 3: EXAMPLES OF SCORES ON ONE PART (4 ) OF THE TEST FOR EACH STUDENT IN A CLASS ARRANGED BY
STUDENT ID NUMBER
TABLE 4: NUMBER CORRECT FOR EACH STUDENT IN A CLASS
ACROSS ALL SEVEN PARTS OF THE TEST
TABLE 5: EXAMPLE OF STUDENT SCORES RANKED BY NUMBER CORRECT FOR QUESTIONS ACROSS THE TEST
TABLE 6: NUMBER CORRECT BY QUESTION INCLUDING THE
DISCRIMINATION INDEX FOR EACH QUESTION
TABLE 7: ANSWER CHOICES BY QUESTION – INCLUDES CORRECT ANSWER DESIGNATION (with an asterisk *) AND
FREQUENCY OF CHOICE FOR EACH QUESTION
SPSS Analyses
• Calculation of scores for each dimension (means, standard deviations, t-tests, ANOVA, etc.)
• Comparison of pretest/posttest mean scores for one year
For each class section (e.g., Honors English 10)For each enrollment year (e.g., 9th grade or 10th grade)
• Comparison of mean scores across years for a cohort of students (e.g., 9th graders 1999 – 11 graders 2002)
(Tables 8 & 9)
Table 8: 2003 Graduates Mean Scores Basic Skills
0
2
4
6
8
10
12
14
16
18
Sentence Errors S/ V/P Agreement Punctuation Mechanics Numbers Spelling Usage
Skills
Cor
rect
Ans
wer
s - M
ean
Scor
e
Pretest 1999
Post Test 2000
Post Test 2001
Post Test 2002
Table 9: 2003 Graduates Grand Total Mean Scores
57.91
63.13
65.3766.06
52
54
56
58
60
62
64
66
68
1999 Pretest 2000 Post Test 2001 Post Test 2002 Post Test
Co
rre
ct
An
sw
ers
- M
ea
n S
co
res
Findings
• Mean scores increase from year to year at all grade levels
• Grade 9 shows largest increase from Pretest to post test
• Overall, each skill area reveals student improvement
• Measures students’ progress over academic years
• Identifies specific strengths/weaknesses in individual student performance
• Scoring/analyses useful for connecting teaching/learning
• Assessments useful for evaluating the effectiveness of department curricula and teaching strategies
• Additional analysis of score leveling from grades 10 to12
• Additional analysis of substantial increase in 9th grade scores
Implications
References
Beason, L. (2001). Ethos and error: How business people react to errors. College Composition and Communication, 53(1), 33-64.
Ellsworth, B. (Ed.). (1999). English Simplified (7th ed.). Scranton, PA: Harper Collins.
Hiebert, J., Gallimore, R., & Stigler, J. (2002). A knowledge base for the teaching profession: What would it look like and how can we get one? Educational Researcher, 31(5), 3-15.
Langer, J. A. (2000). Beating the odds: Teaching middle and high school students to read and write well. CELA Research Report Number 12014. Albany, NY: National Research Center on English Learning & Achievement.
Strong, W. (1999). Coaching writing development: Syntax revisited, options explored. In C. R. Cooper & L. Odell (Eds.), Evaluating writing: The role of teachers’ knowledge about text, learning, and culture (pp. 72-92). Urbana, IL: National Council of Teachers of English.
