10
Psychology in the Schools, Vol. 47(8), 2010 C 2010 Wiley Periodicals, Inc. View this article online at wileyonlinelibrary.com DOI: 10.1002/pits.20509 TEACHERS JUDGMENTS OF STUDENTS EARLY LITERACY SKILLS MEASURED BY THE EARLY LITERACY SKILLS ASSESSMENT: COMPARISONS OF TEACHERS WITH AND WITHOUT ASSESSMENT ADMINISTRATION EXPERIENCE JOHN C. BEGENY North Carolina State University HEATHER BUCHANAN John Carroll University Teacher judgments about students academic abilities are important for several reasons, including their day-to-day instructional decision making. Not surprisingly, previous studies have investigated the accuracy of teachers judgments about their students reading abilities. Previous research, how- ever, has not investigated teachers judgments about students early literacy skills, nor has previous research systematically examined how teachers training and use of an objective assessment instru- ment impact their judgments of students performance on that instrument. This exploratory study offers the rst investigation of teachers judgment accuracy of pre-kindergarten students early literacy skills, and compares the judgment accuracy of teachers with and without opportunities to administer the Early Literacy Skills Assessment (ELSA). Findings suggest that teachers with opportunities to administer the ELSA had a signicantly higher percentage of accurate judgments across half of the ELSA subtests, but their judgment accuracy was still no better than 50% 60% on all but one ELSA subtest. Implications for school psychologist practitioners and researchers are presented. C 2010 Wiley Periodicals, Inc. Providing evidence-based reading instruction in the early grades is commonly viewed as one of the best strategies to prevent students from experiencing difculties when learning to read (Snow, Burns, & Grifn, 1998). Yet, even when kindergarten and early-elementary teachers integrate gener- ally effective core reading curricula into their classrooms, many students still struggle to learn early literacy skills (Al Otaiba & Fuchs, 2002). To most effectively promote student learning, teachers are thus challenged with the task of regularly modifying their instructional plans after collecting and interpreting information about each students learning. In fact, the ability of teachers to adjust daily to students instructional needs is considered a hallmark of expert teachers (Berliner, 2004; Kohler, Henning, & Usma-Wilches, 2008). For many years, educators have acknowledged that teachers judgments about their students academic achievement is highly related to teachers ongoing instructional decision making (e.g., Coladarci, 1992; Eckert & Arbolino, 2005; Gerber & Semmel, 1984; Hurwitz, Elliott, & Braden, 2007). For instance, based on judgments of student achievement, teachers make daily decisions about instructional materials, teaching strategies, and student learning groups (Clark & Peterson, 1986; Kohler et al., 2008; Sharpley & Edgar, 1986). These judgments have been shown to inuence teachers expectations of student achievement, the ways in which teachers and students interact, and student outcomes (e.g., Cadwell & Jenkins, 1986; Good & Brophy, 1986; Hurwitz et al., 2007). Of course, more objective data that are derived from a students performance on a specic assessment should also inuence teachers instructional decision making. This fact is particularly relevant in a Response to Intervention (RTI) model of data-based decision making, with curriculum- based measures of reading (CBM-R) serving as the most common method for assessing elementary- age students reading skills once every three to four months (for most students), and as regularly as once every one or two weeks (for students receiving intervention services; Burns & Gibbons, Correspondence to: John Begeny, College of Humanities and Social Sciences, Department of Psychology, 640 Poe Hall, Campus Box 7650, Raleigh, NC 27695-7650. E-mail: john [email protected] 859

Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Embed Size (px)

Citation preview

Page 1: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Psychology in the Schools, Vol. 47(8), 2010 C© 2010 Wiley Periodicals, Inc.View this article online at wileyonlinelibrary.com DOI: 10.1002/pits.20509

TEACHERS� JUDGMENTS OF STUDENTS� EARLY LITERACY SKILLS MEASURED BYTHE EARLY LITERACY SKILLS ASSESSMENT: COMPARISONS OF TEACHERS WITH

AND WITHOUT ASSESSMENT ADMINISTRATION EXPERIENCE

JOHN C. BEGENY

North Carolina State University

HEATHER BUCHANAN

John Carroll University

Teacher judgments about students� academic abilities are important for several reasons, includingtheir day-to-day instructional decision making. Not surprisingly, previous studies have investigatedthe accuracy of teachers� judgments about their students� reading abilities. Previous research, how-ever, has not investigated teachers� judgments about students� early literacy skills, nor has previousresearch systematically examined how teachers� training and use of an objective assessment instru-ment impact their judgments of students� performance on that instrument. This exploratory studyoffers the Þrst investigation of teachers� judgment accuracy of pre-kindergarten students� earlyliteracy skills, and compares the judgment accuracy of teachers with and without opportunitiesto administer the Early Literacy Skills Assessment (ELSA). Findings suggest that teachers withopportunities to administer the ELSA had a signiÞcantly higher percentage of accurate judgmentsacross half of the ELSA subtests, but their judgment accuracy was still no better than 50%�60%on all but one ELSA subtest. Implications for school psychologist practitioners and researchers arepresented. C© 2010 Wiley Periodicals, Inc.

Providing evidence-based reading instruction in the early grades is commonly viewed as oneof the best strategies to prevent students from experiencing difÞculties when learning to read (Snow,Burns, & GrifÞn, 1998). Yet, even when kindergarten and early-elementary teachers integrate gener-ally effective core reading curricula into their classrooms, many students still struggle to learn earlyliteracy skills (Al Otaiba & Fuchs, 2002). To most effectively promote student learning, teachers arethus challenged with the task of regularly modifying their instructional plans after collecting andinterpreting information about each student�s learning. In fact, the ability of teachers to adjust dailyto students� instructional needs is considered a hallmark of expert teachers (Berliner, 2004; Kohler,Henning, & Usma-Wilches, 2008).

For many years, educators have acknowledged that teachers� judgments about their students�academic achievement is highly related to teachers� ongoing instructional decision making (e.g.,Coladarci, 1992; Eckert & Arbolino, 2005; Gerber & Semmel, 1984; Hurwitz, Elliott, & Braden,2007). For instance, based on judgments of student achievement, teachers make daily decisionsabout instructional materials, teaching strategies, and student learning groups (Clark & Peterson,1986; Kohler et al., 2008; Sharpley & Edgar, 1986). These judgments have been shown to inßuenceteachers� expectations of student achievement, the ways in which teachers and students interact, andstudent outcomes (e.g., Cadwell & Jenkins, 1986; Good & Brophy, 1986; Hurwitz et al., 2007).

Of course, more objective data that are derived from a student�s performance on a speciÞcassessment should also inßuence teachers� instructional decision making. This fact is particularlyrelevant in a Response to Intervention (RTI) model of data-based decision making, with curriculum-based measures of reading (CBM-R) serving as the most common method for assessing elementary-age students� reading skills once every three to four months (for most students), and as regularlyas once every one or two weeks (for students receiving intervention services; Burns & Gibbons,

Correspondence to: John Begeny, College of Humanities and Social Sciences, Department of Psychology, 640Poe Hall, Campus Box 7650, Raleigh, NC 27695-7650. E-mail: john [email protected]

859

Page 2: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

860 Begeny and Buchanan

2008). Even with objective assessment data being generated for some students as much as once perweek, teachers realistically make more frequent and ongoing instructional decisions based on theirjudgments of students� academic performance (Berliner, 2004; Kohler et al., 2008). Furthermore,teachers must continue to make instructional decisions as a result of their judgments, because eventhe most commonly used and psychometrically sound progress-monitoring assessment methods(such as CBM-R) often result in high degrees of data variability until data can be analyzed overextended periods of time (Christ & Silberglitt, 2007; Poncy, Skinner, & Axtell, 2005).

Related to the topic of RTI and data-based decision making, Gerber (2005) argued that, becauseof factors such as variation in teachers� judgments about students� responsiveness to instruction,even a standardized protocol of RTI strategies is not likely to be successful at a meaningful scale ofimplementation. More speciÞcally, Gerber stated that �the teacher actively measures the distributionof responsiveness in her class by processing information from a series of teaching trials and perceivessome range of students as within the [teachable range]� (p. 516). He then posited that teacherperceptions of students� differential responsiveness to instruction ultimately inßuence how teachersrefer students for special education.

Thus, even with important educational advances in developing progress-monitoring assessmentmethods that have potential for assisting teachers in making more frequent data-based instructionaldecisions, the regular occurrence and overall importance of teachers� judgments for everyday in-structional decision making remain. It is therefore imperative that teachers make accurate judgmentsabout their students� academic abilities.

Unfortunately, previous research has not yet systematically explored the effectiveness and valid-ity of teachers� ongoing instructional decisions in the classroom. There is, however, a relatively smallresearch base that has explored teachers� judgment accuracy of students� academic performance.The primary purpose of these teacher-judgment studies (and of the present study) is to evaluate theextent to which teachers� judgments align with more objective assessments of students� academicprogress. Procedurally, teachers are generally asked during a brief interview with the researcher toestimate (i.e., judge) students� performance on the objective assessment that is administered by amember of the research team. The basic rationale of these teacher-judgment studies is that teachersare unlikely to make sound instructional decisions for their students if they are inaccurate judges ofstudents� scores on objective measures of performance (Coladarci, 1992; Eckert, Dunn, Codding,Begeny, & Kleinmann, 2006; Stiggins, 2001).

Early research examining teachers� judgments of students� academic abilities showed thatteachers were relatively good judges. In a meta-analysis of 16 teacher-judgment studies, Hoge andColadarci (1989) found a moderate correlation between student achievement and teacher judgments(Mdn r = 0.66). They also found that correlations among the studies ranged from 0.28 to 0.92. Theyinterpreted this range of correlation coefÞcients as a presence of moderating variables, such as typesof rating methods used in the studies and degree of similarity between teacher-judgment measuresand the achievement measures.

More recent studies in this area have added clarity to the range of correlation coefÞcients re-ported by Hoge and Coladarci (1989). SpeciÞcally, these studies suggest that correlation coefÞcientsalone mask teachers� actual judgment accuracy (Begeny, Eckert, Montarello, & Storie, 2008; Eckertet al., 2006; Feinberg & Shapiro, 2003). In the two most recent studies exploring this possibility(Begeny et al., 2008; Eckert et al., 2006), Þndings showed that evaluating teachers� percentage ofagreements between estimated and actual reading levels demonstrates a more speciÞc criterion forestimating teachers� judgment accuracy. Both of those two studies evaluated teachers� judgmentsabout students� reading abilities, and both studies reported similar results.

Using a sample of two second-grade teachers and 33 students, Eckert and colleagues (2006)found that teachers generally overestimated students� reading abilities and accurately judged

Psychology in the Schools DOI: 10.1002/pits

Page 3: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Teacher Judgments of Students� Early Reading 861

students� reading level no more than approximately 50% of the time. Based on a sample of 10teachers and 87 Þrst-, second-, and third-grade students, Begeny and colleagues (2008) found thatteachers tended to overestimate students� reading performance. They also reported that teachersaccurately judged low- and average-performing students no more than 45% of the time.

Interestingly, no previous studies have examined teachers� judgments about students� early (pre-kindergarten) literacy skills. Although early literacy skills has been deÞned somewhat differently byeducators and researchers (e.g., Snow et al., 1998), for the purposes of this article we deÞne early(pre-kindergarten) literacy skills as knowing the overall purpose of reading and writing, naming andwriting letters of the alphabet, hearing rhymes and sounds in words, listening to stories for meaning,and learning the meaning of basic words from stories. This deÞnition comes from commonly useddeÞnitions recognized by the National Association for the Education of Young Children (Roskos,Christie, & Richgels, 2003).

Another gap in the teacher-judgment research is that no studies have systematically examinedwhether teachers are more accurate judges of students� reading skills when they receive structuredtraining and experience with an assessment instrument that is relevant to the types of daily judgmentsthey make about students� reading skills. Given these gaps in the teacher-judgment research, thepurpose of this study was to provide an initial evaluation of the relative judgment accuracy of pre-kindergarten teachers with and without training and assessment experience with the Early LiteracySkills Assessment (ELSA; DeBruin-Parecki, 2004). Although this research question is unique inmany ways, we hypothesized that training and experience with the ELSA would result in moreaccurate judgments about students� performance on the ELSA.

Because there is a meaningful relationship between a teacher�s judgment of a student�s abil-ities and instructional decisions for that student (Coladarci, 1992; Gerber, 2005), we believe thatbetter understanding teachers� judgment accuracy should help school professionals improve teach-ers� instructional decision making. Also, because early literacy skills often predict the strength ofsubsequent literacy skills, it is important for educators to understand teachers� judgment accuracy atthis critical period of reading development.

METHOD

Participants

Teachers. Five pre-kindergarten teachers and their Þve teacher assistants (TAs; all female)from three southeastern public pre-schools participated in this study. Teachers and TAs taught indyads and taught two half-day pre-kindergarten classes. Thus, with separate morning and afternoonclasses, and with Þve teaching dyads, 10 separate classrooms of students were represented in thisstudy.

Of the Þve teachers, two had a bachelor�s degree and three had a master�s degree. The totalnumber of years teaching ranged from 3 to 35 years (mean [M] = 19.0, standard deviation [SD] =13.5). The highest level of formal education for the Þve TAs was as follows: two had a highschool diploma, two had a high school diploma plus two years of college coursework, and onehad a bachelor�s degree. TAs� total number of years teaching ranged from 1 to 21 years (M =9.0, SD = 7.6). Each teacher also reported how much reading instruction time is allocated daily inthe classroom, as well as the types of reading instruction activities used. All teachers allocated 30�40 minutes for reading instruction per half-day class. Instructional activities were highly consistentacross classrooms and included Þnger plays, sing-alongs, rhyming, read-alouds to students, letterwriting, and practicing letter sounds. Within each classroom, teachers and TAs shared instructionalresponsibilities and opportunities to lead instructional activities. The highly similar teaching respon-sibilities of the teacher and TA per classroom provided the primary basis for us considering teachers

Psychology in the Schools DOI: 10.1002/pits

Page 4: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

862 Begeny and Buchanan

and TAs as a �teacher� in the analyses described later. Henceforth, �teacher� will refer to teachersand TAs, unless otherwise speciÞed.

Students. All participating teachers� students were eligible for participation in this study,although only eight students were randomly selected per classroom (N = 80) and subsequentlyparticipated. We selected this number of students to obtain a large enough student sample size(commensurate with the sample reported by Begeny et al., 2008) that would also represent a feasiblenumber of students for teachers to judge academically for the purposes of this project.

Of the 80 student participants, half attended a morning pre-kindergarten class and half attendedan afternoon class. Participants� median age was 4 years, 3 months; 52.5% were male, 53.8% wereBlack, 43.8% were White, and 2.6% were Hispanic. Most students (62.5%) received free or reducedlunch, and one student received special education services.

Materials

ELSA. The ELSA (DeBruin-Parecki, 2004) was used to evaluate students� early literacyskills and as the basis for having teachers estimate students� reading abilities. The ELSA is anauthentic measure of students� early literacy skills and measures reading across four content areas:Comprehension (COMP), Phonological Awareness (PA), Alphabetic Principle (AP), and ConceptsAbout Print (CAP). Items are presented as the examiner reads a book to the child and periodicallyasks the child questions related to the four content areas. Examples of tasks for the content areasare as follows. For COMP, the student is asked to predict, retell, and connect the story to his/herlife. For PA, the student must segment words and detect words that rhyme. For AP, the studentmust recognize uppercase and lowercase letters and produce letter sounds. For CAP, the studentmust answer questions about book orientation, the book cover, the title, and book orientation (e.g.,reading text from left to right).

Based on raw scores from ELSA items, students are classiÞed as Early Emergent (Level 1),Emergent (Level 2), or Competent Emergent (Level 3) in each of the four content areas. Cheadle(2007) provides an extensive description of the ELSA�s psychometric properties. Overall, the ELSAdemonstrates acceptable reliability and concurrent validity estimates, with nearly all correlationsgreater than .60, and most greater than .80. Relevant to our study, Cheadle also states that �child-careprofessionalsmay use the ELSA, because the ELSAdoes not require [advanced degree] professionalsto administer� (p. 1).

Teacher Interview Data Sheets. The ELSA Child Summary sheet used for test administrationpurposes was also used to record teachers� estimates of students� reading skills. During the teacherinterview, the interviewer used this sheet to record teachers� estimates of a student�s Total Raw Scoreand Level for each content area. A separate data sheet (developed for the purposes of this project)was also used during the interview to record teacher information about reading activities employedin the classroom and teacher background information.

Procedure

Teacher Training with the ELSA. Of the 10 participating teachers, Þve received training toadminister the ELSA. Selection of teachers for ELSA training was random. Of the Þve teachers whoreceived training and subsequent experience administering the ELSA, three were TAs and two wereteachers. Two of these three TAs worked in the same classroom with the two teachers who receivedtraining. The second author of this study (a school psychologist working in each of the participatingschools) conducted the ELSA training for teachers at an after-school, 2-hour workshop. Duringthis time, the trainer (a) provided key background information about COMP, PA, AP, and CAP;

Psychology in the Schools DOI: 10.1002/pits

Page 5: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Teacher Judgments of Students� Early Reading 863

(b) discussed how these four early literacy skills are measured, generally, and speciÞcally withELSA items; and (c) allowed teachers to practice administering the ELSA to each other whileproviding them with implementation feedback. As noted by Cheadle (2007), this is the level oftraining needed for educators to properly administer the ELSA.

ELSAAdministration. The aforementioned trainer and teacher participants who received train-ing were responsible for administering the ELSA to all student participants. The trainer administeredthe ELSA to all students of teachers who did not receive training, as well as for some students ofteachers who did receive training. The schedule was arranged so that, of teachers who administeredthe ELSA, they administered the assessment with students who were not preselected to partici-pate in the study. Thus, no teacher administered the ELSA to a student whom she would be laterasked to judge. To further avoid potential confounding variables, (a) teachers were not aware thatthey would be asked to estimate students� performance until just prior to the teacher interview,(b) teachers did not have access to students� ELSA scores until after completion of the study, and(c) teacher interviews were typically conducted within hours of student participants receiving theELSA, which we did to minimize the possibility that a teacher or TA would mention a student�sELSA results to the other teacher in the class. Overall, we have no reason to suspect that a teacherparticipant received information about a student participant�s performance prior to the teacherinterview.

Also, we have legitimate reason to conclude that ELSA-trained teachers consistently adminis-tered the ELSA accurately. The ELSA trainer regularly observed teachers as they administered thisassessment and veriÞed that all teachers administered the assessment properly.

Teacher Interviews. Teacher interviews occurred in March (the students� school year beganin September and ended in June). Prior to interviews, we ensured that each ELSA-trained teacheradministered at least Þve ELSAs to pre-kindergarten-age students. All teachers were then asked totake part in a brief teacher interview, completed by the second author.

All interviews were conducted one-on-one with teachers, and the interviewer followed a spec-iÞed interview protocol created by (and attainable from) the Þrst author. Each teacher was asked toestimate eight students� reading abilities from her classroom. The interviewer completed all ques-tions regarding one student before asking questions about the next, and the order of students queriedabout during the interview was determined randomly.

Each interview began by saying to the teacher, �We are interested in better understanding theearly literacy skills of the students in your classroom. To do so, we are going to ask you to estimatehow well you think the students did on this ELSA assessment. When determining your estimates,we would like you to base your judgment on the student�s performance within the last three weeks.�Next, the interviewer showed the teacher the ELSA materials describing the developmental levels(i.e., Early Emergent, Emergent, and Competent Emergent) for each content area. The teacher readthe descriptive deÞnitions of the levels and was shown the ranges of raw-scores points that coincidedwith each of the levels. The interviewer answered teacher questions and made sure she understoodthe descriptions of the different levels before proceeding.

The interviewer then showed the teacher the ELSA Child Summary sheet. The interviewerexplained (a) how this sheet was used for ELSA administration purposes, and (b) how the raw scorepoints and levels shown on this sheet coincided with the materials describing the developmentallevels. The teacher was then asked to estimate the target student�s total raw score and level for theCOMP, PA, AP, and CAP content areas. The interviewer recorded all teacher estimates, obtaineda new Child Summary sheet, and then asked the teacher to estimate the next student�s scores.This continued until all eight students� scores were estimated. Finally, the interviewer used the

Psychology in the Schools DOI: 10.1002/pits

Page 6: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

864 Begeny and Buchanan

appropriate data sheet to inquire about and record additional teacher information (e.g., readingactivities employed in the classroom, highest degree earned).

Dependent Measures and Analyses

Dependent measures included students� observed ELSA reading performance (raw scores andreading levels on each of the four ELSA content areas), as well as teachers� estimates of students�reading abilities on thosemeasures. Based onmethodology andÞndings reported in previous research(Begeny et al., 2008; Eckert et al., 2006; Feinberg & Shapiro, 2003), we analyzed data usingcorrelational analyses and the percentage of agreements between student performance and teacherjudgments. To evaluate possible judgment differences between our groups of teachers (those withand without ELSA training), we used chi-square analyses for percentage of accurate reading leveljudgments.We also analyzed teacher demographic information (years of teaching and degree earned)between teachers who did and did not receive ELSA training. These analyses conÞrmed that therewere no signiÞcant differences between the two groups, with p values ranging from .53 to .92.

RESULTS

Correlation coefÞcients (Pearson�s r and Spearman�s rs, respectively) were computed for per-formance/judgment raw scores and reading levels for each ELSA content area. Table 1 displays theresults of these correlation coefÞcients for teachers who did and did not receive ELSA training.Overall, correlation coefÞcients ranged from low to moderate. For the trained group, the lowestraw score correlation was in PA, and the largest correlations were in the AP and CAP areas. Forthe nontrained group, the lowest raw score correlations were in the PA and CAP areas, and thelargest correlation was in the AP area. Similar Þndings were obtained for reading level correlations.These data suggest that the trained group of teachers, compared to the nontrained teacher, obtained anotably stronger relationship between their CAP judgments and students� actual CAP performance.

Table 2 shows the percentages of agreement across each ELSA content area and betweeneach group of teachers. Findings suggest that the group of teachers receiving ELSA training wasbetter overall at estimating students� reading levels and were signiÞcantly better (p < .05) than thenontrained group in the areas of AP and CAP. Interestingly, with the exception of AP estimates,even trained teachers� judgment accuracy was still no better than approximately 50%�60% acrossthe other three content areas.

Table 1Relationship between Teacher Judgment and Students� Reading Performance across Content Areas on the ELSA

Trained Nontrained

Dependent Measure Group Group

Students� raw score on COMP and teachers� estimates of students� raw score on COMP r = 0.34 r = 0.32∗

Students� raw score on PA and teachers� estimates of students� raw score on PA r = 0.06 r = −0.09Students� raw score on AP and teachers� estimates of students� raw score on AP r = 0.58∗∗ r = .48∗∗

Students� raw score on CAP and teachers� estimates of students� raw score on CAP r = 0.53∗∗ r = 0.03

Students� COMP reading level and teachers� estimates of students� COMP reading level rs = 0.18 rs = 0.35∗

Students� PA reading level and teachers� estimates of students� PA reading level rs = 0.03 rs = 0.23

Students� AP reading level and teachers� estimates of students� AP reading level rs = 0.56∗∗ rs = 0.42∗∗

Students� CAP reading level and teachers� estimates of students� CAP reading level rs = 0.25∗ rs = .01

∗p < .05; ∗p < .01.

Psychology in the Schools DOI: 10.1002/pits

Page 7: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Teacher Judgments of Students� Early Reading 865

Table 2Percentage of Accurate Teacher Judgments across ELSA Content Areas for Teachers With and Without ELSATraining

Teacher Group

ELSA Content Area Trained Nontrained p Value from Chi-Square Analysis

COMP 63.3%a 56.1%a .54

PA 53.3%b 46.3%b .56

AP 80.0%b 52.5% .02c

CAP 50.0%b 24.4%a .03c

Overall percentage agreement 61.7% 44.8% <.01c

a70% or more of the disagreements were overestimates of students� performance.b70% or more of the disagreements were underestimates of students� performance.cTrained group had signiÞcantly higher percentage agreement.

Using analytic strategies similar to Begeny and colleagues� (2008) study, Table 2 also showswhether teachers tended to over- or underestimate students� reading levels when their judgmentswere inaccurate. Findings show that both groups� inaccurate judgments of COMP levels wereoverestimates at least 70% of the time, whereas both groups underestimated students� PA level atleast 70% of the time. Furthermore, using this same 70% criterion, trained teachers were more likelyto underestimate students� abilities, and nontrained teachers more often overestimated students�abilities. It is also interesting to note that consistent patterns of over- and underestimates emerged innearly every category across both groups of teachers. In the only category that did not meet the 70%criterion (i.e., nontrained group, AP area), students� abilities were overestimated 63.2% of the time.

DISCUSSION

The primary purpose of this study was to make the Þrst attempt at (a) assessing teachers�judgments of students� early literacy skills, and (b) systematically assessing possible differences inteacher judgment accuracy depending on teachers� training and use of an early reading assessmenttool (i.e., the ELSA). Our Þndings partially supported our primary hypothesis: The group of teacherswho received training and administration experience with the ELSA judged their students� readinglevel on the AP and CAP areas signiÞcantly better than did the group of teachers without ELSAtraining. Considered within the context of this study�s limitations (described later in this section), thisis a potentially meaningful Þnding because teachers seem more likely to make better instructionaldecisions in areas such as CAP and AP if they are better able to judge students� abilities in these areas(Begeny et al., 2008; Coladarci, 1992; Stiggins, 2001). Because AP and CAP represent two majorreading skills that students must develop as part of early literacy (DeBruin-Parecki, 2004; Snowet al., 1998), teachers should better improve students� early literacy by making stronger instructionaldecisions in these areas.

Findings from this study also showed that ELSA-trained teachers still judged students� readinglevel inaccurately approximately 40%�50% of the time across three of the four ELSA areas, whichassess critical early literacy skills (Roskos et al., 2003). Although the primary purpose of this studywas unique, the Þnding that trained teachers usually judged student reading levels accurately only50%�60% of the time (and even less, in the case of nontrained teachers) is consistent with relatedresearch (e.g., Begeny et al., 2008). Interestingly, much of the previous research shows that teachersoverestimate students� reading abilities (e.g., Eckert et al., 2006; Feinberg & Shapiro, 2003). We

Psychology in the Schools DOI: 10.1002/pits

Page 8: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

866 Begeny and Buchanan

found, however, that trained teachers more often underestimated students� abilities. Future researchshould help determine the reasons for teachers� over- or underestimates, as well as the possibleinstructional implications.

As would be expected of any preliminary study, our study contains limitations and directions forfuture research. For instance, although our study included as many teacher and student participants(or similar numbers) as did related research in this area (Begeny et al., 2008; Eckert et al., 2006),our sample size of teacher and student participants unfortunately did not allow for more detailedanalyses regarding factors that may inßuence teachers� judgments about early literacy (e.g., teachervariables, such as years of experience, and student variables, such as students� reading ability).

Also, our study did not speciÞcally evaluate whether ELSA judgments led teachers to makedifferent instructional decisions. As noted previously, no previous research has systematically ex-plored the effectiveness and validity of teachers� ongoing instructional decisions in the classroom.Although this limitation is addressed by related research and strong theoretical arguments supportingthe link between teacher judgments and ongoing instructional decision making (e.g., Gerber, 2005;Hoge & Coladarci, 1989; Stiggins, 2001), future research should begin to devote the extensive timeand resources needed to speciÞcally and systematically examine teachers� day-to-day instructionaldecision making. Given the Þndings from this initial study investigating the relationship betweenteacher training and judgment accuracy, future research would beneÞt from examining teachers�knowledge and use of objective academic assessments as part of the instructional decision-makingprocess.

Another important variable to evaluate in future research is the degree to which trainingin a particular assessment generalizes to estimates of student performance on a related academicconstruct (or assessment) withwhich a teacher does not receive speciÞc training. For example, trainedteachers in our study demonstrated signiÞcantly better judgment accuracy when estimating students�CAP ability, as measured by the CAP content area of the ELSA. It would be interesting to knowwhether this judgment accuracy would also be signiÞcantly better than that of nontrained teachers ifteachers were asked to estimate reading abilities related to CAP, as measured by other early readingassessments such as the Test of Early Reading Ability (Reid, Hresko, & Hammill, 2001). Futureresearchersmight also evaluate teachers� previous training and experiences in educational assessment(broadly) to determine whether such experiences moderate a relationship between training in aspeciÞc assessment and judgments of students� academic performance.

Future research that includes TAs as estimators of student performance should also considerthe practical relevance of those estimates. In our study, TAs carried about as much instructionalinßuence in their classroom as the primary classroom teacher; therefore, we had good reason tobelieve that TA instructional decision making was important to measure. Indeed, there are severalexamples of classrooms in which TAs carry meaningful instructional responsibilities. In such cases,TAs certainly inßuence, if not act on, important decisions, such as special education referrals andstudent assignments to learning groups within a classroom. This is not true in all classrooms,however. Thus, future researchers should carefully consider the relevance of including TAs withintheir teacher sample.

Implications

Considering the aforementioned limitations, we maintain that our initial study in this area offersmethodology and Þndings that should stimulate further research in this area, as we feel that thistype of research has important applied implications for school psychologists and related educators.Although our study focused mainly on assessment (i.e., assessing students with the ELSA andassessing teachers� judgments), we believe that our Þndings have implications for school-based

Psychology in the Schools DOI: 10.1002/pits

Page 9: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

Teacher Judgments of Students� Early Reading 867

intervention. SpeciÞcally, our study suggests that school psychologists may help teachers with theirperceptions about students� reading abilities by providing themwith in-service training on assessmentinstruments that are meaningful for teachers and student learning. Indeed, previous researchers haveshown that many teachers do not receive adequate training in assessment during their preserviceexperiences (Begeny & Martens, 2006; Schafer, & Lissitz, 1987). As an extension of this andrelated studies, future research should help to answer important questions about the relationshipsbetween teachers� training with assessment instruments, their judgments about students� academicperformance, and their daily decisions about student learning based on their perceptions of students�abilities.

We also note that our study in no way implies that a teacher�s judgment about a particularstudent should ever preclude a more formal, thorough educational assessment if a teacher�s opinionis that a student should qualify for special education services. Of course, prior to any recommen-dation for special education services, school psychologists are needed to facilitate comprehensivepsychoeducational assessment and/or data-based decision-making strategies like those espousedwithin a model of RTI. Again, teacher judgments are associated with a student�s special educationentitlement (Gerber, 2005;Gresham,MacMillan,&Bocian, 1997), but, equally important, a teacher�sinaccurate judgment about a student�s academic abilities is more likely to prevent the teacher frommaking everyday instructional decisions that would best facilitate that student�s learning. In par-ticular, with the increasing number of studies (including ours) suggesting that teachers commonlyoverestimate students� reading abilities, helping teachers improve their academic judgment accuracyshould also help to improve their instructional decision making and, ultimately, students� learning.

In addition to the implications for practitioners described earlier in this article, we hope that thisstudy assists researchers by providing a foundational methodology for evaluating whether teachertraining inßuences teachers� judgments. In fact, we think that this line of research is ideal forpractitioner�researcher collaborations. For instance, several of the procedures employed in this typeof a study can be feasibly managed as part of a school psychologist�s role in a typical school (e.g.,providing training for teachers, meeting with teachers to discuss student performance), whereasother components are more feasible for researchers (e.g., providing the time and resources to runstatistical analyses, develop procedural protocols, and assess procedural integrity). Thus, we stronglyencourage practitioner�researcher collaboration in this area and even recommend that interestedpractitioners contact us about possible collaborative efforts.

REFERENCES

Al Otaiba, S., & Fuchs, D. (2002). Characteristics of children who are unresponsive to early literacy intervention: A reviewof the literature. Remedial and Special Education, 23, 300�316.

Begeny, J. C., Eckert, T. L., Montarello, S., & Storie, M. S. (2008). Teachers� perceptions of students� reading abilities: Anexamination of the relationship between teachers� judgments and students� performance across a continuum of ratingmethods. School Psychology Quarterly, 23, 43�55.

Begeny, J. C.,&Martens, B.K. (2006). Assessing pre-service teachers� training in empirically-validated behavioral instructionpractices. School Psychology Quarterly, 21, 262�285.

Berliner, D. (2004). Describing the behavior and documenting the accomplishments of expert teachers. Bulletin of Science,Technology, & Society, 24, 200�212.

Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools:Procedures to assure scientiÞc-based practices. New York: Routledge.

Cadwell, J., & Jenkins, J. (1986). Teacher�s judgments about their students: The effects of cognitive simpliÞcation strategieson the rating process. American Educational Research Journal, 23, 460�475.

Cheadle, J. E. (2007). The early literacy skills assessment: Psychometric report for both English and Spanish versions.High/Scope Early Childhood Reading Institute. Retrieved July 8, 2008, from http://www.highscope.org/Þle/Assessment/ELSAJacobs.pdf

Christ, T. J., & Silberglitt, B. (2007). Estimates of the standard error of measurement for curriculum-based measures of oralreading ßuency. School Psychology Review, 36, 130�146.

Psychology in the Schools DOI: 10.1002/pits

Page 10: Teachers' judgments of students' early literacy skills measured by the Early Literacy Skills Assessment: Comparisons of teachers with and without assessment administration experience

868 Begeny and Buchanan

Clark, C. M., & Peterson, P. L. (1986). Teachers� thought processes. In M. C. Wittrock (Ed.), Third handbook of research onteaching (pp. 255�296). New York: Macmillan.

Coladarci, T. (1992). Teachers� knowledge of what students know: The case of reading. Reading Improvement, 29, 34�39.

DeBruin-Parecki, A. (2004). Early literacy skills assessment. Ypsilanti, MI: High/Scope Press.

Eckert, T. L, &Arbolino, L. A. (2005). The role of teacher perspectives in diagnostic and program evaluation decision-making.In R. Brown-Chidsey (Ed.), Beyond labels: Noncategorical individualized assessment methods (pp. 65�81). New York:Guilford.

Eckert, T. L., Dunn, E. K., Codding, R. S., Begeny, J. C., & Kleinmann, A. E. (2006). Assessment of mathematics and readingperformance: An examination of the correspondence between direct assessment of student performance and teacherreport. Psychology in the Schools, 43, 247�265.

Feinberg, A. B., & Shapiro, E. S. (2003). Accuracy of teacher judgments in predicting oral reading ßuency. School PsychologyQuarterly, 18, 52�65.

Gerber, M. M. (2005). Teachers are still the test: Limitations of response to instruction strategies for identifying children withlearning disabilities. Journal of Learning Disabilities, 38, 516�524.

Gerber, M. M., & Semmel, M. I. (1984). Teacher as imperfect test: Reconceptualizing the referral process. EducationalPsychologist, 19, 1�12.

Good, T. L., & Brophy, J. R. (1986). School effects. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed.).New York: Macmillan.

Gresham, F. M., MacMillan, D. L., & Bocian, K. M. (1997). Teachers as tests: Differential validity of teacher judgments inidentifying students at-risk for learning difÞculties. School Psychology Review, 26, 47�60.

Hoge, R. D., & Coladarci, T. (1989). Teacher-based judgments of academic achievement: A review of the literature. Reviewof Educational Research, 59, 297�313.

Hurwitz, J. T., Elliott, S. N., & Braden, J. P. (2007). The inßuence of test familiarity and student disability status uponteachers� judgments of students� test performance. School Psychology Quarterly, 22, 115�144.

Kohler, F., Henning, J. E., & Usma-Wilches, J. (2008). Preparing preservice teachers to make instructional decisions: Anexamination of data from the teacher work sample. Teaching and Teacher Education, 24, 2108�2117.

Poncy, B. C., Skinner, C. H., & Axtell, P. K. (2005). An investigation of the reliability and standard error of measurement ofwords read correctly per minute using curriculum-based measurement. Journal of Psychoeducational Assessment, 23,326�338.

Reid, D. K., Hresko, W. P., & Hammill, D. D. (2001). Test of early reading ability (3rd ed.). Austin, TX: Pro-Ed.

Roskos, K. A., Christie, J. F., & Richgels, D. J. (2003). The essentials of early literacy instruction. Retrieved September 21,2009, from http://www.journal.naeyc.org/btj/200303/Essentials.pdf

Schafer, W. D., & Lissitz, R. W. (1987). Measurement training for school personnel: Recommendations and reality. Journalof Teacher Education, 38, 57�63.

Sharpley, C. F., & Edgar, E. (1986). Teachers� ratings vs standardized tests: An empirical investigation of agreement betweentwo indices of achievement. Psychology in the Schools, 23, 106�111.

Snow, C. E., Burns, M. S., & GrifÞn, P. (Eds.). (1998). Preventing reading difÞculties in young children. Washington, DC:National Academies Press.

Stiggins, R. J. (2001). Student-involved classroom assessment. Upper Saddle River, NJ: Prentice Hall.

Psychology in the Schools DOI: 10.1002/pits