Embed Size (px)
Citation preview
This article was downloaded by: [Monash University Library]On: 04 December 2014, At: 12:57Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Teaching EducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/cted20
Improving teachers’ assessmentliteracy through professionaldevelopmentKim H. Koh aa Curriculum, Teaching and Learning Academic Group, NationalInstitute of Education, Nanyang Technological University ,SingaporePublished online: 15 Aug 2011.
To cite this article: Kim H. Koh (2011) Improving teachers’ assessment literacy through professionaldevelopment, Teaching Education, 22:3, 255-276, DOI: 10.1080/10476210.2011.593164
To link to this article: http://dx.doi.org/10.1080/10476210.2011.593164
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Improving teachers’ assessment literacy through professionaldevelopment
Kim H. Koh*
Curriculum, Teaching and Learning Academic Group, National Institute of Education,Nanyang Technological University, Singapore
(Received 25 June 2010; final version received 3 February 2011)
This study examined the effects of professional development on teachers’assessment literacy between two groups of teachers: (1) teachers who wereinvolved in ongoing and sustained professional development in designingauthentic classroom assessment and rubrics; and (2) teachers who were givenonly short-term, one-shot professional development workshops in authenticassessment. The participating teachers taught Year 4 and 5 English, science, andmathematics. The findings showed that the assessment literacy of teachers whowere involved in ongoing, sustained professional development had increasedsignificantly during the second year of study. These teachers had also gained abetter understanding of authentic assessment.
Keywords: teachers’ assessment literacy; professional development; authenticassessment
Introduction
In many countries’ educational reform movements, assessment has become a keypolicy lever for improving education. Basil Bernstein (1990), a prominent Britishsociologist of education, has long held that assessment will ultimately pull curricu-lum and pedagogy along. This is because of teachers’ tendency to reorient their cur-riculum and pedagogy to assessment. As noted by the Australian researchers behindthe Queensland School Reform Longitudinal Study (Lingard et al., 2001), develop-ing productive assessments acts as one of the best levers for engaging teachers withpedagogical change for higher intellectual demand in their daily classroom. Like-wise, many educators and policy-makers in the United States believe that ‘what getsassessed is what gets taught’ and that the assessment format influences the formatof instruction (O’Day & Smith, 1993).
Given the tendency of teachers to mirror classroom instruction to assessment, anobvious educational reform strategy is to change the content and format of assess-ments to enhance the coverage of higher intellectual learning outcomes (e.g., com-plex thinking, reasoning, problem-solving, communication, and conceptualunderstanding of subject matter) and to move curriculum and instruction toward thedevelopment of these skills (Smith & O’Day, 1990). In response to these ideas,many assessment programs around the world have been revised over the past two
*Email: [email protected]
Teaching EducationAquatic InsectsVol. 22, No. 3, September 2011, 255–276
ISSN 1047-6210 print/ISSN 1470-1286 online� 2011 Taylor & FrancisDOI: 10.1080/10476210.2011.593164http://www.informaworld.com
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
decades to reflect more challenging learning goals and to include more authentic,open-ended assessment tasks.
Proponents of alternative, authentic assessment have long advocated holisticassessment of student outcomes and learning progress on authentic tasks that areclosely aligned with higher order instructional goals. In contrast to conventionalpaper-and-pencil tests that focus on knowledge reproduction and low-level cognitiveprocessing skills in artificial, contrived contexts, authentic assessment tasks empha-size knowledge construction, complex thinking, elaborated communication, collabo-ration and problem-solving in authentic contexts. These are the essential skills forstudents to succeed in the twenty-first century knowledge-based economy. One ofthe notable school reform programs by Newmann, Marks, and Gamoran (1996) inthe United States has demonstrated that students’ exposure to authentic assessmenttasks or assignments that focus on higher intellectual demands produced more intel-lectually complex work and achieved better academic performance across differentsubjects and grades. However, the Newmann study was not based on an experimen-tal design and there is a possibility that the teachers using more authentic taskswere just better teachers (Wiliam, Lee, Harrison, & Black, 2004).
Although authentic assessment is widely accepted as a tool of educationalreform in the last two decades, a commonly encountered problem by many educa-tion systems around the world is the relative lack of assessment literacy amongteachers and school leaders. Many teachers are found not competent in developingand implementing authentic performance assessments due to inadequate trainingand support during the pre-service teacher education programs (Bol, Nunnery,Stephenson, & Mogge, 2000; Hargreaves, Earl, & Schmidt, 2002; Stiggins, 1995).The problem of teachers’ low level of assessment literacy is exacerbated by theexternal pressures for accountability in student learning and achievement as well asother practical constraints at school (e.g., time constraint, content coverage, class-room management, and support from school leaders). As a result, most teachersresort to modifying the content and format of instruction to fit the content and for-mat of high-stakes assessments. Moreover, classroom assessments or teacher-madetests tend to mimic high-stakes, standardized achievement tests, which often onlyassess discrete bits of knowledge and skills or low-level knowledge reproduction(Fleming & Chambers, 1983).
In the classroom assessment and teacher education community, there is a con-sensus that high-quality professional development will provide in-service teacherswith training and support to improve their assessment literacy, specifically indesigning and implementing authentic assessments at the classroom level(Aschbacher, 1991; Bol et al., 2000; Newmann et al., 1996; Stiggins, 1991a, 2002).The extant literature on teacher professional development has also shown thatimproving the quality of education relies heavily on teachers’ continuing develop-ment and learning of new knowledge and skills to change or improve classroompractice, which in turn leads to increased student learning and achievement (Fullan& Miles, 1992; Desimone, Porter, Garet, Yoon, & Birman, 2002; Desimone, 2009).As assessment is noted as a key lever for driving teachers’ instructional practice,changing or improving classroom practice will require teachers’ improved knowl-edge and skills in designing and implementing new forms of assessment. In fact,teacher professional development has long been touted by both teacher educatorsand policy makers as a cornerstone of systemic reform efforts to increase teachers’capacity to teach to high standards. Therefore, as we enter the second decade of the
256 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
twenty-first century, teacher professional development in assessment literacy hasbecome increasingly important because teachers are expected to master the knowl-edge and skills relevant to the teaching and assessment of twenty-first century com-petencies.
Assessment literacy and professional development
The term assessment literacy was first coined by Stiggins (1991b) as an under-standing of the principles of sound assessment. According to Stiggins (1991b),teachers who are assessment literates for the twenty-first century classroom shouldknow how to meet the following five standards of high-quality classroom assess-ment: (1) starting with clear purposes of assessment; (2) understanding the impor-tance of assessing different kinds of interrelated achievement targets (i.e.,mastering of content knowledge, developing reasoning proficiencies, attaining per-formance skills, and developing high-quality products); (3) selecting proper assess-ment methods for the different kinds of achievement targets; (4) sampling andcollecting student achievement based on representative performance tasks; and (5)avoiding assessment bias and distortion that arise from technical and practicalproblems. Although the term authentic assessment has not been used directly inhis article, all the five standards spelt out by Stiggins correspond to the ideas andprinciples of authentic assessment. In short, teachers’ assessment literacy involvesbeing prepared to define, teach, and assess the different kinds of competenciesthat match the higher order instructional goals for the twenty-first century. Inorder to be assessment literate, teachers must not only be competent to developand use high-quality authentic assessments and scoring rubrics, but also be ableto master evaluative skills to make sound judgments about student performance(Saddler, 1998).
Previous studies on classroom assessment have consistently shown that manyteachers are inadequately trained and ill-prepared to develop, administer, and inter-pret the results of various types of assessments (Bol, Stephenson, O’Connell, &Nunnery, 1998; Stiggins & Conklin, 1992; Wiggins, 1989). In general, teacherswho were less prepared and skilled in developing authentic assessments perceivedthe assessments as being more difficult to develop than traditional paper-and-penciltests. Moreover, teachers’ assessment practices were not well aligned with theirinstructional goals and tended to demand a low level of cognitive processing inclassroom assessment tasks. Many teachers were also not good judges of the qualityof their own assessment tasks (Black & Wiliam, 1998; Bol & Strage, 1996).
Although there is a considerable body of research on the links between teacherprofessional development and classroom assessment, most of the empirical studieshave focused on professional development and teachers’ formative assessment prac-tices since the seminal work by Black and Wiliam (1998). The meta-analysis of250 empirical studies conducted by Black and Wiliam (1998) showed that the inte-gration of formative assessment practices into teachers’ everyday teaching practiceresulted in substantial gains in student achievement scores on standardized tests.The effect sizes of the formative assessment experiments were found to be between0.4 and 0.7, larger than most of those found for educational interventions. This find-ing prompted Black and Wiliam (1998) to suggest that sustained programs for pro-fessional development and support must be in place in order for teachers toimprove their formative assessment practice.
Teaching Education 257
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
The only study to date that focused on teacher professional development andauthentic assessment was conducted by Borko and her colleagues in Colorado,USA (Borko, Mayfield, Marion, Flexer, & Cumbo, 1997). Borko et al. (1997)examined the process of change experienced by teachers who had undergone thetraining for developing and implementing mathematics performance assessment. Inthe study, 14 third-grade teachers participated in a year-long series of weekly work-shops that focused on topics such as selecting, extending, and designing materialsand activities for assessment and instruction; observing students and keepingrecords of observations; analyzing student work; and developing and using scoringrubrics. Teachers’ change process was examined by analyzing conversations aboutscoring tasks between teachers and researchers during workshops conductedthroughout the school year and interviews conducted at the beginning, middle, andend of the year. The study showed that teachers benefited from professional devel-opment experiences that provided them with the opportunities to explore newassessment strategies and ideas in the context of their own classroom practice.Additionally, the collective participation of teachers as members of a learning com-munity enabled professional conversations about assessment to take place. It is aneffective tool for the social construction of new ideas and practices.
In an intervention study, Wiliam et al. (2004) worked with a group of 24 sec-ondary mathematics and science teachers to develop their formative assessmentpractice for two years. Using a quasi-experimental design in their study, Wiliamet al. found a modest effect size of formative assessment on student achievement.Wiliam et al.’s study also indicates that sustained professional development isneeded for developing teachers’ formative assessment practice.
In their work on building teacher capacity in classroom assessment, McMunn,McColskey, and Butler (2004) reiterated that professional development, as situatedor embedded in the daily work lives of teachers is critical for classroom improve-ments that can lead to increased student achievement. Furthermore, high-quality oreffective professional development must be aligned with a more constructivistmodel for teacher learning, wherein teachers are involved in active learning throughprofessional conversations. It is important for teachers to work together on assess-ment issues in a collaborative setting. Some of the worthwhile assessment issuesare clarifying the instructional goals and purposes of assessment, integrating moreauthentic assessments into classroom assessment methods, examining the quality ofassessment tasks, and looking together at the quality of student work. McMunnet al. further emphasized that ongoing, sustained professional development is moreeffective than one-time, ad-hoc workshops to support teachers’ efforts at improvingtheir assessment practices.
Sato, Wei, and Darling-Hammond (2008) conducted a longitudinal study totrack the changes of mathematics and science teachers’ classroom formative prac-tices as a result of their participation in the National Board Certification process.The National Board Certification provided teachers with professional developmentin using rigorous assessment and teaching standards. Based on their analyses of vid-eotaped lessons, student work samples, and interviews with the teachers, Sato et al.(2008) found pronounced changes in teachers’ use of a variety of assessments andthe way in which assessment information was used to support student learning inthe everyday classroom instruction. Their results also indicate that effective profes-sional development strategies that focused on teachers’ actual classroom practice –classroom interactions and analysis of student work, are essential for improving
258 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
teachers’ assessment practices. The content focus on such strategies is more consis-tent with teachers’ knowledge and goals as they are directly related to the work ofteaching. Moreover, the analysis of student work reflects teachers’ active learning ina professional community (Borko, 2004; Garet, Porter, Desimone, Birman, & Yoon,2001). It is recognized as a powerful strategy for teachers to examine the evidenceof student learning and to reflect on the teaching and assessment associated withstudent learning. As teachers are engaged in reviewing student work in the topicareas being covered, it helps them develop a deep understanding of how such eval-uations of learning can inform their instructional choices and improve their class-room practices (Shepard, Hammerness, Darling-Hammond, et al., 2005).
In Graham’s study (2005), teacher candidates in the United States reported thatthey were strongly influenced by professional dialogue about planning and assess-ment in both teacher training program and mentored field experiences. This impliesthe importance of active learning. Most teacher candidates accepted alternativeassessment as a valuable source of evidence that indicated student learning. How-ever, they were concerned about their skills in identifying goals, designing rubrics,and determining the technical accuracy of assessments. The Graham (2005) findingswere supported by Volante and Fazio’s (2007) study of primary/junior teacher can-didates in Canada. They found that the majority of the teacher candidates reported alow level of assessment literacy and expressed the need for improving their assess-ment knowledge through specific courses in classroom assessment and evaluation,including good mentorship in the field. Although both studies involved pre-serviceteachers, the findings did suggest that ongoing support and professional develop-ment opportunities should be given to in-service teachers who would then mentorteacher candidates on how to apply effective classroom assessment practices.
In short, the studies on teacher professional development and classroom assess-ment reiterate the importance of the following five core features of effective profes-sional development: content focus, coherence, active learning, collective participation,and duration. In their view of building teachers’ capacity in assessment for twenty-firstcentury teaching and learning, Wiliam and Thompson (2008) aptly summed up that:
. . . teacher professional development is more effective when it is related to the localcircumstances in which the teachers operate, takes place over a sustained period ratherthan being in the form of sporadic one-day workshops, and involves the teachers inactive, collective participation. (Wiliam & Thompson, 2008, p. 55)
Purpose of study
Since 1997, the Singapore Ministry of Education has launched many policy initia-tives to reform the nation’s education system. The government’s key initiatives fordeveloping a productive, resilient, and lifelong learning nation to face the chal-lenges of the twenty-first century knowledge-based economy are as follows: ‘Think-ing Schools, Learning Nation’ (TSLN), ‘Innovation and Enterprise’ (I&E), ‘TeachLess, Learn More’ (TLLM), and Curriculum 2015 (C2015). These initiatives haveadvocated teaching for deep understanding and higher-order thinking skills ratherthan rote memorization of factual and procedural knowledge. The initiatives alsoimply that changes in teachers’ assessment practices are imperative if the ultimategoal is to enhance students’ mastery of twenty-first century competencies. In theirefforts to promote students’ higher-order thinking skills, real-world problem-solvingskills, positive habits of mind, and communication skills, teachers in Singapore are
Teaching Education 259
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
encouraged to move toward more constructivist teaching approaches and to adoptnew forms of assessment, such as authentic assessment.
Because of the need for improvement in teachers’ assessment practices, the Min-istry of Education has provided teachers with resources, support, and professionaldevelopment over the past four years. Although millions of dollars have beeninvested in teachers’ professional development at the school level, most of the pro-fessional development programs are designed and delivered as ad hoc, one–two dayworkshops and teachers may not be able to benefit from such workshop experience.In general, Singapore teachers are enthusiastic about the learning of new forms ofassessment as they are receptive to MOE’s policy initiatives. But they are oftencaught between the learning of new forms of assessment and the intransigence oflong-established practices of conventional assessment. This tension is exacerbatedby the demands of high-stakes accountability examination system. According toGuskey (2002), change in teachers’ attitude toward new classroom practices takesplace primarily after some change in student learning has been evidenced. Giventhe tension faced by Singapore teachers, there is a pressing need for more evidence-based professional development research to support teachers’ efforts to change theirclassroom practice in the context of local schools.
Prominent educators and authors of teacher learning and professional develop-ment (Borko, 2004: Desimone, 2009; Garet et al., 2001; Luke & McArdle, 2009)have called for systematic longitudinal research to be conducted on the effects ofprofessional development on improvements in teacher knowledge, classroompractice, and student learning. They also highlighted the importance of using morerigorous research design and data collection methods, which include quasi-experi-mental comparison of professional development intervention versus control schools;longitudinal tracking of teachers and students; analysis of longitudinal changes instudent performance, work and outcomes; and focus groups of teachers. Suchcollection of valuable formative and summative data will inform ongoing develop-ments of programs and design decisions about future professional development.
The purpose of this study was to examine the effects of professional development(as a form of intervention) on teachers’ assessment literacy and student learning.Teachers’ assessment literacy was measured by two indicators: the quality of class-room assessment tasks and teachers’ conceptions about authentic assessment.Because changes in teachers’ assessment literacy are expected to bring about changesin student learning, it was also important for this study to examine the changes in thequality of student work in response to changes in the quality of teachers’ assessmenttasks. Research on the effects of classroom assessment on student learning tends tofocus on using standardized test scores as a proxy for student learning. Standardizedtests might lack curricular validity (McClung in Wiliam et al., 2004) because they donot accurately reflect what the teachers were teaching in their classrooms. To estab-lish curricular validity, samples of student work for each classroom assessment taskwere collected and used as a proxy for student learning in this study.
The rationale for improving teachers’ assessment literacy in the form of authen-tic assessment task design and rubric development supports the need for equippingteachers with contemporary knowledge and skills in developing assessment tasksthat would elicit students’ higher-order thinking skills (Cizek cited in Pellegrino,Chudowsky, & Glaser, 2001). Teachers’ literacy in setting high-quality assessmenttasks would, in turn match the higher-order goals in teaching and learning (Wiliamet al., 2004). According to Navarro (2008):
260 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
. . . the general framework that guides the design and development of both formativeand summative assessments often does not address testing the full range of learning,from the procedural and memorization level, all the way through transfer to the high-est conceptual understanding that is demonstrated through transfer to new situations orto solving new problems. (Navarro, 2008, p. 254)
This notion reiterates the importance of improving teachers’ assessment literacy indesigning authentic classroom assessments that could be used to assess the fullrange of learning.
Method
Research design
Using a longitudinal, quasi-experimental group design, this study examined teach-ers’ assessment literacy over the course of two school years. The study’s partici-pants were teachers who taught Year 4 and 5 English, science, and mathematicsfrom eight neighborhood schools. The schools were matched based on their socio-demographic characteristics (i.e., type of school and ranking of students’ academicachievement) and were randomly assigned to one of two groups: four interventionschools or four comparison schools.
Intervention schools
Teachers from the intervention schools received ongoing, sustained professionaldevelopment over the course of two school years. These teachers were engaged in aseries of professional development workshops that focused on authentic assessmenttask design and rubric development. Additionally, the teachers participated in twomoderation meetings at the end of each school year to look together at the qualityof their classroom assessment tasks and student work while using a set of authenticintellectual quality criteria. During the monthly school meetings, the project’sresearcher and trained research assistants also met with the teachers to discussissues regarding the implementation of authentic assessment tasks and rubrics. Theprofessional development program was designed to include almost all the core fea-tures of effective professional development as below.
Content focus
The teachers engaged in the learning of concepts and principles of authentic assess-ments and rubrics in their respective subject areas. Such knowledge is related totheir teaching in the daily classroom.
Active learning
The teachers were actively involved in the analysis and moderation of assessmenttasks and related student work samples.
Coherence
The design and use of new forms of assessment were consistent with the curriculumreforms and policy initiatives in Singapore.
Teaching Education 261
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
Duration
Ongoing, sustained professional development workshops with activities that spreadover two school years were provided to the teachers. They were involved in work-shops during school holidays and monthly school meetings.
Collective participation
Teachers from the same school, grade, and department participated in the profes-sional development.
Comparison schools
During each school year, teachers from the comparison schools were given a one-day professional development workshop. The two ad-hoc workshops provided anoverview of authentic assessment and two hands-on sessions focused on task designand rubric development. Over the course of two teacher moderation sessions, theteachers were also taught how to analyze the quality of assessment tasks and stu-dent work using the given authentic intellectual quality criteria. However, nomonthly follow-up meetings were held with the teachers from the comparisonschools.
Data sources
The assessment tasks and associated student work samples were collected from boththe intervention and comparison schools at three points in time – before the inter-vention (baseline), at the end of the first year (Phase I), and at the end of the sec-ond year (Phase II). Toward the end of the study, a focus group interview wasconducted with the intervention schoolteachers about their conceptions of authenticassessment. The data served to corroborate the quantitative findings of teachers’assessment literacy.
Authentic intellectual quality rubrics
Five authentic intellectual quality criteria were used to train the participating teach-ers in authentic assessment task design and rubric development. Following New-mann and associates’ (1996) framework of authentic intellectual work, teachers’assessment tasks are expected to give students opportunities to demonstrate higher-order thinking, real-world problem-solving, and communication skills. The fiveauthentic intellectual-quality criteria are: depth of knowledge, knowledge criticism,knowledge manipulation, sustained writing, and making connections to the realworld beyond the classroom. Depth of knowledge includes three types of knowl-edge: factual knowledge, procedural knowledge, and advanced concepts based onthe revised Bloom’s knowledge taxonomy (Anderson & Krathwohl, 2001). Higher-order thinking is defined by two criteria: knowledge criticism and knowledge manip-ulation. Knowledge criticism is exemplified by tasks that ask students to compareand contrast different sources of information and critique knowledge, while knowl-edge manipulation is exemplified by tasks that demand the following from students:organize, analyze, interpret, synthesize, and evaluate information; apply knowledge
262 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
and skills; and construct new meaning or knowledge. In addition, sustained writingand making connections to the real world beyond the classroom are important forstudents to engage in the three knowledge domains.
A four-point rating scale was used for the scoring rubric of authentic intellectualquality for both teachers’ assessment tasks and student work. Through rigoroustraining of the participating teachers, the assessment tasks and student work sampleswere scored on each of the authentic intellectual quality criteria in four moderationsessions. The percentages of exact agreement ranged from 67% to 90% in English,65% to 99% in science, and 69% to 97% in mathematics, indicating moderate tohigh interrater reliability.
Results
The quantitative results of the analyses using both descriptive statistics and t-testswere presented on the following two aspects: (1) differences in the quality of teach-ers’ assessment tasks and student work between the intervention and comparisonschools; and (2) changes in the quality of teachers’ assessment tasks and studentwork over time for both intervention and comparison schools.
Comparisons of the quality of English teachers’ assessment tasks
As shown in Table 1, the mean score differences for the quality of English assessmenttasks between the intervention and comparison schools were compared on each of theauthentic intellectual criteria for each time point. The mean score differences were notstatistically significant, at the a = 0.05 level. At Phase I, the mean differences on theauthentic intellectual quality criteria were less obvious between the intervention and
Table 1. Changes in the quality of English assessment tasks.
Criteria
Mean score difference
Intervention schools Comparison schools
Baseline vs.Phase I
Baseline vs.Phase II
Baseline vs.Phase I
Baseline vs.Phase II
Factual knowledge 0.29 0.43 0.01 0.67⁄Procedural knowledge 0.43 0.57 �0.33 �0.18Advanced concepts 0.57 0.57 �0.16 �0.23Presentation of knowledge as given �0.43 �1.00⁄ �0.50 0.79⁄Comparing and contrastingknowledge
0.43 0.86⁄ 0.35 �0.54
Critique of knowledge 0.14 0.72⁄ 0.67 �0.51Knowledge reproduction 0.00 �1.00⁄ �0.78 0.33Organization, interpretation,analysis, synthesis,and/or evaluation
0.86⁄ 1.15⁄ 0.32 �0.07
Application 0.15 0.58 �0.34 �0.51Generation of new knowledge 0.43 0.86⁄ 0.35 �0.25Sustained writing 0.86⁄ 1.57⁄ 0.60 �0.24Connections to the real worldbeyond the classroom
0.71⁄ 1.43⁄ 1.57⁄ 1.37⁄
Note: ⁄Mean score differences between the intervention and comparison schools are significant (p < .05).
Teaching Education 263
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
comparison schools. At Phase II, the means on both factual and procedural knowl-edge were lower in intervention schools than in comparison schools. The same patternwas noted in presentation of knowledge as given and knowledge reproduction. DuringPhase II, the mean scores on higher-order thinking skills, such as understandingadvanced concepts, comparing and contrasting knowledge, critique of knowledge,application, generation of new knowledge, sustained writing, and making connectionsto the real world beyond the classroom, were significantly higher in interventionschools than comparison schools.
The results displayed in Table 1 also showed that most of the changes in meanscores from baseline to Phase II were statistically significant in the interventionschools. There were significant fewer assessment task demands on presenting knowl-edge as given and knowledge reproduction. Most of the English assessment taskstended to focus on assessing students’ higher-order thinking skills, sustained writing,and real-world application. In contrast, the assessment tasks collected from the com-parison schools from baseline to Phase II showed an increased demand for factualknowledge, presentation of knowledge as given, and knowledge reproduction.
Comparisons of the quality of student work in English
At baseline, the mean score differences on all of the authentic intellectual criteriawere not significant between the intervention and comparison schools. During PhaseII, student work from the intervention schools had lower mean scores than the stu-dent work from the comparison schools on factual knowledge, presentation ofknowledge as given, and knowledge reproduction. As for the criteria of higher-orderthinking, the mean scores of the intervention schools were significantly higher thanthose of the comparison schools.
Similar to the quality of English assessment tasks, the changes in scores from thestudent work in English on the authentic intellectual criteria were statistically signifi-cant for intervention schools from baseline to Phase II (see Table 2). Student workdemonstrated a significant decrease in presentation of knowledge as given and knowl-edge reproduction, whereas there was a significant increase in higher-order thinkingskills, sustained writing, and real-world application. An opposite pattern wasobserved in the change scores of student work in the comparison schools from base-line to Phase II.
Comparisons of the quality of science teachers’ assessment tasks
There were no significant baseline differences between the intervention and compari-son schools on the authentic intellectual quality criteria. During Phase II, the meanscores on factual knowledge, presentations of knowledge as given, and knowledgereproduction, were lower in the intervention schools as compared to the comparisonschools. In contrast, the mean scores of the intervention schools were higher thanthose of the comparison schools on the higher-order thinking domain, which includedunderstanding advanced concepts, comparing and contrasting knowledge, critique ofknowledge, organization, interpretation, analysis, synthesis and evaluation, problem-solving, generation of new knowledge, sustained writing, and making connections tothe real world.
As displayed in Table 3, the mean scores on factual knowledge, presentation ofknowledge as given, and knowledge reproduction had decreased from baseline to
264 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
Phase I in the intervention schools. These mean scores decreased further from base-line to Phase II with significant change scores. In contrast, a significant increase inthe mean scores from baseline to Phase II was observed in the intervention schoolson the following criteria: advanced concepts, critique of knowledge, organization,
Table 3. Changes in the quality of science assessment tasks.
Criteria
Mean score difference
Intervention schools Comparison schools
Baselinevs.Phase I
Baselinevs.Phase II
Baselinevs.Phase I
Baselinevs.Phase II
Factual knowledge �0.80⁄ �2.60⁄ 0.08 �0.64⁄Procedural knowledge 1.00⁄ 1.76⁄ �0.92⁄ �0.71⁄Advanced concepts 1.10⁄ 1.82⁄ �0.33 0.32Presentation of knowledge as given �0.50 �1.90⁄ 0.83⁄ �0.36Comparing and contrasting knowledge �0.30 1.06⁄ 0.33 0.54⁄Critique of knowledge 0.10 1.02⁄ 0.25 0.00Knowledge reproduction �0.20 �1.60⁄ 0.83⁄ �0.39Organization, interpretation, analysis,synthesis, and/or evaluation
0.80⁄ 1.84⁄ �0.50 0.11
Application/problem-solving 1.80⁄ 2.20⁄ �0.58⁄ 0.00Generation of new knowledge 0.90⁄ 2.14⁄ 0.08 0.57⁄Sustained writing 0.90⁄ 1.70⁄ 0.00 �0.07Connections to the real worldbeyond the classroom
0.40 2.00⁄ 0.08 0.89⁄
Note: ⁄Mean score differences between the intervention and comparison schools are significant (p < .05).
Table 2. Changes in the quality of student work in English.
Criteria
Mean score difference
Intervention schools Comparison schools
Baselinevs.Phase I
Baselinevs.Phase II
Baselinevs.Phase I
Baselinevs.Phase II
Factual knowledge �0.08 0.40 �0.01 0.61Procedural knowledge 0.23 0.62⁄ �0.06 �0.27Advanced concepts �0.01 0.71⁄ 0.15 �0.47Presentation of knowledge as given �0.65⁄ �1.37⁄ �0.56 1.23⁄Comparing and contrasting knowledge 0.25 1.08⁄ 0.10 �0.52Critique of knowledge 0.09 0.89⁄ 0.26 �0.71⁄Knowledge reproduction �0.38 �1.07⁄ �0.44 1.25⁄Organization, interpretation, analysis,synthesis, and/or evaluation
0.10 0.75⁄ 0.25 �0.03
Application �0.03 0.78⁄ 0.24 �0.18Generation of new knowledge 0.08 0.92⁄ 0.39 �0.44Sustained writing 0.61⁄ 1.54⁄ 0.51 �0.23Connections to the real worldbeyond the classroom
0.59⁄ 1.79⁄ 0.61 1.26⁄
Note: ⁄Mean score differences between the intervention and comparison schools are significant (p < .05).
Teaching Education 265
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
interpretation, analysis, synthesis and evaluation, problem-solving, generation ofnew knowledge, sustained writing, and making connections to the real world.
Comparisons of the quality of student work in science
The patterns of the mean score differences of science student work were similar tothose of the science assessment tasks. Table 4 shows that, at Phase II, the meanscores from the intervention schools were significantly lower than those from thecomparison schools on the following criteria: factual knowledge, presentation ofknowledge as given, and knowledge reproduction. A significant increase in meanscores was noted in the intervention schools on advanced concepts, comparing andcontrasting knowledge, critique of knowledge, organization, interpretation, analysis,synthesis and evaluation, problem-solving, generation of new knowledge, sustainedwriting, and making connections to the real world.
The changes in mean scores of student work on the authentic intellectual crite-ria followed the same patterns of assessment tasks (see Table 4). In the interven-tion schools, the changes in scores on the authentic intellectual criteria weresignificantly larger from baseline to Phase II than those changes from baseline toPhase I. An opposite pattern was noted for the comparison schools. From baselineto Phase I, student work in the comparison schools demonstrated a significantincrease on presentation of knowledge as given and knowledge reproduction. Incontrast, there was a significant decrease in the quality of student work in termsof advanced concepts, comparing and contrasting knowledge, organization, inter-pretation, analysis, synthesis, and evaluation, and problem-solving. The changescores from baseline to Phase II were somewhat smaller than those from baselineto Phase I in the comparison schools.
Table 4. Changes in the quality of student work in science.
Criteria
Mean score difference
Intervention schools Comparison schools
Baselinevs.Phase I
Baselinevs.Phase II
Baselinevs.Phase I
Baselinevs.Phase II
Factual knowledge 0.48 �1.22⁄ �0.52⁄ 0.27Procedural knowledge 1.10⁄ 1.76⁄ �0.84⁄ �0.82⁄Advanced concepts 0.52⁄ 1.42⁄ �0.42⁄ 0.45⁄Presentation of knowledge as given �0.50⁄ �2.00⁄ 1.28⁄ �0.38Comparing and contrasting knowledge 0.05 1.04⁄ �0.44⁄ 0.07Critique of knowledge 0.31 1.33⁄ �0.13 �0.01Knowledge reproduction �0.24 �1.89⁄ 1.27⁄ �0.34Organization, interpretation,analysis, synthesis, and/or evaluation
0.43⁄ 1.03⁄ �0.94⁄ �0.13
Application/problem-solving 1.23⁄ 1.68⁄ �0.40⁄ 0.26Generation of new knowledge 0.76⁄ 1.22⁄ 0.08 �0.25Sustained writing 0.96⁄ 1.75⁄ �0.03 0.10Connections to the real worldbeyond the classroom
0.96⁄ 2.01⁄ �0.17 0.80⁄
Note: ⁄ Mean score differences between the intervention and comparison schools are significant (p < .05).
266 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
Comparisons of the quality of mathematics teachers’ assessment tasks
On factual and procedural knowledge, presentation of knowledge as given, andknowledge reproduction, the mean scores from the intervention schools were lowerthan those from the comparison schools during Phase II. For the interventionschools, the mean scores were higher than those for the comparison schools on thehigher-order thinking domain, which included understanding advanced concepts,comparing and contrasting knowledge, critique of knowledge, organization, inter-pretation, analysis, synthesis, and evaluation, problem-solving, generation of newknowledge, sustained writing, and making connections to the real world. In thecomparison schools, the mean scores at Phase II were noted to be lower than thoseat baseline on the higher-order thinking criteria, such as advanced concepts, com-paring and contrasting knowledge, and organization, interpretation, analysis, syn-thesis, and evaluation, problem-solving, generation of new knowledge, sustainedwriting, and making connections to the real world.
As presented in Table 5, the mean scores from baseline to Phase II decreased sig-nificantly in the intervention schools on procedural knowledge, presentation of knowl-edge as given, and knowledge reproduction. There was a moderate increase inadvanced concepts, critique of knowledge, and generation of new knowledge. For themathematics assessment tasks in the comparison schools, a significant decrease wasfound in advanced concepts, comparing and contrasting knowledge, application orproblem-solving, sustained writing, and making connections to the real world.
Comparisons of the quality of student work in mathematics
The mean scores on all the authentic intellectual criteria at Phase II were higher inintervention schools than in comparison schools. The changes in mean scores in
Table 5. Changes in the quality of mathematics assessment tasks.
Criteria
Mean score difference
Intervention schools Comparison schools
Baselinevs.Phase I
Baselinevs.Phase II
Baselinevs.Phase I
Baselinevs.Phase II
Factual knowledge �0.07 �0.44 �0.30 �0.22Procedural knowledge �0.03 �0.47⁄ 0.25 �0.05Advanced concepts 0.43 0.50⁄ 1.15⁄ �0.45⁄Presentation of knowledge as given �0.69 �0.78⁄ �1.00⁄ 0.00Comparing and contrasting knowledge 0.32 0.27 0.02 �0.68⁄Critique of knowledge 0.52 0.95⁄ 0.71⁄ �0.09Knowledge reproduction �0.56 �0.58⁄ �1.22⁄ 0.18Organization, interpretation, analysis,synthesis, and/or evaluation
�0.09 �0.19 1.09⁄ �0.16
Application/problem-solving 0.14 0.16 1.07⁄ �0.73⁄Generation of new knowledge 0.13 0.97⁄ 0.82⁄ �0.18Sustained writing �0.09 �0.27 1.02⁄ �0.49⁄Connections to the real worldbeyond the classroom
0.54 �0.01 0.75⁄ �0.45⁄
Note: ⁄ Mean score differences between the intervention and comparison schools are significant (p < .05).
Teaching Education 267
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
intervention schools showed that mathematics student work demonstrated less pre-sentation of knowledge as given and knowledge reproduction at Phase I as com-pared to baseline (see Table 6). However, the mean scores on these two criteria atPhase II had increased, although there was a significant increase on comparing andcontrasting knowledge, organization, interpretation, analysis, synthesis and evalua-tion, and problem-solving. For the comparison schools, the mean scores on all theauthentic intellectual criteria decreased from baseline to Phase II. Further, there wasa significant decrease on problem-solving and sustained writing.
Teachers’ conceptions about authentic assessment
The focus group interview data were transcribed and coded for teachers’ concep-tions about authentic assessment as they emerged. Three themes related to teachers’conceptions of authentic assessment emerged from the data. They were authenticassessment, changes in task design, and rubrics.
Authentic assessment
The focus group interview excerpts revealed that teachers from the interventionschools had a better conception or understanding of what authentic assessmentwas after participating in the professional development for two years. They wereable to associate the features of authentic assessment with the criteria of authen-tic intellectual quality as used in the professional development workshops. Thefollowing paragraphs present the comments made by the participating teachersfrom each subject area regarding the authentic intellectual quality of their assess-ment tasks.
Table 6. Changes in the quality of student work in mathematics.
Criteria
Mean score difference
Intervention schools Comparison schools
Baselinevs.Phase I
Baselinevs.Phase II
Baselinevs.Phase I
Baselinevs.Phase II
Factual knowledge 0.09 0.55⁄ �0.21 �0.21Procedural knowledge �0.01 0.14 0.13 �0.16Advanced concepts 0.15 0.07 0.30 �0.43Presentation of knowledge as given �0.60⁄ 0.11 �0.19 �0.66Comparing and contrasting knowledge 0.31 0.59⁄ �0.21 �0.62Critique of knowledge 0.23 0.16 0.25 �0.11Knowledge reproduction �0.43⁄ 0.59⁄ �0.46⁄ �0.28Organization, interpretation, analysis,synthesis, and/or evaluation
0.05 0.79⁄ 0.19 �0.02
Application/problem-solving 0.30 0.70⁄ 0.38 �0.50⁄Generation of new knowledge 0.11 0.33 0.38 �0.02Sustained writing �0.20 0.04 0.56⁄ �0.39⁄Connections to the real worldbeyond the classroom
0.55⁄ �0.06 0.55⁄ �0.29⁄
Note: ⁄Mean score differences between the intervention and comparison schools are significant (p < .05).
268 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
English
Teacher J: Hmm, it can relate to the real world. Pupils can see that it is worth learningby doing authentic assessment tasks rather than taking tests and exam. Ithink authentic tasks can help us to develop children in a holistic way, likebeing artistic, being creative, being able to do things. For the normal tasksthat we are doing, we just develop children to take an exam. How has itchanged my teaching? I guess, I used to be more result-oriented. I lookedat their final results, but now, with the rubrics, I tend to think of the pro-gress they make.
Teacher C: I think it made lessons more interesting to the pupils because now theywere no longer just doing pen and paper work, worksheets; now, they aremore involved in authentic tasks. They will get to see themselves improv-ing, and, at the same time, their improvements will motivate them.
Science
Teacher H: It allows them to think more because it is less guided. We leave them tothink of what kind of materials they want to use. They have to think ofthe steps taken for the experiments. The tasks are more investigative, morehands on, with a more student-centered kind of learning. This method ismore open ended. It requires them to do research. Again, it is somethingdifferent from the activity book. They will have a more in-depth knowl-edge of this topic.
Teacher Y: Umm, they do this on their own, and I have already given them the notesand the information. So they are supposed to make use of what theyalready have and reorganize the whole thing to make linkages. So it isreally applying the skills. Activities in the activity book definitely haveless focus on thinking skills. For the authentic task, it is about learningthrough hands-on experience. It is also about linking whatever conceptsthey have learned and putting them together.
Teacher I: I think it also allows them to think what they are writing. Thinking of whatthey have learnt. Because sometimes if you just do quiz, you just try to catchwhat is the main thing. I mean just the answer only but here they have to planthe idea first and think over it. So it assesses their higher order thinking.
Mathematics
Teacher A: OK, the task is authentic; I think it is fun. It is a real project-based task;this is what the people are doing outside as in their workplace. We inte-grated different topics. We also focused on factual knowledge, for exam-ple, the area of a triangle and the facts about money, something that theyhave already learned and, procedural knowledge, whether they can calcu-late the cost of the flooring.
Teacher T: I set out to identify the task. The key criterion is whether it is related to a real-world situation. We wanted something that is real and not just hypothetical,so it would be more interesting for the girls. From the accuracy of their calcu-lations, we can tell whether they know their concepts; from their reasoning,we know whether they can analyze and think clearly, and we understand theirlogical process to problem-solving. Factual knowledge-wise, we want toknow if they know the area of a rectangle and a triangle. Procedural knowl-edge, they need to know the operations, the steps to complete the task.
Teaching Education 269
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
These teachers’ comments all indicated that their conceptions about authenticassessment had improved as a result of active learning in the professional develop-ment workshops. In designing authentic assessment tasks, teachers in all three sub-ject areas had taken into account the criteria of authentic intellectual quality. Theywere competent in articulating the key features of authentic assessment tasks andmaking meaningful associations to the higher order learning outcomes. These arethe two key criteria of assessment literacy as defined by Stiggins (1995).
Clearly, the mathematics teachers considered the importance of assessing stu-dents’ factual and procedural knowledge in addition to higher-order thinking skillsand real-world problem-solving. Compared to the mathematics teachers, the Englishand science teachers were more able to make significant changes in their assessmenttasks because they could see how the changes in classroom assessment had shapedtheir teaching and student learning. For example, Teachers J, C, H, I, and Y statedthat they found the differences between authentic assessment and conventionalassessment in measuring students’ outcomes.
Changes in task design
The theme of changes in task design recurred most frequently in the transcripts.The teachers were able to change their thinking and planning of assessment tasksafter their participation in the professional development. Prior to crafting the assess-ment tasks, they began with identifying the learning objectives or goals and the col-lective participation of teachers who taught the same subject, level, and schoolmade the task design process more exciting and effective for the teachers.
English
Teacher M: Before the project, I guess you went ahead to do it without much thoughtabout the end product as in the assessment modes and all that. For thistask, you have to think ahead. How are you going to design it in a waythat it is aligned with the rubrics that you set up? And how are the kidsgoing to perform? You’ve got to think about how they will be able tolevel up.
Science
Teacher H: Before this, I make use of lead.com. Other than textbook right, I alsomake use of lead.com. They provide a tutorial and also quizzes for stu-dents to complete and some fun activities as well. After participating in thisproject, teachers who teach at the same level are supposed to sit downtogether to plan a lesson and of course it must be something different. Morestudent-centered kind of learning. More investigative tasks, more hands-ontasks, more student-centered learning. Our objectives must be clear ofcourse. We must sort of like provide them (students) with an example first.When we are designing the tasks, we are actually very excited.
Mathematics
Teacher A: OK. The task objectives were actually with reference to our scheme ofwork. This task actually emphasizes the mathematical connections toeveryday life.
270 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
Rubrics
Teachers’ knowledge and skills of crafting rubrics is one of the criteria of assessmentliteracy. As evidenced by the focus group data, the participating teachers not only hada better conception of rubrics, but also believed that they were competent in designingrubrics. They also commented that rubric is an essential tool in alternative assessmentand can be used to give formative feedback to students. It indicates that the teacherswere able to appreciate the use of the assessment information for formative assessmentor assessment for learning. For example, Teacher I said, ‘If there is a rubric you canlook into it, and then children understand it because we have a rubric to guide us. Ifyou mark it wrong then you need to explain why it is wrong’.
English
Teacher C: What I’ve done is I’ve applied. Maybe make changes to the way theythink of rubrics. That’s the only impact we’ve managed to make in Star-light, to relook at our rubrics and it’s like a collaborative thing. Beforejoining the project, people had different ideas of rubrics. Now at least therubrics in our school are slightly better.
Science
Teacher I: You can consider it as an alternative kind of assessment. Very differentfrom the usual one we always do, marking! OK, if there is a rubric youcan look into it, and then children understand it because we have a rubricto guide us. If you mark it wrong then you need to explain why it iswrong.
Teacher Y: I think first I need to list down what I’m assessing for that task. What arethe specific skills that I am looking for in an assessment? Whether I amlooking at the product or I am looking more on the process. How muchweightage would I want to give to the product and process?
Mathematics
Teacher A: Designing rubrics. We are all very familiar with rubrics being categorizedinto four levels. Level 1 shows no understanding. Then the second levelwould be showing a little understanding depending on the concepts andthe third level shows some understanding but there are still some errors inthem. Level 4 would be those show complete understanding.
Discussion
The findings in the English and science subject areas indicated that the authentic intel-lectual quality of teachers’ assessment tasks had significantly improved after the inter-vention. Such improvement was also observed in the quality of student work.Teachers’ increased use of authentic assessments in English and science were wellaligned with their improved assessment literacy and conceptions about authenticassessment. In addition, they also found that, through the use of authentic assessments,students were engaged in authentic learning of knowledge and skills. Furthermore, theuse of rubrics allowed students to assess their own progress and motivated them tomake progress toward meeting the standards. It is one of the important strategies ofassessment for learning in the day-to-day learning process. These benefits could nothave been achieved by using conventional paper-and-pen assessments alone.
Teaching Education 271
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
The quality of mathematics assessment tasks and student work had improvedslightly. This slight improvement could be attributed to the nature of the subject,which tends to emphasize the reproduction of factual and procedural knowledge.This finding is not surprising because many mathematics teachers still believe thatstudents’ mastery of factual and procedural knowledge is important for their con-ceptual understanding. This notion also echoes Hiebert and Carpenter’s (1992) argu-ment for the importance of emphasizing both procedural and conceptual knowledgein mathematics teaching.
This study provides some insight into the format of effective professional devel-opment for equipping teachers with contemporary knowledge and skills in develop-ing authentic assessments and rubrics. The findings concur with the teacher learningand professional development literature arguing that ongoing, sustained professionaldevelopment is more powerful than short-term, one-shot professional developmentworkshops (McMunn et al., 2004; Wiliam & Thompson, 2008). Additionally, thecore features of effective professional development such as content focus, activelearning, coherence, duration, and collective participation are necessary to be takeninto consideration in the planning and design of professional development pro-grams. The findings further suggest that, when teachers are better prepared and lit-erate in developing authentic assessments, they tend to adopt the use of authenticassessments in their day-to-day classroom practice, resulting in better quality workfrom students (Bol et al., 1998; Stiggins, 1991b).
One of the most recent recommendations made by the Singapore Primary Edu-cational Review and Implementation Committee (PERI, 2009) has called forschools to explore the use of bite-sized forms of assessments (i.e., topical tests) toprovide regular feedback to students and their parents regarding students’ learning.According to Klenowski (2009), there is a possibility that this form of assessmentcould encourage performance-orientated learning to the detriment of sustained andreal learning. Klenowski’s concern is valid for two reasons: (1) many teachers mis-construed formative assessment as frequent, mini-summative assessments, and (2)many teachers contended that they have adopted formative assessment practices bymerely using summative tests for formative purposes, such as answer checking.According to Harlen (2006), one of the limitations of using summative tests as evi-dence to improve student learning is that the information derived from the summa-tive tests is not sufficiently detailed to be diagnostic. Thus, the most importantcomponent in teacher professional development in assessment is to equip teacherswith the knowledge and skills to develop and implement authentic assessment andrubrics. Such assessment tools have a closer alignment with specific learning goalsand give more detailed diagnostic information. These authentic assessment tools aredeemed more appropriate for formative assessment purposes. As evidenced by thefocus group interview data, teachers in this study were able to appreciate the valueof using authentic assessment tasks and rubrics to shape their own teaching and stu-dent learning. The professional development experience provided them with ampleopportunities to engage in active learning of task design and rubric development.The collective participation of teachers in the learning communities brought aboutpositive changes in their conceptions of authentic assessment, task design, and rub-rics. This finding explains the improved quality of teachers’ assessment tasks andstudent work. It also suggests that changing teachers’ assessment practices entaileda shift in conceptions and beliefs about the new forms of assessment. When teach-ers are able to make connections between assessment methods and higher order
272 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
instructional goals or learning outcomes, they will be more willing to make changesin their classroom practices.
Implications for teacher education and professional development
The findings of this study call for more localized, ongoing and sustained profes-sional development for teachers. The findings also concur with McMunn et al.’s(2004) contention that actualizing assessment reform at the school or classroomlevel is a long-term endeavor that will not happen as the result of a single or spo-radic workshop. Cohen and Hill (1998) have discussed the limited value of one-time professional development workshops with respect to sustaining changes in tea-cher practice. As demonstrated by the negligible changes in the quality of teachers’assessment tasks and student work from the comparison schools, where teachersreceived only two assessment workshops over the course of the study, professionaldevelopment can no longer be viewed as an ad hoc event that occurs on only a fewdays of the school year. Therefore, it is of paramount importance to take intoaccount the sustainability factors in the planning and implementation of professionaldevelopment programs for in-service teachers.
Professional development must also be part of the daily professional practice ofteachers, meaning that teachers should be encouraged to actively collaboratethrough school-based professional learning communities that are sustained overtime. This type of networking would provide teachers with ample opportunity forself-reflection and dialogue with colleagues and allow for changes in teachers’assessment practices to occur developmentally (Wiliam & Thompson, 2008).
The analysis of student work in moderation meetings is an effective strategy thathas been used increasingly in teacher education and professional development pro-grams to help teachers examine student learning and reflect on the teaching associ-ated with the learning (Shepard et al., 2005). The benefit of such strategy wouldalso help new teachers develop an understanding of how classroom assessment caninform their instructional choices. The literature on teacher professional develop-ment has suggested that teachers can improve their classroom practices when theycollectively review student work to do the following: analyze what has been learntby students, uncover students’ misconceptions, and reflect on their own curriculumor instructional adaptations necessary to promote student understanding. Such anactive learning approach has been recognized as one of the core features of effec-tive professional development in the teacher learning and professional developmentliterature. In addition, the rubric helps teachers to deepen and reflect on the qualityof classroom assessment tasks and the impact on the quality of student work.According to Aschbacher (1999), such a tool is useful in both pre-service and in-service professional development because teachers are able to reflect on the teachingand assessment associated with student learning.
Shepard et al. (2005) also noted that engagement in assessment design is one ofthe promising pedagogical approaches in helping new teachers develop an under-standing of student learning. The participating teachers from this study’s interven-tion schools lamented their lack of assessment preparation during their pre-serviceteacher training programs. However, their assessment literacy had increased afterparticipating in the professional development workshops on developing authenticassessment tasks and rubrics, which is a key component of the intervention. Theteachers had also gained a better understanding of student learning because they
Teaching Education 273
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
were required to think about the learning goals and success criteria in the processof task design and rubric development. These promising findings suggest that theprofessional development workshops can be scaled to include more schools to bene-fit more teachers. Furthermore, the contents of the professional development work-shops used with the intervention school teachers can be adopted for pre-serviceteacher training. Graham’s (2005) and Volante and Fazio’s (2007) findings in theUnited States and Canada, respectively, have shown that many teacher candidatesexpressed the need for improving their assessment literacy. New teachers’ lack ofpreparation in classroom assessment is a commonly encountered problem by manyeducational systems around the world. There is thus a need for assessment trainingto begin as early as possible in pre-service teacher education.
Although teacher professional development has become increasingly importantto prepare teachers for the curriculum and assessment of twenty-first century com-petencies and many programs have been put in place at both the school and systemlevel, policy-makers and teacher educators should not neglect to collect valuableformative and summative data that might be useful to inform ongoing developmentsand the planning of future professional development programs (Luke & McArdle,2009). The quantitative findings of this study derived from a longitudinal, quasi-experimental design show the strength of the data, which ruled out the confoundingvariables (e.g., teacher factors or novelty effects) that might lead to spurious effectsof professional development (intervention). The findings also contribute to the tea-cher learning and professional development literature by establishing the linksbetween the effects of professional development, teachers’ assessment literacy, andstudent learning. Given that the data were based on the artifacts (i.e., assessmenttasks and student work samples) embedded within teachers’ day-to-day classroominstruction, they were the immediate measures of teachers’ classroom practice andstudent learning. Such measures took into account instructional sensitivity andhence the curricular validity of the classroom data was established in the study(Ruiz-Primo, Shavelson, Hamilton, & Klein, 2002). As a result, we can firmlybelieve that the effects of professional development on the quality of teachers’assessment tasks and student work were not due to measurement errors.
One limitation of this study is that it focused on improving teachers’ assessmentliteracy within the realm of authentic assessment task design and rubric develop-ment. Although it is essential to equip teachers with contemporary knowledge andskills in developing assessment tools that tap into conceptual understanding andhigher-order thinking skills, their competency in using assessment information toassist student learning through timely, formative feedback is equally important.Because of the importance of using formative assessment to support learning (Black& Wiliam, 1998; Hattie & Timperley, 2007), future studies should focus on build-ing teachers’ capacity in formative assessment through professional development.
AcknowledgementsThe author would like to thank the Singapore Ministry of Education for funding thisresearch. The author is also grateful to the participating teachers in the study and theresearch assistants who helped with the data collection.
ReferencesAnderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assess-
ing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.
274 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
Aschbacher, P.R. (1991). Performance assessment: State activity, interest, and concerns.Applied Measurement in Education, 4(4), 275–288.
Aschbacher, P.R. (1999). Developing indicators of classroom practice to monitor and supportschool reform. CSE Technical Report 513. Los Angeles, CA: University of California,Los Angeles.
Bernstein, B. (1990). Class, codes and control: The structuring of pedagogic discourse.(Vol. 4). London: Routledge.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Educa-tion: Principles, Policy, and Practice, 5, 7–74.
Bol, L., Stephenson, P.L., O’Connell, A.A., & Nunnery, J.A. (1998). Influence of experience,grade level, and subject area on teachers’ assessment practices. The Journal of Educa-tional Research, 91, 323–330.
Bol, L., & Strage, A. (1996). The contradiction between teachers’ instructional goals andtheir assessment practices in high school biology courses. Science Education, 80, 145–163.
Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Edu-cational Researcher, 33(8), 3–15.
Borko, H., Mayfield, V., Marion, S., Flexer, R., & Cumbo, K. (1997). Teachers’ devel-oping ideas and practices about mathematics performance assessment: Successes,stumbling blocks, and implications for professional development. Teaching & Tea-cher Education, 13(3), 259–278.
Cohen, D.K., & Hill, H.C. (1998). Sate policy and classroom performance: Mathematicsreform in California. CPRE Policy Brief No RB-23. Philadelphia: Consortium for PolicyResearch in Education, University of Pennsylvania.
Desimone, L.M. (2009). Improving impact studies of teachers’ professional development:Toward a better conceptualizations and measures. Educational Researcher, 38(3), 181–199.
Desimone, L.M., Porter, A., Garet, M.S., Yoon, K.S., & Birman, B.F. (2002). Effects of pro-fessional development on teachers’ instruction: Results from a three-year longitudinalstudy. Educational Evaluation and Policy Analysis, 24(2), 81–112.
Fleming, M., & Chambers, B. (1983). Teacher-made tests: Windows on the classroom. In W.E. Hathaway (Ed.), Testing in the schools (pp. 29–38). San Francisco: Jossey-Bass.
Fullan, M.G., & Miles, M.B. (1992). Getting reform right: What works and what doesn’t.Phi Delta Kappan, 73(10), 744–752.
Garet, M.S., Porter, A.C., Desimone, L.M., Birman, B.F., & Yoon, K.S. (2001). What makesprofessional development effective? Results from a national sample of teachers. Ameri-can Educational Research Journal, 38(4), 915–945.
Graham, P. (2005). Classroom-based assessment: Changing knowledge and practice throughpreservice teacher education. Teaching and Teacher Education, 21, 607–621.
Guskey, T.R. (2002). Professional development and teacher change. Teachers and Teaching:Theory and Practice, 8(3/4), 381–390.
Hargreaves, A., Earl, L., & Schmidt, M. (2002). Perspectives on alternative assessmentreform. American Educational Research Journal, 39(1), 69–95.
Harlen, W. (2006). On the relationship between assessment for formative andsummative purposes. In J. Gardner (Ed.), Assessment and learning (pp. 103–117).London: Sage.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research,77, 81–112.
Hiebert, J., & Carpenter, T.P. (1992). Learning and teaching with understanding. In D.A.Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 65–97).New York: Macmillan.
Klenowski, V. (2009). Assessment for learning revisited: An Asia-Pacific perspective.Assessment in Education: Principles, Policy & Practice, 16(3), 263–268.
Lingard, B., Ladwig, J., Mills, M., Bahr, M., Chant, D., Warry, M., Ailwood, J., Capeness,R., Christie, P., Gore, J., Hayes, D., & Luke, A. (2001). The Queensland School ReformLongitudinal Study. Brisbane: Education Queensland.
Luke, A., & McArdle, F. (2009). A model for research-based state professional developmentpolicy. Asia-Pacific Journal of Teacher Education, 37(3), 231–251.
Teaching Education 275
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
McMunn, N., McColskey, W., & Butler, S. (2004). Building teacher capacity in classroomassessment to improve student learning. International Journal of Educational Policy,Research, & Practice, 4(4), 25–48.
Navarro, M.S. (2008). Assessment and school reform: Lessons from 15 years in the field. InC.A. Dwyer (Ed.), The future of assessment: Shaping teaching and learning (pp. 245–262). New York: Lawrence Erlbaum Associates.
Newmann, F.M., & Associates. (1996). Authentic achievement: Restructuring schools forintellectual quality. San Francisco: Jossey Bass.
Newmann, F.M., Marks, H.M., & Gamoran, A. (1996). Authentic pedagogy and student per-formance. American Journal of Education, 104, 280–312.
O’Day, J.A., & Smith, M.S. (1993). Systemic reform and educational opportunities. In S.H.Fuhrman (Ed.), Designing coherent educational policy: Improving the system (pp. 1–34).San Francisco: Jossey-Bass.
Pellegrino, J.W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: Thescience and design of educational assessment. Washington, DC: National AcademyPress.
Ruiz-Primo, M.A., Shavelson, R.J., Hamilton, L., & Klein, S. (2002). On the evaluation ofsystemic science reform: Searching for instructional sensitivity. Journal of Research inScience Teaching, 39(5), 369–393.
Saddler, R. (1998). Formative assessment: Revisiting the territory. Assessment in Education,5, 77–84.
Sato, M., Wei, R.C., & Darling-Hammond, L. (2008). Improving teachers’ assessment prac-tices through professional development: The case of National Board Certification. Ameri-can Educational Research Journal, 45(3), 669–700.
Shepard, L., Hammerness, K. Darling-Hammond, L., et al. (2005). Assessment. In L.Darling-Hammond & J. Bransford (Eds.), Preparing teachers for a changing world:What teachers should learn and be able to do (pp. 275–326). San Francisco: John Wiley& Sons.
Smith, D.C., & O’Day, J. (1990). Systemic school reform. London: Taylor and Francis.Stiggins, R.J. (1991a). Relevant classroom assessment training for teachers. Educational
Measurement: Issues and Practice, 10(1), 7–12.Stiggins, R.J. (1991b). Assessment literacy. Phi Delta Kappan, 72, 534–539.Stiggins, R.J. (1995). Assessment literacy for the 21st century. Phi Delta Kappan, 77(3),
238–245.Stiggins, R.J. (1999, November). Assessment, student confidence, and school success. Phi
Delta Kappan, 81, 191–198.Stiggins, R.J. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta
Kappan, 83(10), 758–765.Stiggins, R.J., & Conklin, N.F. (1992). In teachers’ hands: Investigating the practices of
classroom assessment. Albany: State University of New York Press.Volante, L., & Fazio, X. (2007). Exploring teacher candidates’ assessment literacy: Implica-
tions for teacher education reform and professional development. Canadian Journal ofEducation, 30(3), 749–770.
Wiggins, G. (1989, May). A true test: Toward more authentic and equitable assessment. PhiDelta Kappan, 70, 703–713.
Wiliam, D., Lee, C., Harrison, C., & Black, P. (2004). Teachers developing assessment forlearning: Impact on student achievement. Assessment in Education, 11(1), 49–65.
Wiliam, D., & Thompson, M. (2008). Integrating assessment with learning: What will it taketo make it work? In C.A. Dwyer (Ed.), The future of assessment: Shaping teaching andlearning (pp. 53–82). New York: Lawrence Erlbaum Associates.
276 K.H. Koh
Dow
nloa
ded
by [
Mon
ash
Uni
vers
ity L
ibra
ry]
at 1
2:57
04
Dec
embe
r 20
14
