K–3 Formative Assessment Enhanced Assessment Grant: Final Project Report September 2017
September 29, 2017
Prepared for: Cindy Bagwell North Carolina Department of Public Instruction Contract No. NC10130204 Prepared by: SRI International Suzanne Raber Sara Thayer Megan Cox Kathleen Hebbeler Traci Kutaka Suggested citation: Raber, S., Thayer, S., Cox, M., Hebbeler, K., & Kutaka, T. (2017). K–3 Formative Assessment Enhanced Assessment Grant: Final Project Report. Menlo Park, CA. SRI International. Acknowledgements We would like to thank the dedicated teachers and administrators who participated in the cognitive lab, pilot, and field test phases of this research as well as the state consortium members who provided their guidance to the development and enhancement of high quality formative assessment in early elementary classrooms: Amy, Dan and Steve in North Carolina; Lauren, Nicol, Pat and Leila in Arizona; Penny, Jennifer and Kimberly in Iowa; Jaci, Lee Anne and Sue in Maine; and Judi, Michelle and Diane in Rhode Island. We would especially like to thank Dr. Cindy Bagwell for her steadfast and inspiring leadership of this project.
This document was developed under a grant from the U.S. Department of Education. However, the contents do not necessarily represent the policy of the Department and you should not assume endorsement by the U.S. Federal Government or the North Carolina State Board of Education.
EAG K-3 Formative Assessment: Final Report September 2017 iii
Contents Executive Summary ................................................................................................................... ix
Background ............................................................................................................................ ix
Findings .................................................................................................................................. ix
Chapter 1: Introduction ................................................................................................................ 1
Project background and overview ............................................................................................ 1
Overview of the assessment .................................................................................................... 2
Initial ECD phases ................................................................................................................... 4
ECD codesign teams ........................................................................................................... 4
Cognitive labs ...................................................................................................................... 5
Pilot test ............................................................................................................................... 5
Field test design ...................................................................................................................... 6
Report organization ................................................................................................................. 8
Chapter 2: Validity of assessment content ................................................................................ 10
Are the skill levels of the construct progressions in the proper sequence? ............................ 10
Teacher survey results ....................................................................................................... 10
Distribution of children’s placements on progressions ........................................................ 11
Psychometric models ......................................................................................................... 14
Do the performance levels accurately reflect what is expected at entry to kindergarten? ....... 17
Is the assessment sensitive to children’s growth and development over time? ...................... 24
Teacher survey results ....................................................................................................... 24
Fall to winter change in placements on the progressions ................................................... 25
Placement on the progressions across grade levels .......................................................... 29
Does the assessment provide information that is useful for instruction for grades K–3? ........ 31
Does the assessment provide information useful for instruction for different subgroups, such as children with disabilities and English learners? ................................................................. 34
Children with disabilities ..................................................................................................... 34
English learners ................................................................................................................. 35
Do the assessment results vary for other subgroups (gender, race)? .................................... 35
Gender ............................................................................................................................... 36
Race/Ethnicity .................................................................................................................... 36
Chapter 3: Implementation of assessment process ................................................................... 38
Can teachers learn to implement the assessment with fidelity? ............................................. 38
EAG K-3 Formative Assessment: Final Report September 2017 iv
Implementation of the recommended assessment process ................................................ 39
Teacher confidence in status summary .............................................................................. 41
Teacher reliability in coding status summaries ................................................................... 42
Incorporating the assessment into classroom routines .......................................................... 46
Chapter 4: Implementation supports for teachers ...................................................................... 49
Professional development ..................................................................................................... 49
Materials and ongoing supports ............................................................................................. 51
Materials ............................................................................................................................ 51
Ongoing supports ............................................................................................................... 51
Teachers’ perceptions of the supports ............................................................................... 52
Technology ............................................................................................................................ 54
Chapter 5: Broader implementation supports ............................................................................ 56
Professional development for administrators ......................................................................... 56
Building-level infrastructure ................................................................................................... 58
Classroom conditions ............................................................................................................ 59
Technology available to support the assessment ............................................................... 59
Teacher training and preparation to implement formative assessment ............................... 60
Class time spent in different learning arrangements ........................................................... 60
Chapter 6: Using the K–3 Formative Assessment with all children ............................................ 62
Disability perspective ............................................................................................................. 62
Overview guidance and general recommendations ............................................................ 63
Construct-specific guidance ............................................................................................... 64
English learner perspective ................................................................................................... 67
Race/culture perspective ....................................................................................................... 68
Chapter 7: Summary, revisions, and recommendations ............................................................ 70
Overall assessment findings .................................................................................................. 70
Findings by Construct ............................................................................................................ 73
Language Development & Communication Domain ........................................................... 76
Physical/Motor Development Domain ................................................................................ 77
Cognitive Development Domain ......................................................................................... 77
Social-Emotional Development and Approaches to Learning Domains .............................. 78
Recommended grade levels for each construct ..................................................................... 79
Revisions/recommendations for assessment materials ......................................................... 80
EAG K-3 Formative Assessment: Final Report September 2017 v
Clarify the meaning of the construct ................................................................................... 83
Clarify progression skills (examples, watch fors, assessment means) ................................ 83
Conduct additional research on some constructs ............................................................... 83
Revisions to training materials and recommendations ........................................................... 84
Revisions to teacher and administrator training materials .................................................. 84
Recommendations regarding training ................................................................................. 84
Conclusion ............................................................................................................................ 85
References ............................................................................................................................... 87
Appendices ............................................................................................................................... 89
EAG K-3 Formative Assessment: Final Report September 2017 vi
Exhibits Exhibit Page
1.1. EAG K–3 assessment blueprint ....................................................................................... 3
1.2. Field test sample, by state ............................................................................................... 7
1.3. Field test sample, by grade .............................................................................................. 7
1.4. Demographics of children assessed in field test (Fall) ..................................................... 8
2.1. Unimodal distribution with smooth tails .......................................................................... 12
2.2. Unimodal, smooth tails, skewed to the higher end of the progression ............................ 12
2.3. Unimodal, smooth tails, skewed to the lower end of the progression ............................. 13
2.4. Unimodal, uneven tails .................................................................................................. 13
2.5. Example of Increasing mean “School Readiness” scores for each level of the progression: Emotion Regulation Strategies (Fall Kindergarten) .................................... 15
2.6. Example of skill levels too close together: Following Directions ..................................... 16
2.7. Teacher survey responses for Book Orientation: Percentage of teachers indicating each level ...................................................................................................... 20
2.8. Teacher survey responses and child assessment results for Book Orientation: Percentage of teachers indicating each level ................................................................. 21
2.9. Teacher responses, fall K data and performance levels for Book Orientation: Percentage of teachers indicating each level ................................................................. 22
2.10. Teacher responses and resulting performance levels at kindergarten entry .................. 23
2.11. Teacher ratings of how well construct progressions captured children’s growth ............ 25
2.12. Example comparing fall and winter progression placements: Grip & Manipulation (percentage of children) ................................................................................................. 26
2.13. Examples of fall to winter progression placements of children in kindergarten (percentage of children) ................................................................................................. 27
2.14. Percentage of children at maximum level and grade levels with opportunity for growth for each progression ..................................................................................... 28
2.15. Percent of children not at highest level in fall who were placed at a higher level in winter for each progression ............................................................................... 30
2.16. Example comparing kindergarten and grade 1 progression placements: Percentage of children placed on each level of the progression for Following Directions .................. 31
2.17. Teachers’ rating of constructs as “very” or “somewhat” helpful to instruction ................. 32
2.18. Teachers’ report of benefits of implementing the K–3 Formative Assessment ............... 33
3.1. Percentage of children with status summaries, multiple pieces of evidence, and learning statuses .................................................................................................... 41
EAG K-3 Formative Assessment: Final Report September 2017 vii
3.2. Percentage of teachers confident in progression placement (n = 130) ........................... 42
3.3. Evidence specificity by construct ................................................................................... 45
3.4. Evidence analysis: Percentage of evidence specific enough to code, coded status summary consistent with the teacher status summary, and coded status summary matched the teacher’s exactly ....................................................................................... 46
3.5. Percentage of teachers who reported they were unable to assess constructs during everyday instruction: Most challenging constructs .............................................. 47
3.6. Percentage of teachers who reported each challenge experienced in implementing the assessment ....................................................................................... 48
4.1. Aspects of training teachers new to assessment found most helpful ............................. 50
4.2. Sample page from Teacher Site showing construct materials ........................................ 52
4.3. Percentage of teachers rating materials as very or somewhat helpful ........................... 53
4.4. Percentage of teachers reporting supports provided and suggestions for supports needed for assessment implementation ........................................................................ 54
5.1. Time spent in different classroom learning arrangements .............................................. 61
6.1. Findings from review for each construct by disability ..................................................... 65
7.1. Summary of construct performance for each of the validity claims ................................. 75
7.2. Summary of construct-specific revisions and recommendations .................................... 81
EAG K-3 Formative Assessment: Final Report September 2017 ix
Executive Summary
Background
In 2011, North Carolina’s Department of Public Instruction began work on a formative assessment to be administered at kindergarten entry and throughout kindergarten into grades 1–3 with the primary purpose of informing instruction. In September 2013, North Carolina received an Enhanced Assessment Grant from the U.S. Department of Education to lead a 10-state consortium in the enhancement of this assessment. SRI worked with North Carolina and the other states in the Consortium, along with two other partners, the Build Initiative and Child Trends, to enhance this assessment.
The assessment, known as the K–3 Formative Assessment, consists of 18 construct
progressions in five domains of learning and development. Teachers gather information to
identify the child’s skill level on each progression through their everyday classroom activities.
The teacher captures evidence of what a child can do through notes, work samples, photos, or
videos and then uses this evidence to identify the child’s skill level on the progression. The
teacher also enters both the evidence and the skill level for each construct into a technology
platform. Additional assessment materials available to teachers include background information
on the construct and performance descriptors, examples for each skill level of the progression,
and guidelines on how to assess each construct.
SRI used evidence-centered design (ECD), a highly regarded research-based approach to
assessment development, to enhance the progressions and the assessment materials. The four
phases of the ECD process included review and revisions of the content of the assessment by
ECD codesign teams; cognitive laboratories with teachers to try out the assessment materials
with small numbers of children; a pilot test in K–3 classrooms in four states; and a field test with
additional K–3 classrooms in the same four states.
This report summarizes the results of the field test, which was designed to examine the validity
of the information provided by the assessment and the supports required to implement the
assessment with fidelity. The assessment was field tested in the 2016–17 school year in
Arizona, Iowa, Maine, and Rhode Island with 141 teachers and 1,312 students in kindergarten
through grade 3.
Findings The field test demonstrated that the K–3 Formative Assessment can be implemented by
teachers as intended, provided that teachers have adequate training in the assessment content
and the formative assessment process and have access to functioning technology that supports
EAG K-3 Formative Assessment: Final Report September 2017 x
the assessment. Most of the construct progressions performed well as measures of children’s
knowledge, skills, and abilities, both in terms of the sequencing of the skill levels on the
progression and their sensitivity to growth and development over time. Some of the
progressions presented some challenges for teachers. The analysis of the evidence
documented in the technology platform showed that the reliability with which teachers coded a
skill level for a child varied by construct. Teacher feedback reflected this variation, in that fewer
teachers felt confident about where they placed children on the progressions that had lower
reliability. Teachers also found some constructs more difficult than others to incorporate into
their daily classroom routines. Of particular importance for a formative assessment was the
finding that across constructs, between 20 and 50 percent of teachers did not find a given
construct useful in informing instruction. This may be a characteristic of the assessment that
needs to be improved or it could indicate a need for more professional development around how
to use the results of a formative assessment to inform instruction.
Overall, we were encouraged by the performance of most of the constructs in a field test setting
where most teachers had limited experience with the constructs and formative assessment
process. As teachers continue to work with the K–3 Formative Assessment, we recommend
additional research and professional development on the construct progressions that presented
some issues with their content and proved challenging to implement.
EAG K-3 Formative Assessment: Final Report September 2017 1
Chapter 1: Introduction This is the final report on the K–3 Formative Assessment, which the U.S. Department of Education funded under an Enhanced Assessment Grant in September 2013 to enhance an assessment developed by the North Carolina Department of Public Instruction under its Race to the Top Early Learning Challenge Grant. This report covers the work completed by SRI Education, a division of SRI International, using evidence-centered design.
Project background and overview North Carolina began the original work in 2011 to develop a kindergarten entry assessment
under a Race to the Top Early Learning Challenge Grant (NC K−3 Assessment Think Tank,
2013). North Carolina developed a formative assessment to be administered at kindergarten
entry and throughout kindergarten into grades 1–3 with the primary purpose of informing
instruction. In September 2013 North Carolina received an Enhanced Assessment Grant from
the U.S. Department of Education to lead a 10-state consortium in the further development and
enhancement of this K–3 assessment. The other states were Arizona, Delaware, Iowa, Maine,
North Dakota, Oregon, and Rhode Island, with the District of Columbia and South Carolina
participating as collaborating partners. North Carolina engaged three research partners to
support the enhanced assessment:
• SRI Education to further develop and enhance the assessment
• Child Trends to address longer term implementation of the assessment
• Build Initiative to support stakeholder and family engagement efforts.
SRI used evidence-centered design (ECD), a highly regarded research-based approach to
assessment development (Mislevy & Haertel, 2006), to enhance this assessment. The ECD
process was conducted in four phases over the 4-year project as follows:
1. ECD codesign teams reviewed and revised the assessment materials for each of the
constructs in the assessment.
2. Cognitive laboratories were conducted with teachers to try out the assessment materials
with small numbers of children, and the assessment and materials were revised based on
the findings.
3. The assessment was piloted in K–3 classrooms in four states, and the assessment and
materials were revised based on those findings.
4. The revised assessment was field-tested with additional K–3 classrooms in four states.
EAG K-3 Formative Assessment: Final Report September 2017 2
The major findings from the first three phases of ECD are summarized below and detailed in
Appendices A–C. This report concentrates on the findings from the field test of the K–3
Formative Assessment, which was conducted during the 2016–17 school year.
Overview of the assessment The application of ECD to the development of learning progressions under the Enhanced
Assessment Grant resulted in 18 construct progressions across five early learning and
development domains, ranging from early developing constructs such as fine motor Grip &
Manipulation to more complex cognitive skills such as Problem Solving (Exhibit 1.1). The
constructs included traditional academic skills like Object Counting and Writing and more
difficult to measure skills like Perseverance. Progressions varied in their recommended grade
range (kindergarten, K–1, K–2, or K–3) and number of skill levels on the progression (e.g., 3 for
the Crossing Midline progression, 13 for Mathematical Patterns).
EAG K-3 Formative Assessment: Final Report September 2017 3
Exhibit 1.1. EAG K–3 assessment blueprint
Construct Field Test Grades No. of Skills in Progression Assessment Means1 Associated Standards2
Hand Dominance K 3 O,S ELS
Grip & Manipulation K-1 4 O,S ELS
Crossing Midline K-1 3 O,S,T ELS
Gross Motor Development K-3 9 O,S,T ELS
Emotion Expression K-3 7 O ELS
Emotion Regulation Strategies K-3 5 O ELS
Emotional Literacy K-3 9 O,S ELS
Book Orientation K 4 O,S ELS
Print Awareness K-1 7 O,S ELS, CCSS
Letter Naming K-1 7 O,S ELS, CCSS
Following Directions K-3 6 O,S ELS
Writing K-3 7 O,S ELS, CCSS
Reading Comprehension K-3 8 O,S ELS, CCSS
Vocabulary K-3 8 O,S ELS, CCSS
Perseverance K-3 8 O ELS
Object Counting K-2 10 O,S,T ELS, CCSS
Mathematical Patterns K-3 13 O,S ELS, CCSS
Problem Solving K-3 10 O,S ELS
1 Refers to how the construct was assessed: Observation, Situation, Task 2 ELS = Early Learning Standards; CCSS = Common Core State Standards
EAG K-3 Formative Assessment: Final Report September 2017 4
Initial ECD phases During the first year of the enhancement project, a study was conducted to determine the
alignment of the constructs North Carolina had developed with the consortium states’ early
learning and development standards (Scott-Little, Kagan, Sumrall, & Fox, 2014). Members of
the consortium also identified priority areas for the possible development of constructs to add to
those developed by North Carolina: algebraic thinking and operations, the scientific inquiry
process, and behavior regulation. Project resources were sufficient to develop one additional
construct; Mathematical Patterns was developed as a measure of algebraic thinking.
ECD codesign teams
The North Carolina assessment materials for 15 constructs were submitted to SRI for further
development and enhancement by ECD codesign teams. For each construct, a codesign team
was created of experts in early childhood assessment, ECD, and the respective domain.
Supporting each team were one or two of the North Carolina developers of the original
construct. An SRI technician documented each team’s assessment using the web-based
Principled Assessment Designs for Inquiry system. Some of the constructs were designed to be
assessed at K entry and others were designed for grades K–3. The enhancement process was
to extend other constructs upwards to grade 3 if appropriate.
National domain experts reviewed the construct progressions and suggested revisions. Drawing
on that review, the codesign teams refined the progressions, developed performance
descriptors and examples for each skill on the progression, and refined or developed any
situations and tasks needed to assess a particular construct. This process resulted in the
development of the following materials to support the assessment of each construct:
• A Construct Progression document explaining the background, rationale for assessing,
the overarching understandings, and the specific skills in each progression with
performance descriptors and examples of what each skill looks like in the classroom.
• An Assessment Means Form, essentially an administrator’s manual, covering what a
teacher needs to know to assess a particular construct, including how to observe for the
skills and more structured situations and tasks to pinpoint specific skills.
• Single and Multiple Child Documentation Forms that enable the teacher to record
evidence of where a child falls on each progression, with brief reminders of behaviors to
watch for that differentiate each skill on the progression.
More information on the results of the ECD codesign team work is in Appendix A.
EAG K-3 Formative Assessment: Final Report September 2017 5
Cognitive labs
The assessment materials the codesign teams developed were tested by 122 K–3 teachers in
five states (Arizona, Iowa, Maine, North Carolina, and Rhode Island). The teachers were trained
on the K–3 formative assessment process and then asked to assess three to five children with
the materials for up to four different constructs during spring and fall 2015.3 The teachers
provided construct-specific feedback on the assessment materials through online surveys and
follow-up interviews. Their feedback was used to further refine the progressions and
accompanying assessment materials for the 2015–16 pilot test. The key findings from the
cognitive labs and the changes made in response were as follows:
• Teachers found the social-emotional constructs most difficult to assess. Recommended
changes were made in the progressions for Emotion Expression, Emotion Regulation
Strategies, and Emotional Literacy.
• Teachers reported some confusion about the Reading Comprehension and Writing
progressions because they contrasted with how reading and writing are assessed in more
traditional assessments. Clarifications were made to the related assessment materials.
More information on the cognitive labs is in Appendix B.
Pilot test
The full assessment was piloted with approximately 80 K–3 teachers and 800 children in four
states (Arizona, Iowa, Maine, Rhode Island) during the 2015–16 school year. Each teacher
assessed a sample of 10 children on all 17 constructs. Mathematical Patterns was not included
because it was developed after the pilot.
Teachers’ use of the assessment was supported by a technology platform. The teachers
entered evidence (such as anecdotal notes, photographs, or videos) from their observations for
each construct and child in the sample. They then determined where on the progression each
piece of evidence fell (preliminary learning status) and, across multiple pieces of evidence over
multiple observations, where to place the child on the progression (status summary). All the
assessment data (evidence, learning statuses, status summaries) could be entered into the
platform for easy review, reporting, and analysis. This process was further supported by an
application that allowed teachers to upload directly to the platform evidence (their notes, photos,
and videos) from a tablet or smart phone to assist in selecting a status summary.
3 The North Carolina staff conducted cognitive labs for Problem Solving, the last construct North Carolina developed, in spring 2016.
EAG K-3 Formative Assessment: Final Report September 2017 6
Teachers provided feedback on the manageability and usability of the assessment to inform
instruction through online surveys, focus groups, and interviews. They appreciated the
assessment’s focus on the whole child and coverage of multiple domains of learning and
development. They also appreciated being able to store and retrieve the evidence they collected
in the technology platform. Teachers’ other feedback and the response to it were as follows:
• The assessment process was time consuming, and documenting and interpreting multiple
pieces of evidence to determine where a child fell on each progression was cumbersome.
Consequently, the field test training was modified to further demonstrate how formative
assessment observations can fit into daily instructional routines.
• The social-emotional constructs were particularly difficult to assess. Accordingly, the
Emotional Literacy progression was simplified, the assessment materials for Emotion
Expression and Emotional Regulation Strategies were clarified, and more guidance on how
to assess these constructs was added.
• The Vocabulary construct was challenging to assess. Thus, two progression skill levels that
were difficult to observe in the classroom were combined (using vocabulary in different
contexts) and the assessment materials clarified to emphasize that the construct is about
vocabulary concepts rather than vocabulary word lists.
More information on the pilot test is in Appendix C.
Field test design The field test was designed to answer two overarching research questions:
1. Does the K–3 Formative Assessment provide valid information about what children know
and can do?
2. What supports are required to allow the assessment to be implemented with fidelity and
produce valid information?
Both questions address validity in that teachers who cannot implement the assessment well are
not likely to be able to use it to produce valid information about what children know and can do.
The field test sample comprised 141 K–3 teachers and 1,312 children in four states: Arizona,
Iowa, Maine and Rhode Island (Exhibit 1.2).
EAG K-3 Formative Assessment: Final Report September 2017 7
Exhibit 1.2. Field test sample, by state
State Number of Teachers Number of Children
Arizona 43 399
Iowa 10 101
Maine 50 450
Rhode Island 38 362
Total 141 1,312
Grade K–3 teachers in the four pilot states were asked to assess 10 children in their classroom
on those constructs assigned to their respective grade levels (see Exhibit 1.1), once in the fall
and again in the winter. The sample of teachers and children and the number of constructs
assessed by grade level are shown in Exhibit 1.3.
Exhibit 1.3. Field test sample, by grade
Grade Number of Constructs Number of Teachers Number of Children
Kindergarten 18 89 850
Grade 1 16 22 198
Grade 2 12 15 126
Grade 3 11 15 138
Total 141 1,312
Because the field test states were generally most interested in using this assessment at
kindergarten entry, they were able to recruit more teachers at that grade level. Consequently,
the sample contains far more kindergarten teachers, and the findings reflect the kindergarten
experience more strongly than the other grade levels. In addition, the sample of children
assessed was 68 percent White (Exhibit 1.4), with relatively small numbers of other racial/ethnic
groups, which limited the ability to examine the research questions for these subgroups.
EAG K-3 Formative Assessment: Final Report September 2017 8
Exhibit 1.4. Demographics of children assessed in field test (Fall)
Subgroups Number Percent
Gender (female) 660 50.3
English learner 102 7.8
Has IEP 113 8.6
Race (non-White) 239 18.2
Asian/Pacific Islander 17 1.3
Black 54 4.1
Hispanic 126 9.6
White 894 68.1
Multi/other 43 3.3
Unknown 178 13.6
Total 1,312
As during the cognitive labs and pilot, teacher feedback in the field test was gathered through
online surveys, focus groups, and interviews. In addition to determine performance levels,
kindergarten teachers were surveyed on where they would expect children entering
kindergarten to fall on each of the progressions, based on their experience assessing the
constructs with kindergarten children in the fall. All survey and interview protocols are in
Appendix D. The child assessment data were collected through the technology platform that
supported teachers’ use of the assessment.
Report organization The report chapters present the findings regarding the two research questions addressed in the
field test. The first research question, Does the K–3 Formative Assessment provide valid
information about what children know and can do, was broken into subquestions that
represented validity claims about the assessment content (Chapter 2) and assessment process
(Chapter 3). The second research question, What supports are required to allow the
assessment to be implemented with fidelity and produce valid information, was addressed
through subquestions about implementation supports for teachers (Chapter 4) and broader
system-level supports (Chapter 5). Chapter 6 summarizes the findings and recommendations of
expert reviews from disability, English learner, and racial/cultural perspectives, and describes
EAG K-3 Formative Assessment: Final Report September 2017 9
the changes that were made to the assessment based on the reviews. Chapter 7 summarizes
the findings and presents recommendations for possible revisions and future development
research regarding the assessment materials and related training materials. Finally, the report
appendices provide more detail on all four phases of the ECD assessment enhancement
process.
EAG K-3 Formative Assessment: Final Report September 2017 10
Chapter 2: Validity of assessment content Validity was addressed through a series of research questions that represented validity claims about the assessment content. The questions concerned the sequencing of the skills that make up the progression, performance levels at kindergarten entry, sensitivity of the assessment to change over time, and usefulness of the assessment in informing instruction for children in grades K–3, including children with disabilities and English learners.
Are the skill levels of the construct progressions in the proper sequence? Each progression is intended to be a set of skills that increase in complexity or difficulty with
each step. The sequencing of the skills in the progression was researched and intentionally
addressed as part of the development process. Educators in North Carolina developed the
construct progressions with input from domain experts, curriculum specialists, developmental
psychologists, and teachers, under the guidance of Margaret Heritage, an expert in formative
assessment. Before consideration by the ECD codesign teams, each construct progression had
been reviewed by a nationally known expert in that area of early childhood development to
verify that the progression incorporated the appropriate skill levels in the proper sequence (see
table of domain experts in Appendix E). The experts also had recommended changes as
appropriate. During the codesign process, the order of the skills was reviewed and validated as
we developed performance descriptors and classroom examples for each skill level.
In general, the field test findings indicated that the skill levels for each progression were in the
proper sequence for each grade. Further research is needed to examine some skill levels on
some progressions to determine whether they are sufficiently different to have value for
informing instruction.
The field test provided three kinds of empirical evidence for the sequencing of the skills:
• Teacher survey results
• The distribution of children’s placements on each progression
• An analysis of the child assessment data that produced psychometric models which
examined the ordinality of the skills on each progression.
Teacher survey results
After teachers had used the assessment in the fall and again in the winter, they answered a
question as part of an online survey that asked, Did the order of the levels seem right for all of
the progressions you assessed (that is, the construct progressed from least difficult to most
difficult skill)?
EAG K-3 Formative Assessment: Final Report September 2017 11
Across all constructs, 87 percent of teachers reported that they believed the levels of the
progressions they assessed were in the correct order. This varied by construct, with 92 to 99
percent of the teachers reporting the skill levels were in the proper sequence for any one
individual progression. The construct with the largest percentage of teachers uncertain about
the order of the skill levels was Gross Motor at 8 percent. Additional information from both the
pilot test and the field test suggested that teachers questioned the relative placements of
galloping (Skills F and G) and skipping (Skill I) for the Gross Motor construct.
Distribution of children’s placements on progressions
If the skills on a progression are in the proper sequence from less to more difficult, certain
patterns should be seen in the distributions of the children’s placements on the progression. For
constructs that children would be expected to master in kindergarten, the mode of the
distribution should be skewed to the highest skill level on the progression. For constructs that
are designed to extend well into grade 3, the distribution should extend across the skill levels.
For all of the distributions, we would expect to see relatively smooth curves indicating no
uneven patterns of selection for any particular level of the progression. We also would expect
the distribution of placements on the progression to shift from lower to higher between fall and
winter and as the grade levels increase.
We observed four types of distributions in progression placements.
1. Unimodal (i.e., one peak) with smooth tails on either side—Three progressions: Emotional
Literacy, Emotion Regulation Strategies, Writing
2. Unimodal, smooth tails, skewed to the higher end of the progression—Six progressions:
Book Orientation, Crossing Midline, Following Directions, Grip & Manipulation, Hand
Dominance, Letter Naming
3. Unimodal, smooth tails, skewed to the lower end of the progression—One progression:
Vocabulary
4. Unimodal with uneven tail(s)—Eight progressions: Emotion Expression, Gross Motor, Math
Patterns, Object Counting, Print Awareness, Perseverance, Problem Solving, Reading
Comprehension
Examples of each type of distribution, taken from the fall administration with kindergarten
children, are presented in Exhibits 2.1–2.4. The distributions of all fall kindergarten placements
(the status summary selected by the teacher) on each progression are in Appendix D, as are
the complete assessment results for both fall and winter administrations in grades K–3.
EAG K-3 Formative Assessment: Final Report September 2017 12
Exhibit 2.1. Unimodal distribution with smooth tails
Exhibit 2.2. Unimodal, smooth tails, skewed to the higher end of the progression
1%5%
9%
18%
37%
12%
6% 6% 5%1%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I
Frequency Distribution of Skill Levels on Emotional Literacy - Grade K Fall (n = 803)
0%5%
22%
30%
42%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D
Frequency Distribution of Skill Levels on Grip & Manipulation - Grade K Fall (n = 833)
EAG K-3 Formative Assessment: Final Report September 2017 13
Exhibit 2.3. Unimodal, smooth tails, skewed to the lower end of the progression
Exhibit 2.4. Unimodal, uneven tails
Although any of the four distribution types are possible when skills are in the correct order, the
uneven tails shown in Exhibit 2.4 for Object Counting suggest that some skill levels on the eight
progressions with this pattern may be very similar in difficulty making it hard for teachers to
differentiate them in these progressions. We conducted more rigorous psychometric analyses to
examine this issue further.
2%
13%
29%34%
11%8%
2% 2% 1%0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H
Frequency Distribution of Skill Levels on Vocabulary - Grade K - Fall (n = 796)
0%3% 4%
6% 4% 5%
29%
21%
10%6%
12%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I J
Frequency Distribution of Skill Levels on Object Counting - Grade K Fall (n = 826)
EAG K-3 Formative Assessment: Final Report September 2017 14
Psychometric models
To further validate whether the skill levels in the 18 construct progressions were in the proper
sequence, the research team applied classical and modern test theory methods: exploratory
factor analysis (EFA) and Rasch modeling. EFA was applied to examine unidimensionality, that
is, whether the constructs when taken together were measuring a single, unidimensional latent
trait, such as school or learning readiness. When applied to the K–3 fall data, EFA indicated that
up to three latent factors may have been present among the constructs. Although fit statistics
were within the acceptable range for one-, two, and three-factor solutions, the one-factor
solution is considered more parsimonious in service of substantiating the claim that all
progressions were in the correct order. The Cronbach’s alpha for one overall “school readiness”
construct was .84, exceeding the social science standard of .80; this justified generating a total
score for modeling purposes. A total score was calculated so we could examine whether
children placed at a higher skill level for any construct progression (e.g., Skill C on Object
Counting) had higher mean total scores for “school readiness” than their peers placed at a lower
skill level (e.g., Skill A on Object Counting). Performance across constructs was calculated by
converting the skill levels on each progression to a numerical scale (Emerging = 1, A = 2, B = 3,
etc.) and summing across the progressions. for a total score was calculated solely for the
purposes of statistical modeling since a total score was not part of the design nor intended use
of the assessment.
A pattern of increasingly higher scores across the progression would support the claim that the
levels on the progressions are in the correct order. We examined the total score means of
children placed at each skill level on the progression for the 754 kindergarten children who were
assessed in the fall on all 18 progressions. Across the constructs we observed the same
general pattern: The mean total scores of children increased at each subsequent level on the
progression. This can be seen, for example, in the distribution of total mean scores for Emotion
Regulation Strategies in Exhibit 2.5.
EAG K-3 Formative Assessment: Final Report September 2017 15
Exhibit 2.5. Example of Increasing mean “School Readiness” scores for each level of the progression: Emotion Regulation Strategies (Fall Kindergarten)
There were, however, some exceptions to this pattern, mostly typically when the sample was
not large enough at any given skill on the progression to establish a valid mean (e.g., fewer than
10 at any one skill level) (Linacare, 2004; Wright & Linacare, 1992). Small sample numbers
were most often concentrated at the lower or higher ends of some progression where
kindergarten children were unlikely to be placed. We cannot reach any conclusions about the
sequencing for these skill levels on these progressions because the small number of children
means the findings are unreliable for these skill levels. We also observed essentially equivalent
mean total scores for some adjacent skills with larger numbers of children, suggesting that for
these progressions the skills may be very close together or difficult for teachers to differentiate,
as is evident in the distribution of total mean scores for Following Directions (Exhibit 2.6). For
this progression, the mean score for Level B is higher than for Level C which is not the expected
pattern. Note that we reach any conclusions about the “Emerging” level because it is based on
only 5 children.
7379
88 92102 105
0
20
40
60
80
100
120
Emerging(n = 19)
A(n = 112)
B(n = 139)
C(n = 209)
D(n = 195)
E(n = 80)
Mean Total Score School Readiness
EAG K-3 Formative Assessment: Final Report September 2017 16
Exhibit 2.6. Example of skill levels too close together: Following Directions
The distributions of total mean scores for each progression are in Appendix D. These
distributions suggest the constructs fall into three groups.
1. Properly sequenced skills at all levels of the progression (as shown by increasing total score
means with each skill level)—Ten progressions: Emotion Regulation Strategies, Emotional
Literacy, Grip & Manipulation, Hand Dominance, Crossing Midline, Letter Naming,
Perseverance, Print Awareness, Reading Comprehension, Writing
2. Unsure about the sequence of a few skills because of insufficient observations to validate
the sequence—Two progressions: Book Orientation, Gross Motor
3. Some skills may be too close to differentiate a child’s level (essentially equivalent total score
means at neighboring skill levels)—Seven progressions: Emotion Expression, Following
Directions, Gross Motor, Mathematical Patterns, Object Counting, Problem Solving,
Vocabulary
To further explore the possible dis-ordinality of progressions in the last group, a partial credit
Rasch model was applied to the fall K–3 assessment data to examine with greater rigor whether
steps in each progression were truly ordinal and sufficiently discriminated skill levels. Rasch
modeling is a psychometric approach for analyzing categorical data, such as answers to
questions on a reading assessment or, in this case, skill levels on a learning progression, as a
function of (a) the child’s abilities and (b) the skill difficulty.
71 7081 78
8792
103
0
20
40
60
80
100
120
Emerging(n = 5)
A(n = 36)
B(n = 35)
C(n = 83)
D(n = 125)
E(n = 126)
F(n = 344)
Mean Total Score School Readiness
EAG K-3 Formative Assessment: Final Report September 2017 17
We used an item response theory (IRT) framework to examine the ordering of the skill levels for
each of the 18 construct progressions based on the one-factor unidimensional solution
previously described. We applied the partial credit model (PCM) as a logistic measurement
model for polytomous items or progressions with an ordinal sequence (Embretson & Reise,
2000; Masters, 1982).
Factor scores were generated based on the PCM, using the full sample of 1312 K-3 children
assessed in the fall. In general, the IRT factor scores increased with higher skill levels on each
progression, lending support to the claim that the skills are in the correct order. We did,
however, encounter the same issues described previously when examining the total mean
scores for K children. There were insufficient numbers of observations (less than 10 children) at
the lowest skill level (Emerging) for several progressions (Book Orientation, Crossing Midline,
Emotion Expression, Following Directions Letter Naming, Object Counting, and Print
Awareness) to verify the ordinality of the Emerging level (Linacre, 2004; Wright & Linacre,
1992). The small number of observations was also a problem for verifying the ordinality of the
first three skills of the Gross Motor progression (Emerging, Skills A and B). Of greater concern
for the assessment are the two cases when the factor scores were equal or reversed at two
adjacent skill levels on the progression, suggesting that these skills are too close together to be
considered sequenced for instructional utility: Object Counting - Skills D and E, and
Mathematical Patterns - Skills B and C. The IRT factor scores for each construct progression
are presented in Appendix D, along with the previously discussed total mean scores.
Do the performance levels accurately reflect what is expected at entry to kindergarten? The K–3 Formative Assessment was designed to be used at kindergarten entry to inform
teachers’ instruction from kindergarten through grade 3. Kindergarten teachers often asked
where they should expect their students to fall on these progressions at the beginning of the
school year. Identifying where children are expected to be provides teachers with information
about which children are and are not on track. It also can provide administrators and
policymakers with a snapshot of children’s skills at kindergarten entry vis-à-vis expectations.
Parents, too, are interested in knowing where their child is relative to expected performance. For
the field test, we designed and implemented a method for establishing expected performance
levels for entry to kindergarten.
EAG K-3 Formative Assessment: Final Report September 2017 18
The research team used a mixed-methods approach and data from three different perspectives
to set performance levels at kindergarten entry:
• Teacher survey data regarding performance expectations on each progression
• Progression placement data for children in the fall of kindergarten
• National domain experts’ review of these data in light of their knowledge of kindergarten
entry skills for the construct progression they helped refine.
The performance levels were thus based on years of teacher experience with multiple
kindergarten classrooms and domain experts’ knowledge of current developmental literature on
each construct. We were able to establish performance levels for all the constructs, indicating
that the content of the assessment is appropriate for kindergarten children. We used an
accepted process for establishing performance levels based on teacher experience and domain
expertise that produced credible performance levels consistent with the actual performance of
the kindergarten children in the field test. We therefore conclude that the performance levels are
an accurate reflection of what is expected of children at kindergarten entry.
The research team used a standard-setting method (American Educational Research
Association, American Psychological Association, & National Council on Measurement in
Education, 2014; Bejar, 2008; Hambleton, 2000; Livingston & Zieky, 1982) that had four steps:
1. Identify who will determine performance levels at kindergarten (kindergarten teachers and
domain experts).
2. Define the performance descriptors teachers and domain experts will use to choose levels
on the progressions.
3. Collect and analyze judgments from teachers and domain experts.
4. Determine placement of performance levels.
See Appendix E for a list of the experts who reviewed the performance level data for each
construct.
Teacher surveys and domain expert questionnaires
After completing the collection of the field test assessment data in the fall and again in the
winter, all 89 kindergarten field test teachers were asked to complete a survey on which they
were to place a typical student on each construct progression at kindergarten entry based on
their experience assessing the constructs and teaching kindergarten. The teachers were asked
to identify two levels for each progression:
EAG K-3 Formative Assessment: Final Report September 2017 19
1. Select the level on the progression that represents the skill level of an average or typical
incoming kindergarten child in the first week of school; and
2. Mark the highest level on the progression where a child is so far below the skill level of a
typical kindergarten child that he or she may experience significant difficulties in the
classroom in this construct.
For the national experts, we created performance descriptors to help them determine which skill
levels on the progression would be included in different performance levels at the beginning of
kindergarten. Each descriptor was conceptualized based on instructional purposes. The
performance descriptors were the following:
• Performance level 1 (Developing): Students in this category have acquired some, but not all,
of the knowledge skills and abilities necessary to meet expectations or learning goals at
kindergarten entry. More support may be needed to help students achieve their learning
goals during the first few weeks of kindergarten.
• Performance level 2 (Demonstrating): Students in this category have acquired the
knowledge skills and abilities necessary to meet expectations or learning goals at
kindergarten entry.
Teacher survey responses were combined with frequency distributions for the fall 2016
kindergarten student placements on each progression in charts for easy comparison. Domain
experts were asked to review the charts, respond questions about where children’s knowledge
and skills fall on the progression at the beginning of kindergarten, and provide a written rationale
for their recommendations. The teacher survey and expert questionnaire are in Appendix D.
Teacher Results
Ninety-one percent of kindergarten field test teachers (n = 81) responded to the performance
level survey and 82 percent (n = 71) provided sufficient data for the analysis.
Teacher responses to the first instruction for each construct (select the level on the progression
that represents the skill level of an average or typical incoming kindergarten child in the first
week of school), varied by the perceived difficulty of the construct. For constructs that the
assessment data indicated were mastered by most children in kindergarten (Book Orientation
and Hand Dominance), teachers placed typical learners near the top of the progression. For
example, 49 percent of teachers responded that typical children would most likely be at skill
level C (opens book from front cover) on the Book Orientation progression, with 26 percent
EAG K-3 Formative Assessment: Final Report September 2017 20
responding that typical children would enter kindergarten with skills at the highest level of the
progression, level D (turns pages in order one at a time). A similar pattern emerged for Hand
Dominance and Crossing Midline, with teachers responding that typical children would exhibit
skills at the higher end of the progression.
For progressions that were shown to extend from kindergarten through grade 3, teachers’
responses followed expected patterns and were clustered toward the lower end of each
progression. For instance, when asked at which level they would expect a typical kindergarten
student to be placed on for Writing, 92 percent of teachers responded with skill level A or B.
When asked the same question for Mathematical Patterns, 69 percent responded that a typical
incoming kindergarten student would exhibit skills at level A or lower, and 90 percent responded
with level C or lower.
Teachers’ responses to the second instruction for each construct (select the level where a child
is so far below the skill level of a typical kindergartener that he or she may experience
significant difficulties in the classroom) followed expected patterns, with teachers choosing skills
an average of 1.9 (SD = .56) levels lower on each progression. Exhibit 2.7 illustrates teacher
responses to the first and second instruction for Book Orientation. Charts in Appendix D provide
these results for each progression.
Exhibit 2.7. Teacher survey responses for Book Orientation: Percentage of teachers indicating each level
0%3%
21%
50%
26%
51%
37%
10%
1% 0%0%
10%
20%
30%
40%
50%
60%
Below A A B C DBook Orientation (n = 68 Teachers)
Typical Far Below
EAG K-3 Formative Assessment: Final Report September 2017 21
Teachers’ responses were then combined with kindergarten assessment data collected in fall
2016 to compare where teachers expected children’s knowledge and skills at kindergarten entry
and where children’s knowledge and skills were during the fall data collection period. Because
the fall data collection for the field test occurred between 2 and 12 weeks after school began,
we expected that the teachers’ response to the performance setting for kindergarten entry would
be lower than actual student performance, which was the pattern in the data. Exhibit 2.8
illustrates this pattern for Book Orientation.
Exhibit 2.8. Teacher survey responses and child assessment results for Book Orientation: Percentage of teachers indicating each level
Domain expert review
Domain experts largely agreed with where the majority of teachers placed expectations at
kindergarten entry. On 14 constructs, overlap was apparent between where teachers reported
they would place an average incoming kindergarten student and the kindergarten entry
performance level experts recommended. For the other four constructs, experts noted concerns
that teachers were underestimating skills at kindergarten entry. Each expert provided a rationale
for the placement decision that included references to literature on kindergarten entry skills
when available. For these four constructs, we based the performance levels on those the
experts recommended. Exhibit 2.9 illustrates the final performance levels compared with
teacher responses for Book Orientation. Exhibit 2.10 presents the performance levels for all 18
constructs, along with the percentage of kindergarten teachers who identified each skill level as
0% 3%
21%
50%
26%
51%
37%
10%1% 0%1% 0% 3%
11%
86%
0%
20%
40%
60%
80%
100%
Below A A B C DBook Orientation
Teacher survey:Typical Teacher survey: Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 22
“typical” for that construct. Based on the performance levels, we calculated the percentage of
children who would have been identified as “developing” or “demonstrating” on each construct
based on their fall progression placements (Appendix D).
Exhibit 2.9. Teacher responses, fall K data and performance levels for Book Orientation: Percentage of teachers indicating each level
0% 3%
21%
50%
26%
51%
37%
10%1% 0%1% 0% 3%
11%
86%
0%
20%
40%
60%
80%
100%
Below A A B C D
Teacher survey:Typical Teacher survey: Far Below Child Assessment
Developing Demonstrating
EAG K-3 Formative Assessment: Final Report September 2017 23
Exhibit 2.10. Teacher responses and resulting performance levels at kindergarten entry
Book Orientation 3% 21% 50% 26%
Crossing Midline 1% 12% 64% 23%
Emotional Expression 1% 38% 20% 25% 10% 4% 1%
Emotion Regulation Strategies
51% 35% 14%
Emotional Literacy 26% 16% 38% 16% 1% 3%
Following Directions 17% 33% 32% 14% 1% 1%
Grip & Manipulation 28% 61% 10% 1%
Gross Motor 4% 7% 10% 22% 19% 20% 12% 4% 1%
Hand Dominance 10% 70% 20%
Letter Naming 7% 20% 3% 26% 40% 3% 1%
Math Patterns 4% 65% 7% 13% 6% 4%
Object Counting 25% 41% 13% 4% 6% 9% 1% 1%
Print Awareness 23% 40% 20% 14% 1% 1%
Perseverance 1% 33% 35% 17% 12% 1%
Problem Solving 1% 35% 42% 16% 6%
Reading Comprehension 4% 19% 46% 29% 1%
Vocabulary 1% 45% 48% 4% 1%
Writing 14% 78% 6% 1%
SKILL LEVELS: Below A A B C D E F G H I J K L M
Note. Teacher responses are percentage of teachers who indicated that skill level as typical for a child entering kindergarten.
Developing Demonstrating
EAG K-3 Formative Assessment: Final Report September 2017 24
Is the assessment sensitive to children’s growth and development over time? The field test findings showed changes in children’s progression placement over time and
indicated that most of the constructs in the assessment were sensitive to children’s growth and
development within a school year from fall to winter. For a few constructs, many children were
placed at the highest levels of the progression in the fall, so there was little opportunity for
growth beyond kindergarten entry. Although there were smaller numbers of children assessed in
grades 1–3, their placement was consistent with the hypothesis that children in the upper
grades would be placed at higher progression levels than children in the lower grades on those
progressions with levels that extended beyond kindergarten.
To answer this question, we examined the field test data from three sources:
• Teacher survey results
• Student placement on the progressions from fall to winter
• Student placement on the progressions across grade levels.
Teacher survey results
In an online survey administered after they had used the assessment for the second time in the
winter, teachers were asked:
• For each of the constructs you administered, indicate how well the progression was able to
capture children’s progress and development over time: very well, somewhat well, not well
or not at all well.
Across all constructs, 84 to 99 percent of teachers reported that the constructs they assessed
captured children's progress and development over time (either “very well” or “somewhat well”).
Teachers were less confident that the social-emotional constructs and Vocabulary captured
children’s progress over time. For these constructs, fewer teachers selected “very well” or
“somewhat well” than for the other progressions, and far fewer selected “very well”
(Exhibit 2.11).
EAG K-3 Formative Assessment: Final Report September 2017 25
Exhibit 2.11. Teacher ratings of how well construct progressions captured children’s growth
Fall to winter change in placements on the progressions
Teachers’ placement of children on the progressions provided further support for the validity
claim that the construct progressions are sensitive to student growth over time. This was evident
in the
• Shifts to higher levels in the distribution of progression placements from fall to winter on
each construct
• Higher placements of the same children tested in the fall and again in the winter for each
construct and grade level.
62%
40%
60%
50%
63%
52%
64%
47%
69%
62%
62%
64%
62%
30%
28%
29%
59%
67%
30%
44%
30%
38%
33%
39%
28%
39%
24%
37%
30%
33%
34%
55%
56%
55%
37%
30%
8%
16%
10%
12%
4%
9%
8%
14%
7%
9%
3%
4%
15%
16%
16%
4%
2%
0% 20% 40% 60% 80% 100%
Writing
Vocabulary
Reading Comprehension
Problem Solving
Print Awareness
Perseverance
Object Counting
Mathematical Patterns
Letter Naming
Hand Dominance
Gross Motor
Grip & Manipulation
Following Directions
Emotional Literacy
Emotion Regulation
Emotion Expression
Crossing-Midline
Book Orientation
Very well Somewhat well Not well
EAG K-3 Formative Assessment: Final Report September 2017 26
These patterns also provided evidence for the claim that the levels are sequenced correctly. If
the sequencing had not been correct, we would not see movement across the progression with
time.
Comparing the fall and winter assessment results for each construct across grade levels, we
found that the distribution of placements shifted from lower placements in the fall to higher
placements in the winter for all the progressions. This pattern can be seen, for example, in the
distribution of fall and winter placements for Grip & Manipulation for kindergarteners
(Exhibit 2.12).
Exhibit 2.12. Example comparing fall and winter progression placements: Grip & Manipulation (percentage of children)
The strongest evidence to address this question came from comparing individual change for
those children who were assessed in the fall and again in the winter. Of the 1,312 children
assessed in the fall, 1,278 also were assessed in the winter. We examined the differences
between the fall and winter means among children at each grade level by converting the
progression placements to numerical values (where Emerging = 1, A = 2, B = 3, C = 4, etc.) and
applying t tests for differences in the means. Winter means were significantly higher than the fall
means for all constructs and grade levels assessed with two exceptions: Perseverance in
grade 2 and Emotion Regulation Strategies in grade 3. The results comparing fall and winter
placement (tables in Appendix D) supported the claim that the 18 constructs are sensitive to
growth.
0%5%
22%30%
42%
0% 1% 5%12%
82%
0%
20%
40%
60%
80%
100%
Emerging A B C D
Grip and Manipulation - Grade K
Fall (n = 850) Winter (n = 785)
EAG K-3 Formative Assessment: Final Report September 2017 27
Across grade levels, children were generally placed at the same or higher levels on the
progressions in the winter. On some constructs children had limited opportunity to show growth
within grade level because many were already near the top at fall within that grade level.
Comparing the placements of children tested in the fall of kindergarten with their placements in
the winter, we found that children most often were at either a higher skill level in the winter or at
the same skill level if a child had already reached the highest level on the progression in the fall.
On three constructs, the opportunity to show growth in kindergarten was limited because most
children had reached the highest level on the progression in the fall: Book Orientation
(86 percent), Hand Dominance (74 percent), and Crossing Midline (71 percent). On four
constructs, at least 70 percent of children had higher placements on the progression in the
winter than the fall: Mathematical Patterns (76 percent), Object Counting (73 percent), Reading
Comprehension (73 percent), and Writing (70 percent) (Exhibit 2.13).
Exhibit 2.13. Examples of fall to winter progression placements of children in kindergarten (percentage of children)
13
19
19
70
73
73
76
86
71
74
5
11
6
4
26
12
12
19
4
2
4
11
4
5
0 20 40 60 80 100
Book Orientation (n = 763)
Crossing Midline (n = 773)
Hand Dominance (n = 777)
Writing (n = 770)
Reading Comprehension (n = 731)
Object Counting (n = 773)
Math Patterns (n = 733)
Higher Same, at the Max Same Lower
Constructs showing most opportunity for growth
Constructs showing least opportunity for growth
EAG K-3 Formative Assessment: Final Report September 2017 28
To determine the recommended grade levels for a construct, we identified the grade levels at
which a substantial majority of the children (70 percent) were placed at the highest level on the
progression in the fall. These constructs were classified as showing opportunity for growth at the
grade levels below where this occurred. The percentage of children at the maximum level in the
fall along with the grade levels with opportunity for growth for each construct are summarized in
Exhibit 2.14 with more detail provided in Appendix D.
Exhibit 2.14. Percentage of children at maximum level and grade levels with opportunity for growth for each progression
Field Test Percentage of Children at
Maximum Level in Fall Opportunity for Growth
Book Orientation (A-D) K 86 na na na Below K
Hand Dominance (A-C) K 74 na na na Below K
Crossing Midline (A-C) K-1 71 71 na na Below K
Grip & Manipulation (A-D) K-1 42 59 na na K-1
Letter Naming (A-G) K-1 31 65 na na K-1
Print Awareness (A-G) K-1 41 94 na na K
Object Counting (A-J) K-2 11 43 77 na K-1
Emotion Express (A-G) K-3 6 16 26 26 K-3
Emotion Regulation Strategies (A-E) K-3 6 19 33 42 K-3
Emotional Literacy (A-I) K-3 1 2 23 21 K-3
Following Directions (A-F) K-3 42 45 61 44 K-3
Gross Motor (A-I) K-3 42 43 83 84 K-1
Math Patterns (A-M) K-3 0 0 0 8 K-3
Perseverance (A-H) K-3 7 12 7 13 K-3
Problem Solving (A-J) K-3 0 1 10 1 K-3
Reading Comp (A-H) K-3 5 21 30 33 K-3
Vocabulary (A-H) K-3 0 4 10 0 K-3
Writing (A-G) K-3 0 0 2 8 K-3
na = not assessed
EAG K-3 Formative Assessment: Final Report September 2017 29
Six progressions did not show opportunities to measure student growth through grade 3:
• For three progressions—Book Orientation, Hand Dominance, and Crossing Midline—more
than 70 percent of the children tested in the fall of kindergarten were placed at the highest
skill level on the progression, so these constructs have limited instructional utility for most
children beyond K entry.
• Print Awareness was field-tested in grades K and 1, but it may be most appropriate for
informing instruction in kindergarten only, with 94 percent of children already placed at the
highest/maximum level on the progression in the fall of grade 1.
• Object Counting was field-tested in grades K–2, but it may be most useful for informing
instruction in grades K–1, with 77 percent of children already placed at the highest level on
the progression in the fall of grade 2.
• Gross Motor was field-tested in grades K–3 but appears to be providing useful information
only in grades K–1, with 83 and 84 percent of children already placed at the highest level on
the progression in the fall of grades 2 and 3, respectively.
We further examined the extent to which the progressions provided opportunities for growth by
looking only at those children who were not at the highest level of the progression in the fall
(Exhibit 2.15). Across nearly all progressions at all grade levels, most of these children were
placed at a higher level in the winter.
Placement on the progressions across grade levels
Another way to examine growth is by looking at the placements on the progressions across
grade levels. For constructs with opportunities for growth into the next grade level, we observed
a shift in the distributions from one grade level to the next. The kindergarten to grade 1 shift
pattern can be seen, for example, by comparing the distributions of kindergarten and grade 1
fall placements for Following Directions, with more children at the upper end of the progression
in grade 1 than in kindergarten (Exhibit 2.16).
EAG K-3 Formative Assessment: Final Report September 2017 30
Exhibit 2.15. Percent of children not at highest level in fall who were placed at a higher level in winter for each progression
Field Test Percentage of Progression Placements Higher in Winter
(for Children Not at Maximum Level in Fall)
Book Orientation (A-D) K 91 na na na
Hand Dominance (A-C) K 75 na na na
Crossing Midline (A-C) K-1 65 57 na na
Grip & Manipulation (A-D) K-1 85 59 na na
Letter Naming (A-G) K-1 82 69 na na
Print Awareness (A-G) K-1 93 82* na na
Object Counting (A-J) K-2 82 75 77* na
Emotion Express (A-G) K-3 66 70 45 70
Emotion Regulation Strategies (A-E) K-3 60 64 48 78
Emotional Literacy (A-I) K-3 64 70 64 46
Following Directions (A-F) K-3 66 46 51 77
Gross Motor (A-I) K-3 70 77* 74* 75*
Math Patterns (A-M) K-3 76 74 43 71
Perseverance (A-H) K-3 60 73 50 73
Problem Solving (A-J) K-3 66 76 66 82
Reading Comp (A-H) K-3 76 47 74 61
Vocabulary (A-H) K-3 68 57 67 50
Writing (A-G) K-3 70 60 42 46
na = not assessed; *n < 30 children
EAG K-3 Formative Assessment: Final Report September 2017 31
Exhibit 2.16. Example comparing kindergarten and grade 1 progression placements: Percentage of children placed on each level of the progression for Following Directions
Comparing the distribution and means for constructs assessed at multiple grades, we found the
expected variation in distributions by grade level. For the constructs for which opportunity for
growth extended into the next grade level, the distributions and the means moved to higher
levels on the progression as the grade level increased. Some exceptions from grade 2 to
grade 3 were noted where sample numbers were smaller (126 for grade 2 and 138 for grade 3).
These differences between grade level mean placements (kindergarten to grade 1, grade 1 to
grade 2, grade 2 to grade 3) were statistically significant (p < .05). The summary of results for
grades K–3 placements is in Appendix D.
Although higher mean scores at each successive grade level were found for most constructs, a
few did not show the expected pattern between some grade: Crossing Midline (between K
and 1), Grip & Manipulation (K–1), Emotion Regulation Strategies (grades 1–2), Following
Directions (K–1, grades 2–3), Perseverance (grades 2–3), Problem Solving (grades 2–3), and
Vocabulary (grades 2–3).
Does the assessment provide information that is useful for instruction for grades K–3? We asked teachers how they used information from the assessment in the surveys at the end of
the fall and winter administrations and in focus groups at the end of the field test. The majority of
teachers reported using the information for instruction and that it was useful, although they
1%4% 5%
11%16% 18%
45%
0%3%
6% 8%12% 13%
58%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E FFollowing Directions (Fall)
Grade K (n = 827) Grade 1 (n = 194)
EAG K-3 Formative Assessment: Final Report September 2017 32
found the assessment more useful for some conditions and constructs. A small number of
teachers felt the assessment did not provide new or relevant information outside what they were
already observing or assessing in the classroom.
After the fall administration, 86 percent of teachers reported using information from the K–3
Assessment to inform instruction. Over half the teachers (54 percent) used the information “a
great deal” or at least “somewhat.” The most helpful constructs (those that 75 percent or more
of teachers who assessed it indicated the construct was helpful) were Writing, Problem Solving,
Object Counting, Reading Comprehension, Print Awareness, and Following Directions
(Exhibit 2.17).
Exhibit 2.17. Teachers’ rating of constructs as “very” or “somewhat” helpful to instruction
When asked about the benefits of implementing the assessment, about half the teachers
(53 percent) reported that they developed a deeper understanding of the whole child and the
child’s developmental needs (Exhibit 2.18). Teachers frequently commented in surveys and
interviews that the information was especially useful in identifying struggling students and in
grouping students who needed additional support.
51%
60%
63%
64%
64%
65%
66%
68%
68%
70%
73%
73%
75%
75%
76%
77%
79%
81%
Gross Motor
Hand Dominance
Crossing Midline
Emotion Expression
Emotional Literacy
Emotion Regulation
Book Orientation
Grip & Manipulation
Mathematical Patterns
Vocabulary
Letter Naming
Perseverance
Following Directions
Print Awareness
Reading Comprehension
Object Counting
Problem-Solving
Writing
Percentage of Teachers (n = 132)
EAG K-3 Formative Assessment: Final Report September 2017 33
Exhibit 2.18. Teachers’ report of benefits of implementing the K–3 Formative Assessment
After the winter administration, 86 percent of the teachers reported that they used information
from the assessment to inform instruction, and 68 percent found the assessment a helpful
resource for informing instruction in their classrooms.
The survey questions and focus groups also explored what kinds of difficulties teachers
encountered in using the assessment to inform instruction. One was that the data from the
assessment were redundant with information teachers already had collected. Most teachers
(81 percent) reported that they were able to use information from other assessments to place
children on progressions in the K–3 assessment, which suggests they were indeed already
collecting some of the information included in the assessment. The K–3 Formative Assessment
is intended to complement other assessments, and we encouraged teachers to use data from
other assessments in placing children on the progressions. Still, it is possible that there is too
much overlap among assessments for some constructs for teachers to appreciate the value of
these progressions in informing their instruction. Other impediments to instructional utility the
teachers reported were that the assessment was not aligned well enough with state standards
or district expectations, that they did not find it necessary to formally document what they
already knew about their students, and that the information from the assessment was not
relevant later in the school year.
23%
27%
27%
29%
30%
33%
52%
53%
No real benefits
Being better prepared to plan instruction that isdevelopmentally appropriate
Increased knowledge of how technology can be used fordata collection and for guiding instruction
Being better equipped to respond to individual studentneeds
Having data all in one place to review for instructionalplanning
Having readily available data to share with support staff,administrators, and families
Deeper understanding of individual child’s skills & needs
Better understanding of the whole child
Percentage of Teachers (n = 132)
EAG K-3 Formative Assessment: Final Report September 2017 34
Does the assessment provide information useful for instruction for different subgroups, such as children with disabilities and English learners? Most of the teachers who used the assessment with either children with disabilities or English
learners reported that they were able to assess children in these two subgroups and that the
assessment provided information useful for their instruction. Examining the progression
placements for these two subgroups, we found differences for children with disabilities for most
of the constructs but few placement differences for English learners compared with other
children in the class.
Children with disabilities
Children with disabilities comprised 8.6 percent of the field test sample overall (n = 113).
Teachers generally placed children with individualized education programs (IEPs) lower on each
construct progression than children without IEPs in both fall and winter administrations and
across grade levels. We examined the statistical significance of these subgroup differences in
progression placements in kindergarten, where 68 children had IEPs. Differences were
significant (most at p < .0001) for all constructs except Hand Dominance, Book Orientation
(winter only), and Mathematical Patterns (fall only), with children with IEPs placed lower on the
progression than children without IEPs. These results are presented in Appendix D.
Interpreting the differences across constructs is difficult because we do not know the
composition of the sample of children with IEPs, the nature of their disabilities, or whether
teachers made the accommodations indicated in their IEPs in implementing the assessment.
For two of the three constructs that did not consistently show group differences, Hand
Dominance and Book Orientation, many children were placed at the highest level of the
progression in the fall for Hand Dominance and by the winter for Book Orientation. For
Mathematical Patterns, children with IEPs were placed lower on the progression in the fall than
other children, but the mean difference did not quite reach significance (p = .06).
Slightly more than half (55 percent, n = 73) of the teachers implemented the assessment with a
child with an IEP. These teachers assessed on average two to three children with disabilities.
This was the same as the average number of children with IEPs (two to three) enrolled in
classes of 19 to 20 children in the field test. A majority of these teachers (57 percent) reported
that the assessment was no more difficult to implement with children with disabilities, with 32
percent reporting that it was a little more difficult. More important, 81 percent of these teachers
thought that the assessment provided information that was somewhat or very useful for
informing that child’s instruction.
EAG K-3 Formative Assessment: Final Report September 2017 35
English learners
English learners (ELs) comprised 7.8 percent of the field test sample overall (n = 102). There
were no consistent differences in how teachers placed ELs on each construct progression when
compared with non-ELs across both fall and winter administrations and across grade levels. We
examined the statistical significance of the differences between ELs and other children in
progression placements in kindergarten, where there were 73 EL students. Differences were
significant at p < .05 for six constructs. ELs were placed higher on the progression than non-ELs
for Book Orientation (fall), Crossing Midline (winter), Emotion Regulation Strategies (fall and
winter), Emotional Literacy (fall), and Hand Dominance (winter). ELs were placed lower on the
progression for Print Awareness (fall). These results are in Appendix D.
These differences across constructs are difficult to interpret without knowing more about the
composition of the EL sample, their experience with English, and the extent to which teachers
were assisted by a native language speaker in observing the skills that required communicating
or understanding the child’s first language. The absence of differences in progression
placements on constructs that rely most heavily on language (e.g., Letter Naming, Following
Directions, Reading Comprehension, Writing, and Vocabulary) suggests that the teachers found
ways to assess these constructs in the child’s native language or the children had sufficient
English skills so the teacher could assess them in English.
One quarter of the teachers (n = 33) implemented the assessment with English learners. These
teachers assessed on average three to four children who were learning English in school. On
average, there were two ELs enrolled in classes of 19–20 children in the field test. Most of these
teachers (76 percent) reported that the assessment was not more difficult to implement with
ELs, whereas 24 percent reported that it was a little more difficult. More important, 82 percent of
these teachers thought that the assessment provided information that was at least somewhat or
very useful for informing the instruction of ELs.
Do the assessment results vary for other subgroups (gender, race)? To further explore whether the K–3 Formative Assessment provides information useful for
instruction for all children, we examined the assessment results for gender and racial bias. We
would not expect to see any differences in where teachers placed children on these
progressions based on the race or gender of the child. However, since some studies of early
childhood development and early academic skills have shown gender and racial differences at
school entry and into the early elementary grades, we examined our data accordingly.
EAG K-3 Formative Assessment: Final Report September 2017 36
Gender
We compared the mean differences between boys and girls on the progressions and found no
gender differences in the fall of kindergarten for constructs in the cognitive domain (Object
Counting, Mathematical Patterns, Problem Solving) or for Reading Comprehension and the
physical constructs without a motor component (Hand Dominance and Crossing Midline).
Teachers placed girls higher than boys on constructs in the social-emotional domain and the
other constructs in the language and motor domains in the fall of kindergarten. Mean differences
on these progressions were statistically significant at p < .05 or higher. There were no
statistically significant differences at first grade, but differences reemerged in second and third
grade, favoring girls in Emotion Expression, Emotion Regulation Strategies, Following
Directions, Perseverance, Vocabulary, and Writing. Only one difference favored boys, Object
Counting in the fall of first grade. The findings for construct performance by gender are in
Appendix D. Note that the samples were considerably smaller at the higher grade levels and
therefore less likely to yield statistically significant differences. It is impossible to discern from
the field test information whether these differences reflect actual differences between boys and
girls or whether teachers bring a bias to their observation of some of the constructs.
Race/Ethnicity
As noted in Chapter 1, the sample of children assessed was 68 percent White, 18 percent non-
White, and 14 percent of unknown race/ethnicity. These data are based on information that
teachers provided in the technology platform for each child. The relatively small numbers of
non-White children limit our ability to examine racial/ethnic differences.
Despite the small sample size of non-White children, statistically significant differences were
found in the mean placements of White and non-White children for various constructs at
different grade levels and administrations (fall and winter). Some of the differences favored
White children, primarily in the language and communication domain: Hand Dominance (grade
K, fall), Letter Naming (grade 1), Object Counting (grades K and 2, fall), Perseverance (grade K,
fall), Print Awareness (grade K, fall), Reading Comprehension (grades K, 1, and 2, fall), and
Vocabulary (grade K, fall). Other differences favored non-White children, primarily in the social-
emotional and motor domains: Crossing Midline (grade 1, fall), Emotion Expression (grade 2,
winter), Emotion Regulation Strategies (grades 2 and 3, winter), Emotional Literacy (grade 2,
winter), Grip & Manipulation (grade 1, winter), and Gross Motor (grade 1, fall). Some constructs
showed differences favoring one group at one grade level and the other at a different grade
levels: Following Directions, Problem Solving, and Writing. The findings for construct
EAG K-3 Formative Assessment: Final Report September 2017 37
performance by race/ethnicity are in Appendix D. Given the relatively small number of non-
White children in the field test sample, especially in grades 1–3, we recommend further
research with a larger, more diverse sample before drawing any conclusions about how these
constructs are performing as measures for children from different racial/ethnic backgrounds.
EAG K-3 Formative Assessment: Final Report September 2017 38
Chapter 3: Implementation of assessment process A series of the research questions represented validity claims about important aspects of the assessment process: implementing the assessment with fidelity, including reliably coding status summaries; teacher confidence in their status summary ratings; and incorporating the assessment into classroom routines.
Can teachers learn to implement the assessment with fidelity? There is no fixed set of standardized practices for teachers to follow in implementing the K–3
Formative Assessment but rather a set of general guidelines for how they are to collect, record,
and interpret information about the children in their classroom. The field test results indicated
that the K–3 Formative Assessment could be implemented with fidelity, although some
constructs were more challenging to implement than others. The challenges in implementing
constructs were evident to a small extent in teachers’ selection of which constructs to assess
and to a larger extent in how faithfully teachers followed the assessment process. Variation was
apparent across constructs in teachers’ confidence in their status summary ratings and in the
reliability of their status summary coding. Nevertheless, the majority of teachers were able to
incorporate the assessment of most constructs into their classroom practice, and many teachers
were able to implement the assessment as intended. This is encouraging given the limited
training and experience the teachers had with the K–3 Formative Assessment before the field
test.
To produce valid data, an assessment must be implemented with fidelity to the assessment
process as designed. The K–3 Formative Assessment is based on teachers’ observations of
child behaviors in routine classroom activities and placement of each child on the skill levels on
each construct progression. Teachers were trained to collect multiple pieces of evidence of a
child’s skill level on each construct, determine the point on the progression (preliminary learning
status) indicated by each piece of evidence, and then interpret across these multiple pieces of
evidences to arrive at the child’s placement on the progression (status summary) at a given
time. Teachers were encouraged to gather at least two pieces of evidence for each child and
construct and enough evidence that they were confident in the child’s placement on the
progression. To determine whether teachers can learn to implement this assessment with
fidelity, we examined three indicators of their implementation:
• Implementation of the recommended assessment process
• Confidence in determining a child’s learning status
• Reliability in coding a child’s status summary.
EAG K-3 Formative Assessment: Final Report September 2017 39
Implementation of the recommended assessment process
The information entered into the technology platform suggests that teachers generally followed
the recommended assessment process with some variation by construct. Using the technology
platform for inferring how teachers implemented the assessment has limitations, however. A
lack of evidence entered into the technology platform could mean that a teacher did not collect
evidence, which would be a deviation from how the assessment was to be administered, or it
could mean that the teacher chose not to enter evidence, which is acceptable. Teachers
reported that collecting multiple pieces of evidence for each child and construct was time
consuming and that the process of entering the information onto the platform was cumbersome.
Some teachers also reported that they collected and used evidence to determine status
summary ratings but did not enter the evidence into the platform, further confirming the
importance of recognizing the limitations in reaching conclusions based on the contents of the
platform. Nevertheless, it is one of source of information about how teachers implemented the
assessment.
From the information in the platform, we learned the following about teachers’ adherence to
these steps in the assessment process:
• Assigned child a status summary (required). Teachers assigned a status summary for at
least one construct for 1,312 children in the fall and 1,229 children in the winter. Across
constructs, 93 to 98 percent of the children in the field test sample were assessed in the fall,
96 to 100 percent in the winter. Teachers assessed nearly all the constructs they were
asked to assess at their grade level with all the children they were asked to assess. The
constructs least frequently assessed were Emotional Literacy (93 percent of the children),
Emotional Regulation Strategies (94 percent), and Mathematical Patterns (94 percent), but
even these constructs were assessed for nearly all K–3 children in the sample.
• Documented multiple pieces of evidence (required but might not be in the platform).
Teachers did not consistently document more than one piece of evidence for determining a
child’s placement on a progression in the technology platform. Across constructs, 57 to 71
percent of the status summaries were based on multiple pieces of evidence in the fall, and
fewer (40 to 68 percent) were in the winter. Constructs least often documented with multiple
pieces of evidence were Emotional Literacy and Book Orientation (57 percent of the children
with status summaries) in the fall and Book Orientation (41 percent), Crossing Midline (40
percent), and Gross Motor (42 percent) in the winter.
EAG K-3 Formative Assessment: Final Report September 2017 40
• Assigned evidence a preliminary learning status (optional). Teachers were encouraged
to assign each piece of evidence a preliminary learning status to help them interpret multiple
pieces of evidence and determine the child’s status summary. Teachers followed this
optional step for 88 percent of all the evidence records entered on the platform. When
learning status was tabulated by construct and child, some variation was observed. Learning
status was assigned least often for Vocabulary (89%) and most often for Emotion
Regulation Strategies (99%), both during the winter administration.
These data are summarized in Exhibit 3.1. In both the fall and winter, teachers assigned a
status summary to nearly all constructs for all children who were assessed, and most of the
children were assigned the optional learning status as well.
EAG K-3 Formative Assessment: Final Report September 2017 41
Exhibit 3.1. Percentage of children with status summaries, multiple pieces of evidence, and learning statuses
Field Test
Percentage of Children Assigned a Status
Summary
Percentage of Children Whose
Status Summaries Based on Multiple Evidences
Percentage of Children Whose Evidence Included
a Learning Status
Book Orientation K 98 98 57 41 96 91
Crossing Midline K-1 97 100 62 40 97 94
Emotion Expression K-3 96 98 60 61 94 94
Emotion Regulation Strategies K-3 94 99 59 64 91 99
Emotional Literacy K-3 93 96 57 65 90 92
Following Directions K-3 98 100 66 58 94 92
Grip & Manipulation K-1 98 100 65 47 95 92
Gross Motor K-3 98 98 58 42 92 92
Hand Dominance K 98 100 71 49 98 91
Letter Naming K-1 98 100 64 45 95 90
Math Patterns K-3 94 98 62 64 95 91
Object Counting K-2 97 100 58 58 95 93
Perseverance K-3 96 99 66 62 94 92
Print Awareness K-1 98 98 61 54 92 91
Problem Solving K-3 95 98 60 63 90 92
Reading Comp K-3 95 100 64 59 95 91
Vocabulary K-3 95 98 60 62 92 89
Writing K-3 98 99 66 68 92 91
Teacher confidence in status summary
During the training, teachers often asked how much evidence was enough to support a child’s
placement on the progression (status summary). They understood that multiple pieces of
evidence were important but wanted to know exactly how many they should collect. There is no
exact number; rather, we suggested teachers collect enough evidence to be confident in the
child’s placement.
At the end of the winter assessment, we asked teachers to rate their confidence in having
accurately placed children on each of the progressions (Exhibit 3.2). Across constructs, 71 to 99
EAG K-3 Formative Assessment: Final Report September 2017 42
percent of teachers were at least somewhat or very confident in their ability to accurately place
children on each of the progressions they assessed. Their confidence varied by construct, with
teachers least confident in the accuracy of their placements on the three social-emotional
progressions: Emotional Literacy (75 percent), Emotion Regulation Strategies (71 percent), and
Emotion Expression (74 percent).
Exhibit 3.2. Percentage of teachers confident in progression placement (n = 130)
Teacher reliability in coding status summaries
We examined the evidence teachers provided in the platform from the fall data collection to
address two questions:
• What are the qualities of the evidence teachers entered into the platform (i.e., to what extent
is the evidence construct relevant, progression skill specific, and child specific)?
66%
42%
63%
43%
81%
53%
81%
52%
87%
87%
73%
76%
76%
23%
19%
21%
73%
86%
29%
41%
30%
47%
16%
36%
14%
36%
8%
10%
21%
16%
22%
52%
52%
53%
20%
13%
5%
17%
7%
11%
3%
11%
4%
12%
5%
3%
6%
8%
25%
29%
26%
7%
0% 20% 40% 60% 80% 100%
Writing
Vocabulary
Reading Comprehension
Problem Solving
Print Awareness
Perseverance
Object Counting
Mathematical Patterns
Letter Naming
Hand Dominance
Gross Motor
Grip & Manipulation
Following Directions
Emotional Literacy
Emotion Regulation
Emotion Expression
Crossing-Midline
Book Orientation
Very confident Somewhat confident Not confident
EAG K-3 Formative Assessment: Final Report September 2017 43
• When the evidence is sufficiently specific, how often would a teacher and outside rater
assign a similar level to the child’s skills (the same status summary rating) based on the
evidence documented?
The methodology used for this evidence analysis followed important standards in assessment
development (American Psychological Association, American Educational Research
Association & National Council on Measurement in Education, 2014), and is detailed in
Appendix D. The evidence analysis entailed selecting a sample of records for each construct,
which were then coded by trained reviewers using a structured coding protocol.
• Sampling. Teachers entered 80,000 evidence records into the platform across the fall and
winter field test periods. From an exploratory analysis of teacher evidence collected during
the 2015–16 pilot test, we calculated the number of records needed to adequately examine
the reliability of coding the evidence and status summaries for each construct. This resulted
in drawing a smaller sample for the constructs with higher interrater agreement in the pilot
(e.g., Book Orientation, Grip & Manipulation) and a larger sample for the more complex
constructs that had shown lower interrater agreements in the pilot (e.g., Problem Solving,
Perseverance). The sample was further stratified by state and teacher to ensure that it was
representative of the field test participants.
• Coding protocol. Teachers’ evidence consisted of anecdotal notes, photographs, and
videos. From previous reviews of the evidence collected in North Carolina’s pilot of selected
constructs (Ferrara & Lambert, 2015), we learned that teacher evidence varies in specificity
and that some evidence may not be sufficiently specific to code. Consequently, we
developed a coding scheme that first examined each piece of evidence for three aspects of
specificity: construct relevance (Was the evidence related to the content of this construct?),
skill specificity (Was the evidence related to a skill level on the progression?), and child
specificity (Was the evidence specific to this child?). If the body evidence for the construct
was specific to skills on the progression and to the child sampled, the reviewer continued to
examine whether the evidence supported the status summary endorsed by the teacher. The
reviewer (1) coded a status summary, based on that evidence; (2) compared her status
summary with the teacher’s; and (3) determined whether the teacher’s status summary code
was consistent with the evidence documented. The detailed coding protocol and rubric are
in Appendix D.
• Reviewer training. The reviewers were project staff familiar with the assessment materials.
To ensure consistency among them, a 10 percent sample for each construct was coded by
EAG K-3 Formative Assessment: Final Report September 2017 44
two reviewers before the full sample was coded. Cohen kappa estimates ranged from .85 to
1.00, indicating almost perfect to perfect agreement among reviewers. Reviewers discussed
any disagreements to reach consensus and clarify coding rules to be used in the remaining
90 percent of the sample. Reviewer interrater reliability coefficients are displayed in
Appendix D.
Most of the evidence teachers provided in the platform was construct relevant, ranging from 70
percent for Perseverance to 99 percent for Vocabulary. A smaller proportion was found to be
specific to skills on the progression or specific to the child being observed; Across constructs,
the percent of evidence determined to be specific to skills on the progression ranged from 33
percent for Perseverance to 87 percent for Writing. Besides Perseverance, the percentage of
evidence found to be skill specific fell below 50 percent for several constructs: Emotional
Literacy (41 percent) and Problem Solving, Reading Comprehension, and Emotion Regulation
Strategies (49 percent each). The variation in the extent to which the evidence was specific to
the child being observed ranged from slightly over 50 percent for Emotional Literacy and
Problem Solving to more than 75 percent for Book Orientation, Emotion Expression, Hand
Dominance, and Grip & Manipulation.
Exhibit 3.3 presents the results for all constructs. Note that teachers were documenting
evidence for their own use in placing children on the progressions. They were trained on the
importance of high-quality documentation but also encouraged to find shortcuts in providing
documentation that would be most helpful for them.
EAG K-3 Formative Assessment: Final Report September 2017 45
Exhibit 3.3. Evidence specificity by construct
Percent of Evidence that was . . .
Book Orientation 36 90 74 75
Crossing Midline 38 95 68 68
Emotion Expression 32 78 60 76
Emotional Literacy 55 83 41 51
Emotional Regulation Strategies 55 71 49 58
Following Directions 35 90 84 59
Grip & Manipulation 37 95 79 78
Gross Motor 35 88 74 66
Hand Dominance 37 93 76 76
Letter Naming 32 88 73 69
Math Patterns 56 84 80 64
Object Counting 41 75 72 55
Perseverance 60 70 33 60
Print Awareness 35 96 73 69
Problem Solving 53 84 49 51
Reading Comprehension 66 94 49 58
Vocabulary 65 99 80 67
Writing 67 98 87 73
When reviewers determined the evidence was specific enough to code a status summary, they
arrived at one based on the evidence. Reviewers then determined whether the teacher’s status
summary was consistent with the evidence documented and whether it matched the reviewer’s
status summary. Constructs varied in the proportions of evidence that were specific enough to
determine a status summary, ranging from 33 percent for Perseverance to more than
80 percent for Following Directions, Grip & Manipulation, Hand Dominance, and Writing. The
consistency of the evidence with the teacher’s status summary ranged from below 40 percent
(Perseverance, Emotion Regulation Strategies, Emotional Literacy) to more than 70 percent
(Crossing Midline, Grip & Manipulation, Hand Dominance, and Writing). Exact agreement
EAG K-3 Formative Assessment: Final Report September 2017 46
between the reviewer’s and teacher’s status summary ratings ranged from less than 30 percent
(Emotional Literacy, Perseverance, Emotion Regulation Strategies) to more than 70 percent
(Crossing Midline, Hand Dominance, and Writing). Exhibit 3.4 presents these findings.
Exhibit 3.4. Evidence analysis: Percentage of evidence specific enough to code, coded status summary consistent with the teacher status summary, and coded status summary matched the teacher’s exactly
Note. The total length of the bar indicates the percentage of evidence for each construct that was specific enough to code. The length of the green and blue bars is the percentage where the teacher’s summary was consistent with the evidence. The blue bar is the percentage of codes where the reviewer’s code and the teacher’s matched exactly.
Incorporating the assessment into classroom routines The K–3 Formative Assessment was designed to be manageable and implemented as part of a
teacher’s daily classroom routine. We examined this by surveying and interviewing teachers
who participated in the field test. After implementing the assessment in the fall and again in the
0% 20% 40% 60% 80% 100%
Writing (N = 67)
Vocabulary (N = 65)
Reading Comprehension (N = 66)
Problem Solving (N = 53)
Print Awareness (N = 35)
Perserverance (N = 60)
Object Counting (N = 41)
Math Patterns (N = 56)
Letter Naming (N = 32)
Hand Dominance (N = 37)
Gross Motor (N = 35)
Grip & Manipulation (N = 37)
Following Directions (N = 35)
Emotional Regulation (N = 55)
Emotional Literacy (N = 55)
Emotion Expression (N = 32)
Crossing Midline (N = 38)
Book Orientation (N = 36)
Coder and teacher SS matched Codes consistent with teacher evidence Specific enough to code
EAG K-3 Formative Assessment: Final Report September 2017 47
winter, most teachers indicated they were able to assess the constructs assigned to their grade
level, although some were not able to easily work certain constructs into their instructional
routines.
Most teachers were able to assess their assigned constructs, assessing 93 to 100 percent of
the children on the constructs designated for their grade level. Teachers also reported that they
were not able to assess some constructs as part of everyday instruction. Gross Motor and
Mathematical Patterns were particularly challenging to incorporate into daily classroom routines
(Exhibit 3.5). Teachers reported that Gross Motor was difficult to assess because of limited
interactions with children during recess and/or physical education classes and limited physical
activity outdoors in winter. Mathematical Patterns was also difficult to assess during instruction,
teachers telling us that it is not something they typically teach.
Exhibit 3.5. Percentage of teachers who reported they were unable to assess constructs during everyday instruction: Most challenging constructs
We asked teachers what they thought about the manageability of this assessment for other
teachers at their grade level, and 51 percent thought that other teachers would be able to
incorporate the assessment into their classroom routines. Those who said teachers would not
be able to reported concerns that teachers would need more time for planning, collecting the
data, and implementing with a whole classroom.
When asked about challenges in implementing this assessment, teachers consistently cited
time and lack of support in the classroom as the primary issues. More than half the teachers
22%
22%
23%
24%
46%
54%
Vocabulary
Emotion Expression
Emotional Literacy
Emotion Regulation
Mathematical Patterns
Gross Motor
n = 133 teachers
EAG K-3 Formative Assessment: Final Report September 2017 48
(57 percent) reported not having enough support in the classroom to collect evidence. About
half (54 percent) of the teachers in the sample had paid paraprofessionals or aides in their
classrooms. Another challenge teachers reported was a lack of alignment with state standards.
As noted, Mathematical Patterns was problematic for teachers. Although not formally introduced
in the Common Core State Standards until grade 3, the foundational activities for Mathematical
Patterns appear as early as kindergarten, but this was not obvious to the teachers. Constructs
for nonacademic skills are not directly represented in the standards, so teachers might not have
considered them a priority or their responsibility. The most frequent challenges reported by
teachers are shown in Exhibit 3.6.
Exhibit 3.6. Percentage of teachers who reported each challenge experienced in implementing the assessment
Except for a few constructs, the data suggest the assessment can be incorporated into
classroom routines.
13%
19%
28%
33%
35%
57%
61%
Lack of support/buy-in from school administrators
Not enough information about assessment content
Technical difficulties accessing the platform or app
Not enough staff in the school were trained
Lack of alignment with state standards
Not enough support in classroom to collect evidence
Not enough time to implement
n = 132 teachers
EAG K-3 Formative Assessment: Final Report September 2017 49
Chapter 4: Implementation supports for teachers This chapter addresses the supports teachers require to implement the assessment with fidelity and produce valid information. Field test findings related to the following supports are discussed: professional development, ongoing assistance, assessment materials, and technology.
Professional development Teachers participating in the field test received 2 full days of training by SRI staff and staff
representing the technology platform vendor. Staff from North Carolina’s K–3 Formative
Assessment project provided additional support. The training covered formative assessment,
the assessment content (the constructs), the assessment process, the technology platform, and
field test research procedures. Teachers new to the assessment received the full 2 days of
training, whereas teachers returning from the pilot study received an abbreviated refresher
training (1–1.5 days). Drawing on our experience in training pilot test teachers to use this
assessment, we concentrated the field test training on two topics:
• Assessment content. Because adequately covering all 18 constructs in 2 days is difficult,
the training incorporated different constructs as examples of how to implement the
assessment throughout the training and provided an overview of the most salient aspects of
each domain. Teachers were encouraged to deepen their knowledge with online construct
modules before assessing each construct. One module was provided for each construct.
Teachers also were provided with a suggested administration schedule with the constructs
grouped into clusters that would be easier to administer together to help manage the
number of constructs.
• Assessment process. Teachers who had participated in the pilot appreciated the
opportunities to practice the assessment process, so the field test training incorporated self-
paced activities that teachers could move through individually or in small groups to reinforce
steps in the formative assessment process.
To answer the research question concerning the nature and quality of the initial professional
development provided to teachers, we surveyed teachers approximately 3–4 weeks after they
completed the training and had started to use the assessment materials; 145 teachers
responded. This survey also included questions about their background and experience. Focus
groups conducted at the end of the field test also addressed the training. Copies of the survey
and focus group protocol are in Appendix D, as well as a table describing the teachers who
participated in the field test.
EAG K-3 Formative Assessment: Final Report September 2017 50
Both the new and returning teachers responded positively to the survey questions about the
training, indicating that as a result of the training, they felt prepared to implement the
assessment. The majority of teachers (68 percent) felt they were “very” or at least “adequately”
prepared to implement the assessment after the field test training. Three-fourths of the new
teachers (75 percent) and more than half of the returning teachers (53 percent) felt that the
demonstration of the technology platform was a helpful component of the training. Other
aspects of training that new teachers found especially helpful were the overview of the construct
progression structure (77 percent), overviews of the five domains (69 percent), and
demonstration of Teacher Site (64 percent). In addition, the activities supporting the five steps in
the formative assessment process were rated as helpful by 37 to 52 percent of teachers
(Exhibit 4.1).
Exhibit 4.1. Aspects of training teachers new to assessment found most helpful
Teachers liked the interactive, hands-on nature of the training but suggested improvements
included additional “refresher” training, particularly on the use of the technology. Teachers
mentioned in focus group interviews and on surveys that the training provided a lot of
information—perhaps too much information—to be properly digested in a two-day session and
they would have benefited from an opportunity to revisit the technology after they had started
using it.
Three-quarters of the teachers (75 percent) clustered the administration of the constructs when
implementing the assessment for the first time in the fall. Most teachers who clustered the
37%
37%
39%
45%
50%
50%
52%
64%
69%
75%
77%
Q&A panel with returning pilot teachers
Practice "Getting a plan"
Practice "Systematically using data"
Practice "Summarizing the status"
Overview of formative assessment
Practice "Assigning a learning status"
Practice "Looking for and gathering evidence"
Demonstration of the Teacher Site
Overviews of five domains
Demonstration of technology platform
Overview of construct structure
Percentage of Teachers (n = 107)
EAG K-3 Formative Assessment: Final Report September 2017 51
constructs found it a helpful (39 percent) or at least somewhat helpful (57 percent) strategy for
managing the new content. Several teachers commented that it was more useful to plan and
customize the construct groupings around what they were teaching at that time rather than use
any of the suggested clustering options.
Materials and ongoing supports To address the question about what ongoing PD and other supports are needed to implement
the assessment well, we examined the responses of teachers who were surveyed at the end of
the fall and winter administrations (n = 135 and 133 teachers, respectively), as well as their
comments in the focus groups conducted at the end of the field test (n = 111 teachers). We
begin with a description of the materials and supports provided to teachers.
Materials
Each teacher received a K–3 Formative Assessment binder containing overview guidance on
how to implement the assessment and assessment materials for each of the 18 constructs.
• Construct progression document with performance descriptors and classroom examples
for each skill
• Assessment means form describing how to assess the construct, including situations and
tasks as needed to supplement observations
• Single and multiple child documentation forms for teachers to record their evidence,
associated preliminary learning statuses, and status summaries; these forms also included
“watch fors” for each skill on the progression, brief descriptions of the behaviors that
differentiate each skill from the next.
Ongoing supports
Ongoing professional development and other supports were available after the field test
trainings.
• Teachers were given access to the website Teacher Site that had all these construct-
specific materials, as well as a narrated training module for each construct and the resource
materials used at the field test trainings. A page from this website, showing where to find
materials for each of the 18 constructs, is presented in Exhibit 4.2
EAG K-3 Formative Assessment: Final Report September 2017 52
Exhibit 4.2. Sample page from Teacher Site showing construct materials
• Follow-up webinars were held for interested teachers early during the fall administration and
then either at the end of the fall data collection or at the beginning of the winter data
collection window. This included review and questions about the technology platform and
evidence application.
• Staff members from the technology platform vendor were available via email or help desk to
answer teachers’ questions about the platform or evidence application.
• The field test data collection was monitored, and teachers were offered help via email if they
appeared to have questions about how to document their assessment activities, based on
what they had entered into the platform.
Teachers’ perceptions of the supports
Teachers were asked in surveys and focus group interviews about the quality and helpfulness of
the materials and resources provided and what additional supports would be useful in
EAG K-3 Formative Assessment: Final Report September 2017 53
implementing the assessment beyond the field test. Most of the 134 teachers who responded to
the survey found the materials helpful (Exhibit 4.3). Specifically, they found the construct
progressions (96 percent), technology platform (90 percent), Teacher Site, (88 percent), K–3
Evidence Application (85 percent), and the Assessment Means Form (78 percent) to be the
most helpful materials and resources in implementing the assessment. Most teachers also
reported these assessment materials to be of high or moderate quality (90–94 percent).
Exhibit 4.3. Percentage of teachers rating materials as very or somewhat helpful
Only 44 percent of teachers (n = 59) reported that they watched at least one of the construct
modules, which were available on the Teacher Site and recommended for review after the
training as part of a self-study preparation for assessing each construct. Teachers most
frequently watched the modules for constructs perceived as more difficult to assess, such as the
social-emotional constructs and Mathematical Patterns. Nearly all teachers who viewed the
construct modules found them helpful (ranging from 94 to 100 percent across constructs).
Teachers who did not watch the modules either felt sufficiently prepared from the training,
materials, or from participating in the pilot (49 percent) or did not have time to watch the
modules (39 percent).
When asked whether their school had provided support for the implementation of the
assessment, 56 percent of teachers reported at least some kind of support (Exhibit 4.4).
Teachers’ suggested supports for future successful implementation of the assessment included
increased time allocated for planning (77%) and having team meetings or other opportunities to
48%
50%
63%
78%
85%
88%
90%
96%
Single Child Form
K-Entry Checklist
Multiple Children Form
Assessment Means Form
K-3 Evidence App
K-3 Teacher Site
K-3 Technology Platform
Construct Progression
Percentage of Teachers (n = 134)
EAG K-3 Formative Assessment: Final Report September 2017 54
discuss the assessment with colleagues (35%). In interviews and open-ended survey
responses, teachers also noted that they would benefit from more practice with the technology,
additional examples of activities they can use to assess each construct, more practice coding
real examples of evidence, and examples of how other teachers fit the assessment into
everyday instructional activities.
Exhibit 4.4. Percentage of teachers reporting supports provided and suggestions for supports needed for assessment implementation
Technology Teachers’ use of the assessment was supported by the technology platform and by the use of
tablets or smart phones. Many field test teachers were provided with tablets funded through the
grant to facilitate gathering evidence (especially videos and photos). All the assessment data
were recorded on the platform for easy review and reporting by teachers.
To address the research question concerning the role of technology in supporting fidelity of
implementation and instructional utility, we asked teachers about their use of the technology
platform and the mobile evidence application, both in surveys and during the focus groups. The
majority of teachers found the technology platform (78 percent) and mobile application (71
percent) helpful in implementing the assessment. In focus groups, teachers noted that the
technology was critical for making the assessment more manageable by expediting the
uploading of data (using the application), monitoring their data collection process, monitoring
student progress, and digitally organizing large quantities of information. Some teachers
2%
16%
18%
35%
77%
21%
3%
15%
24%
12%
Access to computer/laptop in classroom
Improved Internet access/connectivity
Professional learning communities
Team meetings to discuss
Increased time allocated for planning
Supports Received Supports Suggested
Percentage of Teachers (n = 131)
EAG K-3 Formative Assessment: Final Report September 2017 55
reported challenges with using the technology, including technical difficulties accessing the
technology platform or application (26 percent), not enough practice with the technology
platform or application before using it in the classroom (13 percent), and problems using the
technology platform or application. Staff members from the tech platform vendor were available
to help teachers with the technology, but they were not able to resolve district and school-level
issues (e.g., availability of tablets, lack of connectivity or bandwidth).
EAG K-3 Formative Assessment: Final Report September 2017 56
Chapter 5: Broader implementation supports Broader supports, beyond the teacher supports discussed in chapter 4, are required to implement the K–3 Formative Assessment with fidelity and produce valid information. The field test examined three broader system-level supports: professional development for administrators, building-level infrastructure, and classroom conditions. The findings are summarized here, with more detail provided in a related report by Child Trends.
Implementation of the K–3 Formative Assessment requires that teachers understand the
contents of the assessment and successfully manage the collection and entering of information.
We hypothesized and the pilot test results confirmed that successful implementation would
benefit from multiple types of support. These could include supports directly related to the
assessment (e.g., the provision of materials and technology), as well as broader supports that
facilitate the formative assessment process (e.g., provision of time, grade-level teams, and
classroom conditions). Administrators can play an active role in supporting successful
implementation by creating an environment conducive to the formative assessment process.
The field test addressed three research questions about broader implementation supports:
• What were the nature and quality of the initial professional development provided to administrators and others?
• What building-level infrastructure is seen as needed to support teachers in implementing
the assessment with fidelity and to use it to guide instruction?
• What are the classroom conditions that support or serve as barriers or facilitators to
implementing the formative assessment well (to fidelity) and using it to guide instruction?
To address these questions, Child Trends surveyed the administrators of teachers participating
in the field test near the end of the child assessment data collection in March−April 2017. In
addition, SRI included questions in teacher surveys and focus groups about the kinds of
building-level supports and classroom conditions that could sustain successful implementation
of this assessment. The factors that teachers reported were important to their assessment
implementation efforts are presented in the Child Trends report (Gebhart, Pratik, & Lin, 2017)
and summarized here.
Professional development for administrators Because we hypothesized that administrators play a key role in supporting teachers in the
formative assessment process, we developed and provided training for administrators of
teachers participating in the field test. This was either direct in-person training or virtual follow-
EAG K-3 Formative Assessment: Final Report September 2017 57
up webinars or training modules developed for building- and district-level administrators. This
training covered four topics, each in a separate online module:
• Module 1: What Does This Assessment Look Like?
• Module 2: Project Purpose and Background
• Module 3: Data Security and the IRB (institutional review board)
• Module 4: Supporting Your Teachers
Module 4 included suggestions on how administrators could support their teachers, such as
providing them with opportunities for reflection and collaboration with other teachers, discussing
the assessment in professional learning communities or staff meetings, and providing planning
time for teachers to work individually or meet with others implementing the assessment within
and across grades. These suggestions were summarized and shared with administrators as
part of the training (Appendix D).
Child Trends surveyed 73 principals and district-level administrators near the end of the field
test (March−April 2017) about the training they had received, their involvement in the K–3
Formative Assessment field test, resources provided to teachers, and facilitators and barriers to
assessment implementation. A total of 29 administrators (22 school principals and 7 district-
level administrators) responded to the survey. This was approximately 40 percent of the
administrators with teachers participating in the field test. Among the survey respondents, 19
reported that they had received training: 18 attended the in-person training, 1 also participated
in a follow-up webinar, and 1 viewed the administrator modules in lieu of the in-person training.
Most administrators reported benefiting from these trainings (e.g., 89 percent reported it as
“very” or at least “somewhat” helpful), and they also recommended additional follow-up trainings
throughout assessment implementation.
Not surprisingly, administrators who did not participate in any training reported that it was
difficult for them to support the implementation of this assessment. The in-person training was
held directly before school started so those administrators who attended had little time to
communicate with teachers about the assessment process or make adjustments in planning or
provision of supports before the field test began. Some administrators had difficulty seeing the
benefit of implementing the K–3 Formative Assessment in their schools, which may have
contributed to less than ideal support or buy-in for the assessment during the field test. The
administrators’ failure to see the benefit of the assessment did not create a problem for the
teachers, however. Relative to other challenges, only 13 percent of the field test teachers
EAG K-3 Formative Assessment: Final Report September 2017 58
specifically cited lack of buy-in or support from administration as a challenge in implementing
this assessment.
On the other hand, teachers did report that administrator support would have improved the
assessment process. As reported in Chapter 4, when asked what kinds of supports would be
needed to implement the assessment beyond the field test, many teachers in both the survey
and focus group interviews identified the kinds of supports that require administrator action;
among them were
• Providing additional planning time (77 percent)
• Providing additional staffing support in the classroom (57 percent)
• Convening team meetings (35 percent) or professional learning communities (18 percent) to
discuss the assessment
• Training additional staff in the building (e.g., administrators and teachers) (33 percent)
• Providing better access to technology (i.e., better Internet connectivity and provision of
tablets) (28 percent).
One teacher made the distinction between moral support and substantive support, noting that
her administrator was very supportive in letting her attend training and implement the
assessment in her classroom but had not provided any of the substantive supports that would
have been useful.
Administrators might be more willing and able to support teachers in using the assessment if
they had a better understanding of how the assessment can inform instruction. A small portion
(14 percent) of teachers reported that they did not use information from the assessment to
inform their instruction, some explaining that they did not know how to use the information
gleaned from the assessment. Supporting the teachers in learning how to use the data was
outside the scope of the field test, but it is likely that both teachers and administrators would
experience more benefits from implementing the assessment if teachers were better prepared
to use assessment results in their instruction.
Building-level infrastructure The administrator training materials identified several effective practices that support the
formative assessment process, including:
EAG K-3 Formative Assessment: Final Report September 2017 59
• Creating a positive classroom culture (ideally an environment with less need for discipline,
less time spent on behavior and more time supporting positive relationships with teachers
and peers)
• Use of multiple instructional formats with opportunities for active learning, including use of
whole group, small group, centers, situations, pairs, and individual learning
• Engagement in purposeful and intentional planning and decision-making, including
collaboration with peers to discuss evidence, the assessment process, and planning.
Teachers confirmed the message provided to administrators in the training; more than three-
quarters (77 percent) recommended that additional time for planning (e.g., how to incorporate
the assessment into instruction) would be helpful in successfully implementing the assessment
beyond the field test. When asked what kind of supports they had received, 44 percent of
teachers reported that they did not receive any of the supports listed to assist with
administration of the K–3 Formative Assessment. Among those who reported receiving support,
the most common were allowing teachers to hold team meetings to discuss the assessment (24
percent) and providing access to a computer or laptop in the classroom (21 percent).
Administrators identified additional administrator training and a desire to incorporate other staff,
such as school psychologists or occupational therapists, into the assessment process. Teachers
concurred that training more school staff and having them assist in collecting data for the
assessment would be beneficial.
Classroom conditions Teachers implemented the assessment under different classroom conditions. These conditions
varied in ways including
• The extent to which teachers had technology available to support the assessment
• Teachers’ own previous training and preparation to implement formative assessment
• How teachers allocated class time across different learning arrangements.
Technology available to support the assessment
Teachers identified technology as a critical component of assessment implementation. All
teachers reported having Internet access in their classrooms, and 77 percent reported having
“good” internet access. Almost all teachers (95 percent) had access to a computer, and 87
percent had a tablet or other mobile device that they could use in administering the assessment.
Although all teachers had access to the Internet, some reported during focus groups that
EAG K-3 Formative Assessment: Final Report September 2017 60
intermittent and slow Internet speed delayed uploading evidence. Some teachers chose to
upload evidence at home or opted not to upload audio, video, and/or image files because of the
Internet issues at the school. Teachers reported in the pilot and again in the field test that
access to reliable technology, including a tablet in the classroom to facilitate data collection and
a high-quality Internet connection to expedite uploading data collected via tablets and
smartphones, was essential to implementing the assessment.
Teacher training and preparation to implement formative assessment
Half the field test teachers reported having participated in training on formative assessment
before the field test, averaging about 15 hours of professional development with a range of 2 to
60 hours. More than 75 percent of teachers reported that they were already using formative
assessment in their classroom before this project. This information would suggest that most
teachers should have been comfortable with the formative assessment process, but the
teachers’ desire for more training with the K–3 Formative Assessment process suggests they
were not familiar with this type of formative assessment.
Class time spent in different learning arrangements
Successful implementation of this assessment requires that teachers be able to observe both
academic and nonacademic constructs in a variety of classroom settings and activities.
Teachers who have students spend time in a variety of arrangements (e.g., working in small
groups, individually, with peers) should find it easier to observe their students and capture the
skills on the assessment progressions. Teachers surveyed during the field test reported
considerable variation in how children in their classrooms spent their time:
• Working with peers (under teacher direction). Approximately half the teachers (51 percent)
reported that children spent 1 hour per day working with peers, and another one-quarter of
teachers reported that children spent 2 hours working with peers.
• Working independently. About half the teachers (48 percent) reported children spent 1
hour working independently, and another one-quarter of teachers reported that children
spent 2 hours working independently.
• Working in small groups (with teacher). About 42 percent of teachers reported they spent
an hour per day working in small groups with students. Few teachers (14 percent) reported
spending 2 or more hours in small group instruction.
EAG K-3 Formative Assessment: Final Report September 2017 61
The amount of classroom time teachers spent in the different learning arrangements is shown in
Exhibit 5.1. These data suggest that teachers used a variety of classroom arrangements, which
should have provided ample opportunity for them to observe the constructs in the assessment.
Exhibit 5.1. Time spent in different classroom learning arrangements
22%
24%
17%
43%
27%
48%
38%
51%
42%
46%
26%
26%
24%
12%
24%
5%
12%
8%
0% 20% 40% 60% 80% 100%
Working independently
Working on individual tasks under teacherdirection
Working with peers under teacher direction
Working in small groups with teacher
Teacher lecture/large group discussion
Percentage of Teachers (n = 132)
Half hour or less About 1 hour About 2 hours 3-4 hrs or more
EAG K-3 Formative Assessment: Final Report September 2017 62
Chapter 6: Using the K–3 Formative Assessment with all children This chapter summarizes the findings and recommendations of expert reviews from three perspectives: disability, dual language, and racial/cultural. We also summarize the resulting revisions to the assessment materials to improve teacher use with all children, including children with disabilities, English learners, and children from diverse racial and cultural backgrounds.
During the field test, experts reviewed the K–3 Formative Assessment for its use with different
populations of children:
• From an ability-diverse perspective, for use with children with disabilities
• From a language perspective, for use with English learners
• From a racial and cultural perspective, for use with children from diverse racial and cultural
backgrounds.
This chapter summarizes the findings and recommendations from these three reviews and
describes the changes made to the assessment based on the reviews.
Disability perspective Because the assessment is based on observing children during typical classroom routines,
multiple opportunities are available for any child to demonstrate competence. This alleviates the
problems that assessments with structured tasks and rigid instructions present for assessing
many children with disabilities. Furthermore, as part of the administration of the K–3 Formative
Assessment, teachers are instructed to use any instructional adaptations or accommodations on
a child’s IEP. As presented in chapter 2, most teachers reported that the assessment was no
more difficult to use for children with disabilities and that the information was useful for their
instruction. We did find, however, that teachers generally placed children with IEPs lower on
each construct progression than children without IEPs in both fall and winter administrations and
across grade levels.
During the ECD codesign process, each construct progression and the associated assessment
means were reviewed and adjusted if needed based on considerations for children with
disabilities. A section of the overview guidance was developed to address using the assessment
with children with disabilities, and specific guidelines were written for several constructs.
In addition, to ensure that we had fully considered the disability perspective, the assessment
materials were reviewed by Dr. Kathleen Hebbeler, a well-known expert in the assessment of
children with disabilities. Dr. Hebbeler was part of the SRI research team. She was involved
only in the enhancement of one construct (Object Counting) so was in a position to conduct an
EAG K-3 Formative Assessment: Final Report September 2017 63
objective review. She made several recommendations, some of which we were able to
incorporate into the final version of the assessment materials. Other recommendations will
require additional research, assessment design work, and field testing, some with children with
low-incidence handicaps. Some of these recommendations pertain to the assessment overall
and are summarized below under “Overview guidance.” Other recommendations more specific
to individual constructs are presented in Exhibit 6.1.
Overview guidance and general recommendations
The Assessment Overview and Directions That Apply to All Constructs, which is an overview
document with guidelines for use with all constructs, is one of the strengths of the assessment
with regard to children with disabilities. This guidance makes clear that
• The assessment is based on authentic interaction.
• Alternative means of communication are acceptable.
• Supports can be offered to the child.
• Accommodations are acknowledged and acceptable, along with flexible
administration/collection of evidence (e.g., positioning support, various times of the day).
Using other feedback in Dr. Hebbeler’s review, we added the following features throughout the
assessment that should make it easier for teachers to understand how to implement this
assessment with all children and the accommodations or exceptions that may be noted:
• Elaborated on guidance and added notes to the Assessment Means Form for each
construct, briefly explaining what considerations would be made when assessing this
construct for children with disabilities. This includes clarifying that the various situations
developed to help a teacher pinpoint a child’s skill level should be used as appropriate given
the child’s disabilities.
• Added examples of children with different types of disabilities throughout the Construct Progression Documents to help teachers see children with disabilities reflected in the
assessment materials.
• Added a space on the Single Child Documentation Forms for teachers to indicate when
the assessment of a construct is not appropriate for that child based on his/her disability to
explain why the teacher has not assessed the construct. We recommend that there also be
a place where teachers can indicate such information in a neutral or straightforward way on
the technology platform used to support this assessment.
EAG K-3 Formative Assessment: Final Report September 2017 64
Construct-specific guidance
In her review, Dr. Hebbeler noted that some constructs are not appropriate for children with
some disabilities. This is not a problem for the assessment as long as teachers have a way to
indicate this and are not required to place every child on a progression. An example of such a
progression is Gross Motor, which would not be appropriate for children with severe mobility
impairments. The review also included recommendations for adaptions or additional
development work for some constructs with some specific disabilities (for example, the need to
develop a construct analogous to Letter Naming for children learning to use Braille). Some of
these recommendations will require assessment development work and further research
specifically for children with low-incidence disabilities These recommendations are summarized
by disability groups in Exhibit 6.1.
EAG K-3 Formative Assessment: Final Report September 2017 65
Exhibit 6.1. Findings from review for each construct by disability
Construct Disability/Impairment More Research Notes
Hand Dominance Not appropriate for children with limited use of their arms or
hands.
Grip & Manipulation
Not appropriate for children with limited use of their arms or hands; situation not appropriate for children with limited vision.
Crossing Midline
Not appropriate for children with limited use of their arms; requires more research on how to assess children with visual impairments.
Gross Motor Not appropriate for children who cannot walk or with limited motor skills.
Emotion Expression Use of this progression likely to be more challenging for
special education professionals trained to focus on behavior.
Emotion Regulation Strategies
Progression’s emphasis on articulating strategies rather than showing appropriate behavior disadvantages children with limited language.
Emotional Literacy
Need more research on relationship between emotional literacy & language disabilities; teachers will need alternative ways to assess children with limited communication skills.
Book Orientation Need an appropriate adaption for children with visual
impairment or insufficient motor skills to hold a book.
Print Awareness
Needs an adaptation for a child with visual impairment; may require more research to determine use for children reading Braille.
Letter Naming Need an adaptation for children with visual impairments; requires an expert in working with students with visual impairments to identify the appropriate skill levels.
EAG K-3 Formative Assessment: Final Report September 2017 66
Construct Disability/Impairment More Research Notes
Following Directions As a measure of receptive language, this progression is
heavily dependent on a child’s language skills.
Writing
Need to clarify that alternative ways of writing are acceptable for children who do not have the motor capability to draw pictures or write letters; also needs to be adapted for children with visual impairments.
Reading Comprehen-sion
Requires more research: assessing children learning to read Braille; development of comprehension strategies in children with intellectual disabilities; how children can communicate these strategies.
Vocabulary
Some skill levels dependent on the production of language; need to develop a different progression for children who sign since steps involve word structure based on spoken or written English.
Perseverance Requires teacher be able to identify activities that are moderately or very (but appropriately) difficult for children with intellectual disabilities.
Object Counting Adaptations needed for children who are not able to move
objects on their own.
Mathematical Patterns
Teacher must consider that each situation requires different skills that would be challenging for children with different impairments (hearing, auditory processing, motor, and visual).
Problem Solving
Heavily dependent on a child’s verbal skills; requires teacher to identify appropriate problems for children relative to their intellectual abilities; more research needed for how to interpret unexpected approaches.
EAG K-3 Formative Assessment: Final Report September 2017 67
English learner perspective The observation-based nature of this assessment may lend itself more readily to learning about
what young speakers of languages other than English know and can do in much the same way
as it does for children with disabilities. As noted in chapter 2, most teachers reported that the
assessment was no more difficult to use for English learners and that the information was useful
for their instruction. We found no consistent differences in how teachers placed ELs on each
construct progression compared with non-ELs across both fall and winter administrations and
across grade levels.
Many of the constructs, however, require that the teacher communicate with the child to
understand where the child falls on the progression. The overview that accompanies the K–3
Formative Assessment instructs teachers “to communicate with the child in a language . . . the
child understands and can respond in. This could be a language other than English (which could
involve an interpreter),” but it is impossible to know how feasible it is for teachers to do this.
As part of the field test, the Build Initiative arranged for Dr. Linda Espinosa, a national expert in
the teaching and assessment of English learners, to review the assessment materials. Key
points from Dr. Espinosa’s review were the following:
• Appropriate use of this assessment requires considerable professional development for
teachers, including how to work with dual language learners (DLLs) and ELs.
• Best assessment practice includes having an adult who speaks the child’s language collect
the observation data (evidence), especially for constructs that rely more heavily on language
and communication.
• The examples in the construct progression and assessment means documents should
include descriptions of children with multiple languages who are acquiring English and what
assessing these children is like.
• The appropriateness of this tool for DLLs/ELLs depends on the state’s policy regarding
English language acquisition and dual language development (which should be reflected in
the state’s comprehensive assessment system).
Although some recommendations related to the content of the assessment materials were
outside the scope of the project, we included several of the recommendations from Dr.
Espinosa’s review in the final assessment and training materials:
• Communicating with the child in a language the child understands and can respond in (part
of the Assessment Overview and Directions That Apply to All Constructs)
EAG K-3 Formative Assessment: Final Report September 2017 68
• Including additional instruction in the Assessment Overview and Directions That Apply to All
Constructs that speak to the need to involve an adult who speaks the child’s language in
collecting observational data, especially for those constructs that rely more heavily on
language and communication
• Adding examples to the Construct Progression and Assessment Means Forms of
children who are acquiring English so teachers can see what it looks like to assess them
• Reinforcing these points in the teacher professional development materials (training slides).
Race/culture perspective The use of teacher observation as the method of data collection in the K–3 Formative
Assessment means that each child is being seen through the race and cultural lens that the
teacher brings to the classroom. Any implicit biases that teachers have will be reflected
throughout their interactions with children and in the specific instructional activities they use,
how they engage with the children, and in where they place children on the progression. Implicit
bias refers to the attitudes people have toward other people or the application of stereotypes
without conscious knowledge. The goal was to make sure that the assessment content does not
incorporate any racial or cultural bias; creating good content is a necessary but not sufficient
condition for addressing this issue in the implementation of the assessment. Some teachers
also may benefit from additional training in culturally responsive instruction, which will improve
their interaction as well as their use of the assessment with children from different racial or
cultural backgrounds.
As part of the field test the Build Initiative arranged for Dr. Tanisha Felder, a national expert in
racial and cultural equity in schools, to review of the assessment materials. Here are key points
from Dr. Felder’s review:
• Teachers should see children from diverse racial and cultural backgrounds in the construct
progression examples of what the skills look like in the classroom. The names of the
children in these examples should reflect this diversity, be current and balanced by gender,
and convey positive meanings.
• Teachers should create ways that the child is authentically engaged in the activities being
observed in order to get a good measure of the child’s skill level on a particular progression.
Relatability and relevance of activities may vary depending on the children’s race and
cultural background. This can best be addressed through professional development for
teachers in culturally responsive instruction.
EAG K-3 Formative Assessment: Final Report September 2017 69
• Teachers must be aware of and consider cultural norms in their evaluation of a child’s
behavior, especially with the social-emotional constructs where the expression and
regulation of emotions may vary somewhat by culture. Cultural norms also may come into
play when teachers are giving children directions for a task. This recommendation needs to
be implemented with the acknowledgement that the cultural norms in the context of this
assessment are the classroom norms. This recommendation also should be addressed
through professional development for teachers in culturally responsive instruction.
Following Dr. Felder’s recommendations, the SRI research team made some changes to the
assessment materials.
• Changed the names of children used in the examples that appear in the Construct Progression Documents and Assessment Means Forms so that they reflect more racial
and cultural diversity and so that they are balanced by gender, in present-day use, and
convey positive connotations
• Added text in the Assessment Overview and Directions That Apply to All Constructs to
remind teachers that they can modify the situation or task directions as appropriate to reflect
the racial and cultural norms of the children in their classroom
• Reminded teachers to use activities or tasks that are likely to be engaging for the child given
his or her background and individual interests and to modify tasks if they are not engaging
• Reinforced these points in the teacher professional development materials (training slides).
Because the assessment is based on teacher observations of authentic behaviors in daily
classroom routines, teachers will need to be skilled in culturally responsive instruction, which
might require additional professional development so they can both instruct and assess a child’s
skill level in culturally responsive ways.
EAG K-3 Formative Assessment: Final Report September 2017 70
Chapter 7: Summary, revisions, and recommendations This chapter summarizes the findings from the field test including how each of the construct progressions performed vis a vis the validity claims. It concludes with a summary of the revisions to the assessment and training materials and recommendations for future research.
Overall assessment findings In chapters 2 through 5 we answered two overarching research questions posed in the field test
regarding the validity claims for the K–3 Formative Assessment:
1. Does the K–3 Formative Assessment provide valid information about what children know
and can do?
2. What supports are required to allow the assessment to be implemented with fidelity and
produce valid information?
To answer the first question, we addressed a series of subquestions that reflect the validity
claims for the K–3 Formative Assessment. Below we summarize the findings for each
subquestion followed by a more detailed, construct-specific examination of the validity claims.
• Are the skill levels in the construct progressions in the proper sequence? Based on
the opinions of experts and teachers and the assessment data collected during the field test,
the answer is yes, the skills in these construct progressions appear to be in the right order.
However, psychometric analyses indicated that two skill levels on two progressions (Object
Counting and Mathematical Patterns) were ranked very close to each other and thus
including both skill levels as unique steps in a progression may offer little instructional utility.
A limitation of the field test was that the sample was too small (i.e., fewer than 10 children)
to verify the sequence of skill levels at the beginning or end of some progressions.
• Do the performance levels accurately reflect what is expected at entry to kindergarten? Using expert and teacher judgment and the fall assessment data, we
established performance levels in each progression that reflect what is expected of children
at kindergarten entry. For four constructs, experts and teachers disagreed about where
entering kindergarteners should be, and experts commented that kindergarten teachers tend
to underestimate children’s abilities for these constructs. The field test data were somewhat
limited for verifying the performance levels, in that the levels were established for entry to
kindergarten and the field test data were collected over the first two months of school, and,
furthermore, most of the field teachers were new to the assessment. We recommend
EAG K-3 Formative Assessment: Final Report September 2017 71
additional validation of the performance levels with assessment data collected in the first few
weeks of school by teachers who are experienced with the assessment.
• Is the assessment sensitive to children’s growth and development over time? Teachers reported and the assessment data confirmed that the constructs captured an
increase in children’s skill level over time, both within the school year and from one grade
level to the next. On several constructs, children were placed at the top of the progression in
the fall of kindergarten or first grade, indicating that most children had no opportunity for
growth. Therefore, these constructs offer limited instructional utility beyond these points, but
teachers may still find them useful for monitoring the learning of children who struggle in
these areas.
• Does the assessment provide information that is useful for instruction for grades K through 3? The answer to this question varied by construct and to some extent by grade
level. A majority of the teachers reported instructional utility for all of the constructs for all
children, including children with disabilities and English learners. However, even though
Writing was the construct with the largest number of teachers endorsing its instructional
utility, nearly a fifth (19 percent) of the teachers reported it was not useful. Likewise, about
one-third of the teachers did not find the constructs in the physical/motor development and
the social-emotional development domains useful for informing instruction; teachers
indicated that they did not see the relationship to the curriculum. We recommend further
research on these constructs and the supporting materials to determine what would be
needed to improve their instructional utility for teachers.
• Can teachers learn to implement the assessment with fidelity, including reliably coding a child’s learning status? The evidence teachers documented on the technology
platform indicated that teachers were able to collect multiple pieces of evidence, assign a
preliminary learning status to the evidence, and interpret that information to reach a status
summary placement on the progression that was consistent with the evidence. The field test
assessed reliability by comparing teachers’ status summaries with ratings of an external
reviewer, and for most constructs there was high agreement. The three constructs where the
agreement was less than 60 percent were Emotional Literacy, Emotion Regulation
Strategies, and Object Counting, suggesting the data from these constructs might be
somewhat less reliable. Teachers also reported some challenges with implementation. For
example, they indicated that the data collection process was time-consuming and
documenting evidence on the platform cumbersome. These issues could hinder teachers’
ability to implement the assessment as intended.
EAG K-3 Formative Assessment: Final Report September 2017 72
• Can teachers confidently determine a child’s learning status? Teachers were confident
in their status summary placements for most constructs. Teachers were less sure about
determining children’s learning status on the social-emotional constructs than on the other
constructs.
• Can teachers incorporate the administration of the assessment into classroom routines? Teachers found that the constructs most directly related to their instruction and to
other assessments, such as those in the language development and communication and
cognitive development domains, were easier to assess and fit into their daily routines. Some
teachers struggled with constructs that they considered outside their typical classroom
activities such as Gross Motor, Mathematical Patterns, and the social-emotional constructs.
To answer the second research question about the implementation supports required to
implement the assessment with fidelity and provide valid information for instruction, we used
data from the teacher surveys, teacher focus groups, and administrator surveys.
• What was the nature and quality of the initial professional development provided to teachers? The field test teachers received two days of face-to-face training that covered the
content and process of the assessment and use of the technology. Follow-up webinars and
online modules on each construct were available to provide more in-depth information.
Teachers generally liked the training provided for the field test and felt it prepared them to
implement the assessment. They recommended more time be devoted to reviewing the
technology and implementation ideas from their colleagues once they began using the
assessment.
• What ongoing professional development and other supports are needed to implement the assessment well? The field test yielded limited information to answer this question
because the full range of ideal supports for the formative assessment was not represented.
Ideal implementation conditions would include, for example, committed administrators,
sufficient planning time, and opportunities for teachers to work together in implementing the
assessment and using the results. Because these supports did not exist in most of the sites,
the field test was not able to explore their effects on implementation. In surveys and
interviews, however, administrators and teachers reported that these kinds of supports
would be helpful to implement the assessment.
• Are there changes to the assessment and training materials that would make the assessment more manageable? Teacher feedback, input from experts from disability,
language and race/culture perspectives, and analysis of the assessment data suggested a
EAG K-3 Formative Assessment: Final Report September 2017 73
need for minor changes to the assessment materials. We have made revisions to the
assessment and training materials based on this input. Specifically, we revised examples
provided in the construct progression documents, clarified progression skill levels in the
assessment means forms, and repackaged the training materials to more fully cover the
most critical topics for implementation of the assessment.
• What was the nature and quality of the initial professional development provided to administrators and others? Training was provided to administrators either in person or
virtually through follow-up webinars or training modules that were developed for building-
and district-level administrators. According to their survey responses, fewer than half of the administrators associated with the field test teachers either attended an in-person training or
took advantage of the other training opportunities. The survey respondents found the
training beneficial and recommended more follow-up training opportunities once teachers
begin implementing the assessment.
• What building-level infrastructure is seen as needed to support teachers in implementing the assessment with fidelity and to use it to guide instruction? Field test
findings indicated that teachers need more time to plan for and implement the assessment,
including collecting evidences and loading the information onto the platform; opportunities to
hold team meetings to discuss the assessment; and a computer or laptop in the classroom.
• What are the classroom conditions that facilitate or serve as barriers to implementing the formative assessment well (to fidelity) and using it to guide instruction? Teachers
need access to the Internet in the classroom to enter or upload the data onto the technology
platform. Although all teachers had Internet access, not all of them had reliable access,
which was a barrier. Teachers reported using a variety of learning arrangements in their
classrooms including small groups, working with peers, and working independently. Having
a variety of arrangements should provide sufficient opportunities for teachers to observe
what is being captured in the construct progressions.
Findings by Construct To summarize the differences in findings across the constructs, we created a table with the
findings for each test of each validity claim for each construct (Appendix D, Exhibit D.7.1). Some
of the claims had multiple tests so we further summarized the evidence for each claim (see
Exhibit 7.1). We color-coded the evidence to indicate how strongly it supported each claim for
the construct, with darker colors indicating stronger support. Looking across the matrix, we can
EAG K-3 Formative Assessment: Final Report September 2017 74
see that some constructs performed quite well for all of the validity claims while others
presented a more mixed picture.
• The constructs with the strongest evidence were in the language development and
communication and physical/motor development domains: Writing, Book Orientation, Print
Awareness, Letter Naming, Reading Comprehension, Following Directions, Hand
Dominance, Grip & Manipulation, and Crossing Midline.
• The evidence was not as strong for constructs in the social-emotional development and
cognitive development domains: Emotion Regulation Strategies, Emotion Expression,
Emotional Literacy, Mathematical Patterns, Problem Solving, and Object Counting
• In addition, the evidence patterns for Gross Motor, Vocabulary, and Perseverance suggest
the need for additional discussion and follow up.
Below we discuss the findings for each construct within their domain and offer some
recommendations for revisions and additional professional development or research.
EAG K-3 Formative Assessment: Final Report September 2017 75
Exhibit 7.1. Summary of construct performance for each of the validity claims
*Approaches to learning
EAG K-3 Formative Assessment: Final Report September 2017 76
Language Development & Communication Domain
The seven constructs in this domain comprise preliteracy skills frequently assessed in preschool
and kindergarten (Book Orientation, Print Awareness, and Letter Naming), more traditional
elementary school subjects (Reading Comprehension, Writing, and Vocabulary), and one
measure of receptive language that is important to success in any elementary classroom
(Following Directions). Collectively, these constructs were well supported by most of the
evidence for most of the validity claims. These constructs have a strong research base and are
prominent in elementary curricula and assessments. The strong research base meant there was
much to draw on in developing the constructs, and their frequent inclusion in standards,
curricula, and assessments meant teachers were familiar with them. It is not surprising that
these progressions performed well with regard to sequencing of levels, opportunity for growth,
teacher confidence, and fit within classroom routines. They also fared better than constructs in
the other domains in terms of their perceived instructional utility and the extent to which
teachers could reliably determine progression placements based on the evidence they
documented.
Three language development and communication constructs would benefit from additional
research and development:
• Book Orientation was found to have limited instructional utility because most children were
placed at the highest level on the progression at kindergarten entry. This also meant the
construct was not sensitive to growth. A future decision is whether Book Orientation belongs
in an assessment designed for kindergarten through Grade 3.
• Following Directions was one of the constructs where the national experts and
kindergarten teachers did not agree on what entering kindergarteners should be able to do.
The performance levels were set based on expert recommendations, which resulted in only
45 percent of the field test sample assessed in the fall of kindergarten categorized as
“demonstrating” the expected skill level, whereas the percentages for most of the other
constructs were 90 percent or higher. Further research is needed to understand the
differences in how experts and teachers interpret the levels of this construct.
• Vocabulary measures a child’s conceptual understanding of how words work in our
language (e.g., synonyms, multi-meaning words, word structure, nuanced language) rather
than whether children have mastered the appropriate words at each grade level. Some field
test data suggested that teachers may be more familiar with the latter conceptualization of
vocabulary. Compared with other constructs, fewer teachers thought this construct captured
EAG K-3 Formative Assessment: Final Report September 2017 77
growth well, the summary status was less reliable, and teachers found the construct difficult
to assess. Together these findings suggest a lack of teacher understanding of the construct.
Teachers would benefit from more professional development on this construct, and more
research could strengthen its effectiveness in the assessment.
Physical/Motor Development Domain
Three of the four constructs in this domain — Hand Dominance, Grip Manipulation, and
Crossing Midline—were supported by the evidence for most of the validly claims. The exception
was Gross Motor which teachers did not see as relevant to what they teach, nor did its
assessment fit into their instructional routines. Instructional utility was a challenge for all of the
motor constructs; fewer teachers saw the constructs in this domain as useful for instruction
compared with constructs in the other domains. Another issue was opportunity for growth. More
than 70 percent of entering kindergarteners were at the top level of the progression on Hand
Dominance and Crossing Midline. These constructs might be most appropriately used as part of
a screening measure to identify children who are struggling in these areas. Teachers would then
need additional support to know how to help children in this area, to refer them for additional
evaluation, or both. The Gross Motor construct would benefit from additional research to
validate the sequencing of some of the skill levels. Also, few children were placed at the lower
end of this progression in the field test which meant that these skill levels are more appropriate
for pre-kindergarten children and that there were very little data at this end of the progression to
examine the ordering of the levels. More research is needed at the higher end of the
progression because teachers had difficulty identifying galloping and distinguishing it from
skipping.
Cognitive Development Domain
The three constructs in this domain (Object Counting, Mathematical Patterns and Problem
Solving) did not fare as well as the other constructs with regard to the validly claims. Teachers
were confident in their progression placements for these constructs and in the ability of these
progressions to capture growth and development over time, which was borne out by the
assessment data. However, the psychometric analyses suggested some minor issues with
sequencing of two of the skill levels in Object Counting and Mathematical Patterns. The
reliability of the status summary for Object Counting was lower than that of most of the other
constructs. The teachers and experts disagreed on performance expectations at kindergarten
entry for Mathematical Patterns and Problem Solving. The teachers were less able to fit
EAG K-3 Formative Assessment: Final Report September 2017 78
assessing Mathematical Patterns and Problem Solving into their classroom routines than other
constructs.
Object Counting appears to need more research on the structure of the progression.
Mathematical Patterns needs more research on structure and also presented problems for
teachers because they did not see it in their standards or curriculum. This might be resolved by
additional professional development or by the development of additional mathematical
progressions that are more aligned with the curriculum. Teachers recognized Problem Solving
as useful, but they may not have understood it well and it was challenging for them to assess.
More research and professional development on this construct may be needed to help teachers
collect information as part of everyday classroom routines.
Social-Emotional Development and Approaches to Learning Domains
For purposes of the construct summary, we have grouped Perseverance, the one measure of
the Approaches to Learning domain, with the three social-emotional constructs (Emotion
Expression, Emotion Regulation Strategies, and Emotional Literacy). The data on the validity
claims present a mixed picture for these four constructs suggesting a need for more work. The
findings support the order of the skills in these progressions and indicate that they are sensitive
to growth over time. Teachers agreed with experts on expectations for K entry performance.
Teachers reported during the pilot and again during the field test that the social-emotional
constructs were difficult to assess and about a third of the teachers did not see their
instructional relevance. The evidence that teachers provided to document where they placed
children on these progressions indicated that some teachers were not able to differentiate the
three constructs and did not understand what each was intended to measure. Although teachers
found the guidance for assessing these constructs helpful, teachers still reported that these
constructs were the hardest to assess, and the least useful in informing instruction, and they
yielded the lowest percentages of interrater agreement. On the other hand, some teachers in
the field test were very excited to see the social-emotional constructs included in the
assessment because they recognized the importance of these skills.
In contrast to the language development and communication domain, these domains are based
on a rapidly expanding but much smaller and newer body of research. The field is still
considering what to call some of the constructs within these domains and to how to differentiate
them from one another. These domains are well represented in standards for early childhood
across states but they are rarely included in K–12 standards. Accordingly, the kindergarten
through grade 3 teachers in the field test were less familiar with the content of these
EAG K-3 Formative Assessment: Final Report September 2017 79
progressions and the implications of the content for what they do in the classroom. Both the
conceptual issues and the absence of standards in these areas presented problems for the
constructs in these two domains. All four constructs would benefit from additional research on
the progression to clarify the content and make it easier for teachers to embed their assessment
into classroom activities. Without an educational context across preservice training and school
systems that recognizes and supports the importance of the social-emotional domain, it is
difficult to see how more professional development focused solely on the assessment will
address the entirety of the challenges in this domain, but clearly more professional development
is needed.
Based on the field test findings, we make the following recommendations:
• The focus on social-emotional development in the early elementary grades is bigger than
this assessment. It requires that teacher preparation programs and schools increase their
recognition of the importance of social-emotional development to learning and that teachers
can and should help children acquire skills and strategies in this domain. Teachers need to
be knowledgeable about social-emotional development and how they can support children
in acquiring skills in this domain. The professional development around these constructs
should continue to stress the contribution of social-emotional development to the child’s
success in school and how teachers can observe these skills in their students.
• These constructs would benefit from further development that incorporates both the
research and teachers’ perspectives in identifying skills that can be more easily observed as
part of classroom routines.
• Consider basing Emotion Regulation Strategies on observable behavior and do not require
the teacher to learn or infer what is going on in the child’s mind regarding the strategies he
or she employs.
Recommended grade levels for each construct Not all of the progressions were intended to be used from entry to kindergarten through grade 3,
but data on child performance were needed to identify precisely which constructs were suited
for which grade levels. The field test provided information to identify the appropriate grade levels
for each construct. We determined the grade levels in which to administer each construct for the
field test based on the results of the 2015–16 pilot study. When the data were unclear we
included higher grade levels to make sure we had tested the construct’s instructional utility at all
potentially relevant grade levels. As shown in Exhibit 1.1, this resulted in field testing all 18
EAG K-3 Formative Assessment: Final Report September 2017 80
constructs in kindergarten, 16 constructs in grade 1, 12 constructs in grade 2, and 11 constructs
in grade 3.
Based on the percentage of children in a grade level placed at the highest level of the
progression in the fall (see Exhibit 2.14), we made recommendations for the grade levels at
which each of the progressions has the greatest instructional utility. These are shown in
Exhibit 7.2. Any of the constructs would be appropriate for a child at any grade level who is not
on the highest level of the progression and therefore in need of additional support around the
skills measured by the construct.
Revisions/recommendations for assessment materials Based on the field test findings, we made some minor changes to the assessment materials to
further clarify some constructs for teachers. The review of evidence teachers documented as
part of placing children on progressions provided some insight into what teachers did not
understand about some skill levels on a particular progression. We also have recommended
additional research where changes in the progression itself should be considered. Where
possible, we identified the specific issue with constructs in need of more research. Exhibit 7.2
shows the construct progressions for which we made revisions to the assessment materials
based on the field test findings, the constructs for which additional professional development
may be needed, and the constructs for which we are recommending additional research. For the
few progressions that did not have good evidence across the set of validity claims, we
recommend that these progressions be used with caution until the additional research is
conducted.
EAG K-3 Formative Assessment: Final Report September 2017 81
Exhibit 7.2. Summary of construct-specific revisions and recommendations
Domain/Construct New
Recommended Grade Levels
Assessment Materials Clarified
Additional Professional Development
Recommended
Additional Research on Progression
Recommended
Additional Information
Hand Dominance (K) K entry Most children were already at top of progression in fall of K.
Grip & Manipulation (K–1)
Crossing Midline (K–1) K entry Most children were already at top of progression in fall of K.
Gross Motor (K–3) K-1 only Teachers continue to confuse galloping (Skills F and G) and skipping (Skill I); consider collapsing these skills or further clarify the difference in PD. Need more data to validate early steps in the progression that are more appropriate for pre-K.
Emotion Expression (K–3)
Teachers documented evidence for one construct that better fit another S-E construct and so may not understand the differences among the three constructs; we have provided examples and guidance to help teachers better distinguish the constructs. Emotion Regulation Strategies. It is difficult to observe children using strategies unless they verbalize them; consider refocusing progression skills more to self-regulation behaviors that can be observed, rather than on strategies.
Emotion Regulation Strategies (K–3)
Emotional Literacy (K–3)
Book Orientation (K) K entry Most children were already at top of progression in fall of K.
Print Awareness (K–1) K only Most children were already at top of progression in fall or winter of K.
Letter Naming (K–1) Clarified that children can identify letters by either naming them or selecting letters named for them to demonstrate letter naming.
Following Directions (K–3) Additional research needed to examine difference between K
teachers and experts on expected performance levels at K entry.
Writing (K–3) Clarified that teacher can establish purpose of writing (Skill D) by nature of assignment & student’s response.
EAG K-3 Formative Assessment: Final Report September 2017 82
Domain/Construct New
Recommended Grade Levels
Assessment Materials Clarified
Additional Professional Development
Recommended
Additional Research on Progression
Recommended
Additional Information
Reading Comprehension (K–3)
Changed name to Reading Comprehension Strategies. Evidence documented suggests additional PD may be needed to emphasize that children monitor for meaning as they read, as indicated by their response to miscues and strategies used.
Vocabulary (K–3)
Changed name to Vocabulary Concepts. Clarified difference between use of synonyms for concrete words (Skill C) and use of more nuanced language with subtle differences (Skills G and H). Multiple indicators suggested that teachers had difficulty assessing this progression so more PD may be needed.
Perseverance (K–3)
Skills E–H involve identifying activities which are moderately or very (but appropriately) difficult for child, which requires detailed knowledge across areas of the curriculum as to the precise difficulty level of a task for each child. Need to provide teachers more guidance on how to identify what is an appropriately challenging task for each child.
Object Counting (K–2) K–1 only Skills B, C and E are conceptually connected (consistently counts 4–6 objects) and need to be differentiated from Skill D (cardinality); lower reliability suggests additional teacher PD may also be needed.
Mathematical Patterns (K–3)
Need to do more to help teachers understand differences between sequential and growing patterns and between recursive and functional rules; may need research into other early mathematics progressions that are more aligned with K–3 curriculum.
Problem Solving (K–3) Need to provide teachers more guidance on how to identify what is an appropriately novel and challenging problem for each child.
EAG K-3 Formative Assessment: Final Report September 2017 83
Clarify the meaning of the construct
Based on teacher questions and comments during the field test and the nature of the evidence
they documented on the technology platform, we recommend that the names of two constructs
be modified to clarify the essence of the constructs:
• Reading Comprehension. Some teachers seemed to assess this construct assuming it
was a conventional measure of reading comprehension in which children read passages
and respond to questions about what they have read. Instead, this construct addresses the
child’s ability to monitor for meaning and apply comprehension strategies when reading. The
first few steps on this progression (Skills A through C) are pre-reading skills so the emphasis
on comprehension strategies is not as important as it is in the later steps. Starting at Skill D,
the child is reading and the focus of the construct is on comprehension strategies.
Consequently, we recommend that the construct be renamed Reading Comprehension Strategies and have changed the name accordingly in the assessment materials.
• Vocabulary. Similarly, some teachers seemed to assume that the Vocabulary construct was
about the word lists and word walls so common in early elementary classrooms. The skills
on the progression address the child's conceptual understanding and use of words. To help
teachers focus on the concepts represented by the different skill levels in this progression,
we recommend that the construct be renamed Vocabulary Concepts and have changed
the name accordingly in the assessment materials.
Clarify progression skills (examples, watch fors, assessment means)
In some of the progressions, there are examples of a particular skill that did not appear to be
well understood by the teachers based on their comments or the evidence they used to
document placements on the progression. Rather than change the skill itself, which had been
vetted by experts in both child development and assessment, we made clarifications in the
assessment materials: examples to better illustrate the skill in the construct progression and
clarifications regarding how to assess skills. We have made the revisions listed in Exhibit 7.2.
Conduct additional research on some constructs
As summarized earlier in this chapter and in chapters 2 and 3, the validity evidence for some of
the constructs was not as strong as we would have liked. We recommend additional
development work for some constructs including reconsideration of the construct itself or some
of the steps in the progressions for constructs in the social-emotional development and
cognitive development domains (see Exhibit 7.2).
EAG K-3 Formative Assessment: Final Report September 2017 84
Revisions to training materials and recommendations We edited the training materials to make them more appropriate for general use. In addition, we
made some changes based on the field test findings.
Revisions to teacher and administrator training materials
Materials have been edited to remove references to the field test and to the technology platform
vendor who supported the field test. Professional development materials were repackaged into
thematic units so that the materials can be more easily adapted for use as states and districts
implement the assessment. We did not change the contents of the individual progressions. The
materials for training and ongoing support continue to include:
• Teacher training materials comprised of presentation slides, handouts, and hands-on
activities that cover the following topics: an overview of formative assessment and the K–3
Formative Assessment; an overview of the five domains covered in the assessment; a
review of the K–3 formative assessment process; an activity guide and related handouts to
accompany the review of the K–3 formative assessment process; and a guide for
customizing the training to state and/or local needs.
• Online modules for each construct, intended for self-study as teachers prepare to assess
each construct.
• Administrator training materials, including presentation slides and online modules that cover
the formative assessment process, data security, and supporting teachers.
Final teacher and administrator materials were reviewed and edited to ensure that field test and
state-specific content were not included. For example, references to state data collection
calendars that were part of the field test were removed. Training materials were also reviewed
and updated for clarity and consistency with changes made to assessment materials.
Recommendations regarding training
Based on teacher feedback regarding the training and technology, we recommend the following:
• Integrate additional practice opportunities with the technology into the training. Teachers mentioned at several points in their responses to survey and interview questions a
desire to have more practice with the technology. Technology practice could be easily
integrated with the training activities that are part of the K–3 Formative Assessment training
materials. For example, teachers could practice entering observation notes into a
technology platform while observing video content.
EAG K-3 Formative Assessment: Final Report September 2017 85
• Provide time between training and implementation for self-study and follow-up training to occur. Teachers mentioned “information overload” in terms of the content
provided during the training which was not surprising given that a lot of information was
presented in a short time. State and district staff responsible for developing and customizing
the training schedules and training content should consider ways in which additional support
can be provided to teachers both before and after the formal assessment window begins
without placing too much burden on teachers over a short time period. The online training
modules and the follow-up trainings did not appear to be useful to teachers as fewer than
half took advantage of them.
• Regional, district, or school staff responsible for supporting technology use in the schools should attend the technology portion of the training to understand the technology needs associated with the K–3 Formative Assessment. Intermittent or slow
access to the Internet and lack of available computers and/or tablets added to the time
teachers spent on collecting, uploading, and managing observation data. Mitigation of the
technology issues would reduce teacher burden.
Conclusion The field test demonstrated that the K–3 Formative Assessment can be implemented by
teachers as intended, provided that teachers have adequate training in the assessment content
and the formative assessment process, and have access to functioning technology that
supports the assessment. Most of the construct progressions performed well as measures of
children’s knowledge, skills, and abilities, both in terms of their sequencing and their sensitivity
to growth and development over time. The analysis of the evidence documented in the
technology platform showed that the constructs varied in how reliably teachers coded a level for
the child. Teacher feedback reflected this variation, in that fewer teachers felt confident about
where they placed children on the progressions of constructs that had lower reliability. Teachers
found some constructs more difficult than others to incorporate into their daily classroom
routines. Of particular importance for a formative assessment was that across constructs,
between 20 and 50 percent of teachers did not find a given construct useful in informing
instruction. This may be a characteristic of the assessment that needs to be improved or it could
indicate a need for more professional development around how to use the results of a formative
assessment to inform instruction. Overall, we are encouraged by the performance of most of the
constructs in a field test setting where most teachers had limited experience with the constructs
and formative assessment process. As teachers continue to work with the K–3 Formative
EAG K-3 Formative Assessment: Final Report September 2017 86
Assessment, we recommend additional research and professional development on the
construct progressions that presented some issues with their content and proved challenging to
implement. These constructs are those in the social-emotional development and cognitive
development domains, along with Perseverance, Vocabulary, and Gross Motor.
EAG K-3 Formative Assessment: Final Report September 2017 87
References American Educational Research Association, American Psychological Association, & National
Council on Measurement in Education. (2014). Standards for educational and
psychological testing. American Educational Research Association.
Bejar, I. I. (2008). Standard setting: What is it? Why is it important. R&D Connections, 7, 1–6.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Taylor & Francis
Group: London.
Ferrara, A. M., & Lambert, R. G. (2015). Findings from the 2014 North Carolina Kindergarten
Entry Formative Assessment pilot. CEME Technical Report No. 2015-16. Charlotte, NC:
The Center for Educational Measurement and Evaluation.
Gebhart, T., Partika, A., & Lin, V. (2017). Implications for long-term implementation: Findings
from the K-3 Formative Assessment Consortium field test data collection. Bethesda, MD:
Child Trends.
Hambleton, R. K. (2000). Response to Hays et al. and McHorney and Cohen: Emergence of
item response modeling in instrument development and data analysis. Medical Care,
38(9), II–60.
Linacre, J. M. (2004). Optimizing rating scale category effectiveness. In E. V. Smith Jr., & R. M.
Smith (Eds.), Introduction to Rasch measurement (pp. 258–278). Maple Grove, MN:
Journal of Applied Measurement Press.
Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of
performance on educational and occupational tests. Princeton, NJ: Educational Testing
Service.
Masters, G.M. (June 1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), pp.
149-174.
Mislevy, R. J., & Haertel, G. (2006). Implications of evidence-centered design for educational
testing. Educational Measurement: Issues and Practice, 25(4), 6–20.
North Carolina K-3 Assessment Think Tank (2013). Assessment for learning and development
in K-3: A report by the state assessment think tank. Raleigh, NC: Author.
EAG K-3 Formative Assessment: Final Report September 2017 88
Scott-Little, C., Kagan, S., Sumrall, T., & Fox, E., (2014). Common early learning and
development standards analysis for the North Carolina EAG Consortium – Summary
report. Greensboro, NC: Build Initiative.
Wright, B. D., & Linacre, J. M. (1992). Combining and splitting categories. Rasch Measurement
Transactions, 6(3), 233–235.
EAG K-3 Formative Assessment: Final Report September 2017 89
Appendices Appendix A ............................................................................................................................. A-1
Appendix B ............................................................................................................................. B-1
Appendix C ............................................................................................................................. C-1
Appendix D ............................................................................................................................. D-1
Appendix E ............................................................................................................................. E-1
Appendix A Evidence-Centered Design (ECD) Codesign Teams: Enhancements to the Construct Progressions Exhibit A.1. Changes made through the ECD co-design process in the NC construct progressions
EAG K-3 Formative Assessment: Final Report September 2017 A-1
Exhibit A.1. Changes made through the ECD co-design process in the NC construct progressions
Domain Original NC Constructs EAG K-3 Constructs Changes Made through ECD Co-Design Process
Engagement in Self-Selected Activities
Perseverance
• Combined aspects of engagement in self-selected activities and perseverance in assigned activities into one construct
• Removed references to strategies to not confuse task strategies with strategies to stay engaged
Perseverance in Assigned Activities
Object Counting Object Counting • Expanded the number of skills from 8 to 10 • Introduced the number of objects counted as a factor in
skill difficulty level (4-6, 6-10, 18-20)
Problem Solving* Problem Solving*
• Reduced number of skills from 13 to 10 • Combined steps that involve solving and attempting to
solve a problem to focus on how children problem solve rather than successful problem solution
• Eliminated step that involves child generating sophisticated problems for others to solve as not typically observed in elementary classrooms
Mathematical Patterns • Added construct to measure algebraic thinking
(recommended by Executive Committee) and developed progression
Book Orientation & Print Awareness
Book Orientation • Separated into two distinct constructs
Print Awareness
Letter Naming Letter Naming
• Shortened and simplified progression; reordered some steps
• Defined skills in terms of letters known (e.g., in name) rather than numbers of letters
• Clarified that child can identify letters by either naming the letter or selecting the letter when asked
Following Directions Following Directions • Simplified and shortened progression from 11 to 6 steps • Removed emphasis on compliance tasks and positional
words
Writing Writing
• Removed skill (imitating conventions) that does not have clear developmental sequence
• Pulled out demonstration of writing with purpose and audience as a separate skill
EAG K-3 Formative Assessment: Final Report September 2017 A-2
Domain Original NC Constructs EAG K-3 Constructs Changes Made through ECD Co-Design Process
Reading Comprehension: Monitoring Meaning Reading Comprehension
• Shortened progression from 9 to 8 steps • Dropped last step requiring child to use optimal self-
correction strategy as this would be difficult for teacher to judge
School-Related Vocabulary Vocabulary
• Simplified and shortened progression from 13 to 8 steps to remove levels within a skill (e.g., begins to use, sometimes uses, consistently uses)
• Removed need for teacher to observe use of words in multiple contexts which is difficult to track
Gross Motor Gross Motor • Extended progression beyond basic locomotion (walking, running, stairs) to include hopping, galloping, and skipping
Fine Motor Hand Dominance • Separated into 2 distinct constructs
• Simplified and shortened progression for grip & manipulation, combining last two skills as indistinguishable Grip & Manipulation
Midline Crossing Midline • No substantive changes other than name to clarify construct
Emotion Regulation
Emotion Regulation Strategies • Split into two distinct constructs to capture dual themes of
expressing and managing emotions • After split, gaps between skills were too large, so additional
skills were added to even out granularity between skills. • Named “Emotion Regulation Strategies” to focus teacher’s
attention on the strategies children can learn to help regulate their emotions
Emotion Expression
Emotional Literacy Emotional Literacy • Combined skills to reduce complexity of progression and
eliminate concerns about developmental sequence • Extended progression to be more inclusive of K-3rd grades.
*The Problem Solving construct as developed by NC and enhanced for the EAG Consortium includes skills that tap two domains: Cognitive Development and Approaches to Learning.
EAG K-3 Formative Assessment: Final Report September 2017 A-3
Appendix B Cognitive Labs Internal Report
EAG K-3 Formative Assessment: Final Report September 2017 B-1
1
Cognitive Labs Report
Prepared by SRI International
July 29, 2016
EAG K-3 Formative Assessment: Final Report September 2017 B-2
Table of Contents Section 1: What Was Done as Part of Cognitive Labs? 5
Teacher Participation 5 Table 1. Number of Teachers Who Participated 5 Table 2. Number and Percentage of Teachers Per Grade Level 5
Survey Completion 6 Table 3. Average Number of Surveys Signed Up for and Completed (in Spring and Fall 2015 Periods) 6
Table 4. Number of Surveys Completed (Partially or Fully) 7 Section 2: What Was Learned from Cognitive Labs? 8
Most Children Could Be Placed 8 Table 5. Number of Child Assessments and Number and Percentage in Which Children Could and Could Not Be Placed 8
Table 6. Number of Teachers Who Selected Each Reason Why Children Couldn’t Be Placed 9
Table 7. Number and Percentage of Teachers Who Selected Positive or Negative Confidence Levels Related to Placing Children 9
Most Teachers Found Progressions Clear and Relevant 9 Table 8. Number Teachers Who Did or Did Not Agree with Statements about Clarity and Relevance of Progression Components 10
Most Teachers Found Progressions Helpful for Planning Instruction and Next Steps for Children 10
Table 9. Number of Teachers Who Did or Did Not Agree with Statements about Helpfulness of Constructs and Whether Children Could Be Placed
10
Most Teachers Found Progressions Relevant to Different Grade Levels 11 Table 10. Number of Teachers Who Did or Did Not Agree with Statements about Whether K Children and Children at Their Grade Level Could Be Placed
11
Most Teachers Found Reading Comprehension, Writing, and Vocabulary Constructs Aligned with Their Requirements 11
Table 11. Number of Teachers Who Did or Did Not Agree with Statement about Alignment of Progressions with Requirements 12
Most Teachers Found They Could Gather Sufficient Info from Routine Activities and Had Suggested Materials 12
Table 12. Number of Teachers Who Did or Did Not Agree with Statements about Routine Activities and Suggested Materials 12
Not All Teachers Used Situations, But Those Who Did Found Them Effective 13
Table 13. Number of Teachers Who Did or Did Not Use Situations 13 Table 14. Of Teachers Who Did Not Use Situations, Number and Percentage Who Selected Each Reason for Not Using Situations 14
EAG K-3 Formative Assessment: Final Report September 2017 B-3
Table 15. Of Teachers Who Used Situations, Number and Percentage Who Found Them Effective 14
Not All Teachers Used Tasks, But Those Who Did Found Them Effective 14 Table 16. Number of Teachers Who Did or Did Not Use Tasks 15 Table 17. Of Teachers Who Did Not Use Tasks, Number Who Selected Each Reason for Not Using Tasks 15
Table 18. Of Teachers Who Used Tasks, Number Who Found Them Effective 15
Teachers Provided Information on the Lowest and Highest Placement of Children That They Assessed 16
Figure 1. Book Orientation 16 Figure 2. Crossing Midline 16 Figure 3. Emotion Expression 17 Figure 4. Emotion Regulation 17 Figure 5. Emotional Literacy 18 Figure 6. Following Directions 18 Figure 7. Grip & Manipulation 19 Figure 8. Gross Motor 19 Figure 9. Hand Dominance 20 Figure 10. Letter Naming 20 Figure 11. Object Counting 21 Figure 12. Perseverance 21 Figure 13. Print Awareness 22 Figure 14. Problem Solving 22 Figure 15. Reading Comprehension 23 Figure 16. Vocabulary 23 Figure 17. Writing 24
Teachers Had Specific Feedback About Constructs 25 Book Orientation 25 Crossing Midline 25 Across Emotion Constructs 25 Emotion Expression 26 Emotion Regulation 26 Emotional Literacy 26 Following Directions 27 Grip & Manipulation 27 Gross Motor 28 Hand Dominance 28 Letter Naming 28 Object Counting 29 Perseverance 29 Print Awareness 29 Problem Solving 29 Reading Comprehension 30
EAG K-3 Formative Assessment: Final Report September 2017 B-4
Vocabulary 30 Writing 31
Section 3: What Changes Were Made in Response to Cognitive Labs Feedback (As Well as Pilot Feedback)? 32
Most Constructs Remained the Same 32 Some Constructs Had Changes to the Assessment Materials to Clarify Specific Points 32
Emotion Expression 32 Emotion Regulation 32 Following Directions 32 Gross Motor 32 Problem Solving 32 Reading Comprehension 33 Writing 33
Two Constructs Had Changes to the Progression 33 Emotional Literacy 33 Vocabulary 33
EAG K-3 Formative Assessment: Final Report September 2017 B-5
Section 1: What Was Done as Part of Cognitive Labs?
Teacher Participation
The purpose of cognitive labs was to give teachers an opportunity to try out individual constructs materials with children and provide construct-specific feedback to SRI. Cognitive Labs occurred in three time periods. The first round of Cognitive Labs occurred in the spring of 2015, the second round occurred in the fall of 2015, and the third round occurred in the spring of 2016. In the spring of 2015, 61 teachers in four states participated. In the fall of 2015, 56 teachers in four states participated. In the spring of 2016, four teachers in one state participated. See Table 1 for a state-by-state breakdown of participants. Table 1. Number of Teachers Who Participated
State Total Spring 2015 Fall 2015 Spring 2016 Arizona 26* 10 18 0
Iowa 14** 10 7 0 Maine 34 7 27 0
Rhode Island 38 34 4 0 North Carolina 4 0 0 4
Total 116 61 56 4 * There were two teachers in both spring and fall cohorts. ** There were three teachers in both spring and fall cohorts.
Of the 116 teachers who participated in cognitive labs, 37.93% were kindergarten teachers, 27.68% were first grade teachers, 14.29% were second grade teachers, 11.61% were third grade teachers, and 9.82% were multi-grade teachers. One teacher’s grade level is not known. See Table 2 for the number of teachers at each grade level. Table 2. Number and Percentage of Teachers Per Grade Level
Grade Number of teachers
Percentage of teachers
K 44 37.93% 1 31 27.68% 2 16 14.29% 3 13 11.61%
Multiage 11 9.82% Unknown 1 0.89%
Total 116
EAG K-3 Formative Assessment: Final Report September 2017 B-6
Survey Completion
In the spring and fall 2015 cognitive labs, teachers were typically given a choice of constructs to try out. If a teacher did not select a construct, one was assigned. Teachers were then given three weeks to use the construct materials (progression, assessment means form). In the spring 2016 cognitive labs, teachers were recruited to give feedback only on the problem solving construct materials. Across the spring and fall 2015 cognitive labs, teachers signed up for an average of 2.89 surveys and completed an average of 1.91 surveys (see Table 3).
Across all three cognitive labs periods, the teachers completed a total of 267 surveys, distributed unevenly across constructs (see Table 4). The constructs surveys were taken in the following order of frequency: grip & manipulation, emotion expression, emotion regulation, reading comprehension, following directions, gross motor, vocabulary, writing, crossing midline, perseverance, emotional literacy, letter naming, book orientation, print awareness, object counting, hand dominance, and problem solving. Table 3. Average Number of Surveys Signed Up for and Completed (in Spring and Fall 2015 Periods)
Surveys signed up for Surveys completed
Mean 2.89 1.91 Range 1-4 0-4
EAG K-3 Formative Assessment: Final Report September 2017 B-7
Table 4. Number of Surveys Completed (Partially or Fully)
Construct Number of surveys partially or fully completed Spring 2015 Fall 2015 Spring 2016 Total
Book orientation* 7 0 0 7 Crossing midline 1 11 0 12 Emotion expression** 24 9 0 33 Emotion regulation** 24 3 0 27 Emotional literacy 0 10 0 10 Following directions 0 19 0 19 Grip & manipulation 43 0 0 43 Gross motor 0 19 0 19 Hand dominance 3 2 0 5 Letter naming 2 7 0 9 Object counting 6 0 0 6 Perseverance 0 11 0 11 Print awareness* 7 0 0 7 Problem solving 0 0 4 4 Reading comprehension 0 20 0 20 Vocabulary 0 18 0 18 Writing 0 17 0 17 Total 117 146 4 267
* Book orientation and print awareness constructs were combined into one survey in the spring but were separated for the purposes of analysis. ** Emotion expression and emotion regulation constructs were combined into one survey in the spring but were separated for the purposes of analysis.
EAG K-3 Formative Assessment: Final Report September 2017 B-8
Section 2: What Was Learned from Cognitive Labs? Most Children Could Be Placed
In the cognitive labs, teachers were asked to try out each construct with 3-5 children.
Teachers generally followed that guideline. Across constructs, there were 982 child assessments. Because teachers did not have to obtain child consent, they tried out each construct with whomever they desired. Some children could have been assessed for multiple constructs, so the number of unique children assessed is not known. The majority of these child assessments allowed for the placement of children on the progression (97.61%). See Table 5 for a breakdown by grade.
When a child assessment did not result in a child being placed on the progression, teachers reported why they could not place the child. Teachers could choose from one or more of the five following reasons: (1) child was above the highest skill level, (2) child was below the lowest skill level, (3) the teacher was not able to collect enough data, (4) the data collected was too inconsistent for the teacher to decide on a placement, and (5) other. The frequency with which each reason was reported is presented in Table 6 and is broken down by grade level.
Teachers also reported how confidence they felt placing children on the progression. They chose between very confident, fairly confident, confident for some children, and not confident. The frequency with which positive or negative confidence levels were selected is reported in Table 7 and is broken down by grade level.
Table 5. Number of Child Assessments and Number and Percentage in Which Children Could and Could Not Be Placed
Number of child assessments
Assessments in which children could be
placed
Assessments in which children could not be
placed Across grades 982 961 (97.61%) 24 (2.39%)
K 363 363 (97.84%) 8 (2.16%) 1 305 296 (98.71%) 4 (1.29%) 2 124 124 (96.12%) 5 (3.88%) 3 128 128 (95.52%) 6 (4.48%)
Multi 62 50 (98.41%) 1 (1.59%)
EAG K-3 Formative Assessment: Final Report September 2017 B-9
Table 6. Number of Teachers Who Selected Each Reason Why Children Couldn’t Be Placed
Above
highest skill level
Below lowest skill
level
Not able to collect
enough data
Data too inconsistent
to decide Other
Across grades 4 3 2 4 3 K 2 1 1 2 1 1 0 0 0 2 2 2 1 0 1 0 0 3 1 1 0 0 0
Multi 0 1 0 0 0 Table 7. Number and Percentage of Teachers Who Selected Positive or Negative Confidence Levels Related to Placing Children
Fairly or Very Confident
Confident for some or not confident
Across grades 235 (89.69%) 27 (10.31%) K 93 (87.74%) 13 (12.26%) 1 65 (89.04%) 8 (10.96%) 2 30 (96.77%) 1 (3.23%) 3 35 (94.59%) 2 (5.41%)
Multi 12 (80.00%) 3 (20.00%) Most Teachers Found Progressions Clear and Relevant
Teachers answered several questions about the clarity, helpfulness, and relevance of the construct progressions components. Across construct progressions, teachers reported that the wording was clear (98.08%), the understandings were helpful (99.45%), and the performance descriptors and examples were helpful (98.47%). Regarding the performance descriptors and examples, a smaller percentage of teachers found them relevant (89.66%). See Table 8 for a breakdown by grade.
EAG K-3 Formative Assessment: Final Report September 2017 B-10
Table 8. Number of Teachers Who Did or Did Not Agree with Statements about Clarity and Relevance of Progression Components
Clear wording Helpful understanding(s)*
Helpful performance
descriptors and examples
Relevant performance
descriptors and examples
Agree Disagree Agree Disagree Agree Disagree Agree Disagree Across grades 256 5 181 1 257 4 234 27
K 104 2 64 0 105 1 98 8 1 72 1 49 0 71 2 65 8 2 24 1 30 1 26 5 22 9 3 37 0 32 0 37 0 32 5
Multi 14 0 12 0 14 0 13 1 * There were no understandings for the motor constructs. Most Teachers Found Progressions Helpful for Planning Instruction and Next Steps for Children
Teachers reported whether they found the construct progressions helpful for planning instruction. 83.91% of teachers reported that they were. For some constructs, teachers also reported whether the construct progressions were helpful for planning next steps for children, and 90.28% reported that they were. See Table 9 for a breakdown by grade. Table 9. Number of Teachers Who Did or Did Not Agree with Statements about Helpfulness of Constructs for Planning Instruction and Next Steps
Helpful for planning instruction
Helpful for planning next steps
Agree Disagree Agree Disagree Across grades 219 42 130 14
K 90 16 14 0 1 59 14 55 6 2 21 5 21 0 3 34 3 31 2
Multi 14 0 9 1
EAG K-3 Formative Assessment: Final Report September 2017 B-11
Most Teachers Found Progressions Relevant to Different Grade Levels
Teachers reported whether they thought most kindergarten children could be placed, and 99.53% agreed. For some constructs, teachers also reported whether children at the teacher’s particular grade level could be placed, and 97.22% agreed. See Table 10 for a breakdown by grade. Table 10. Number of Teachers Who Did or Did Not Agree with Statements about Whether K Children and Children at Their Grade Level Could Be Placed
K children could be placed
My grade level could be placed
Agree Disagree Agree Disagree Across grades 208 1 140 4
K 102 1 14 0 1 57 0 61 0 2 24 2 9 3 3 16 0 31 2
Multi 12 0 10 0 Most Teachers Found Reading Comprehension, Writing, and Vocabulary Constructs Aligned with Their Requirements
Teachers were queried as to whether they felt the reading comprehension, writing, and vocabulary constructs were aligned with what the teachers were required to teach. For these constructs, 81.13% of teachers reported that they were. See Table 11 for a breakdown by grade.
EAG K-3 Formative Assessment: Final Report September 2017 B-12
Table 11. Number of Teachers Who Did or Did Not Agree with Statement about Alignment of Progressions with Requirements
Aligned with my requirements*
Agree Disagree Across grades 43 10
K 1 0 1 15 4 2 28 3 3 15 2
Multi 3 1 * This question was asked only in the surveys for reading comprehension, writing, and vocabulary. Most Teachers Found They Could Gather Sufficient Info from Routine Activities and Had Suggested Materials
Teachers reported whether they could gather enough information from routine activities, and 92.34% of teachers said they could. For some constructs, teachers reported whether they had the suggested materials, and 96.97% of teachers did. See Table 12 for a breakdown by grade. Table 12. Number of Teachers Who Did or Did Not Agree with Statements about Routine Activities and Suggested Materials
Enough info from routine activities
Had suggested materials
Agree Disagree Agree Disagree Across grades 241 20 192 6
K 98 8 64 2 1 66 7 63 0 2 24 1 0 0 3 36 1 29 3
Multi 13 1 12 0
EAG K-3 Formative Assessment: Final Report September 2017 B-13
Not All Teachers Used Situations, But Those Who Did Found Them Effective
Situations were provided for some of the constructs. Eleven constructs had one Situation that was used in Cognitive Labs (Book Orientation, Crossing Midline, Following Directions, Grip & Manipulation, Gross Motor, Hand Dominance, Letter Naming, Object Counting, Print Awareness, Reading Comprehension, Writing); two constructs had three Situations (Emotional Literacy and Vocabulary); and three constructs had no Situations (Emotion Expression, Emotion Regulation, and Perseverance). Teachers reported whether they used the situation(s) (which was not required). Not all teachers used situations (37.55%). See Table 13 for a breakdown by grade.
Teachers who did not use the situation(s) reported whey they did not. The reasons varied, but the majority of times, teachers did not use the situation(s) because they were able to collect enough information through observation. See Table 14 for more information. The teachers who did use the situation(s) reported that they found them appropriate (98.91%) and easy to implement (96.74%). They also found the evidence from the situation(s) helpful (98.91%). See Table 15 for a breakdown by grade. Table 13. Number of Teachers Who Did or Did Not Use Situations
Used Situation Yes No
Across grades 92 153
K 30 45 1 31 46 2 12 20 3 15 27
Multi 4 15
EAG K-3 Formative Assessment: Final Report September 2017 B-14
Table 14. Of Teachers Who Did Not Use Situations, Number Who Selected Each Reason for Not Using Situations
Why wasn’t used
Able to obtain enough info
through observation
Not a good activity
Confusing or hard to use
Not appropriate
for this grade level
Other
Across grades 123 7 1 6 19
K 41 2 0 0 3 1 38 2 0 1 5 2 13 1 1 0 5 3 20 0 0 4 4
Multi 011 7 0 1 2 Table 15. Of Teachers Who Used Situations, Number Who Found Them Effective
Situation appropriate
Situation easy to implement
Situation evidence helpful
Agree Disagree Agree Disagree Agree Disagree Across grades 91 1 89 3 91 1
K 30 0 30 0 29 1 1 31 0 30 1 31 0 2 12 0 11 1 12 0 3 15 0 14 1 15 0
Multi 3 1 4 0 4 0 Not All Teachers Used Tasks, But Those Who Did Found Them Effective
Two constructs had a task that was used in Cognitive Labs (Crossing Midline and Gross Motor). Teachers reported whether they used the task (which was not required). Not all teachers used the task (32.26%). See Table 16 for a breakdown by grade.
Teachers who did not use the task reported whey they did not. The reasons varied, but the majority of times, teachers did not use the task because they were able to collect enough information through observation. See Table 17 for more information. The teachers who did use the task reported that they found them appropriate (100%) and easy to implement (100%). They also found the evidence from the situation(s) helpful (100%). See Table 18 for a breakdown by grade.
EAG K-3 Formative Assessment: Final Report September 2017 B-15
Table 16. Number of Teachers Who Did or Did Not Use Tasks
Used Task Yes No
Across grades 10 21
K 2 2 1 6 14 2 1 2 3 1 1
Multi n/a n/a Table 17. Of Teachers Who Did Not Use Tasks, Number Who Selected Each Reason for Not Using Tasks
Why wasn’t used
Able to obtain enough info
through observation
Not a good activity
Confusing or hard to use
Not appropriate
for this grade level
Other
Across grades 18 2 0 0 2
K 1 0 0 0 1 1 13 2 0 0 0 2 2 0 0 0 0 3 2 0 0 0 1
Multi n/a n/a n/a n/a n/a Table 18. Of Teachers Who Used Tasks, Number Who Found Them Effective
Task appropriate Task easy to implement
Task evidence helpful
Agree Disagree Agree Disagree Agree Disagree Across grades 10 0 10 0 10 0
K 2 0 2 0 2 0 1 6 0 6 0 6 0 2 1 0 1 0 1 0 3 1 0 1 0 1 0
Multi n/a n/a n/a n/a n/a n/a
EAG K-3 Formative Assessment: Final Report September 2017 B-16
Teachers Provided Information on the Lowest and Highest Placement of Children That They Assessed
Teachers reported the lowest and highest skill level at which they placed the children that they assessed. These data are reported in Figures 1-17, broken down by grade, with the number of teachers at each grade level reported in parentheses. Figure 1. Book Orientation
Figure 2. Crossing Midline
EAG K-3 Formative Assessment: Final Report September 2017 B-17
Figure 3. Emotion Expression
Figure 4. Emotion Regulation
EAG K-3 Formative Assessment: Final Report September 2017 B-18
Figure 5. Emotional Literacy
Figure 6. Following Directions
EAG K-3 Formative Assessment: Final Report September 2017 B-19
Figure 7. Grip & Manipulation
Figure 8. Gross Motor
EAG K-3 Formative Assessment: Final Report September 2017 B-20
Figure 9. Hand Dominance
Figure 10. Letter Naming
EAG K-3 Formative Assessment: Final Report September 2017 B-21
Figure 11. Object Counting
Figure 12. Perseverance
EAG K-3 Formative Assessment: Final Report September 2017 B-22
Figure 13. Print Awareness
Figure 14. Problem Solving
EAG K-3 Formative Assessment: Final Report September 2017 B-23
Figure 15. Reading Comprehension
Figure 16. Vocabulary
EAG K-3 Formative Assessment: Final Report September 2017 B-24
Figure 17. Writing
EAG K-3 Formative Assessment: Final Report September 2017 B-25
Teachers Had Specific Feedback About Constructs Teachers provided specific feedback about constructs on the cognitive lab surveys. Sample items of feedback are presented for each construct.
Book Orientation
• How used this to inform instruction: In some ways. The information I am gathering is information that I would have gathered even without this pilot. i.e. book handling skills, print awareness, etc.
• I felt as if both of these professions were too easy for my classroom and although I only "assessed" 3 students, all of my students would fall of the highest profession. Especially at this time of the year, all kindergarteners should know how to easily handle their reading books. If a child cannot do this it is usually handled on a one-to-one basis. Again, I think this would be great information to obtain in a preschool setting, not so much a middle/end of the year kindergarten assessment.
• Examples were relevant although this progression was way to easy for all of the students in my class.
Crossing Midline
• Easy to follow, understandable, teacher friendly. • Examples helpful--used for activity ideas.
Across Emotion Constructs
• I think this is very helpful. I do believe that the special needs department would be glad that these assessments were done because the information provided helps to show interventions and observations that may have been done to fix the problem the student is having with emotion in class.
• I continue to be impressed with the clear and concise language used in the materials. I was especially impressed with the Emotion Regulation language as this is such a critical area in a kindergarten classroom that can negatively or positively impact everything the teacher is trying to accomplish on a daily basis. With today's society being mainly that of both parents working, I have found that many children enter my classroom ill equipped to handle their emotions due to a lack of one-on-one time. The strategies on the construct progression allow for even a brand new teacher to understand how to handle to varied emotions of a kindergarten student who can vary in age from 4-6 years old and therefore be in very different places emotionally.
• It might be helpful to provide teachers with a list of possible strategies that the students might be using (Using Strategies) I also feel that you can not determine where a a child is placed based off of one observation. That child may be having a
EAG K-3 Formative Assessment: Final Report September 2017 B-26
bad day or a situation is a frustrating one. You need a variety of situations to get a full understanding of the child's regulation of emotions (Appropriate Expression of Emotions)
Emotion Expression
• I selected three different students with different emotional needs and I was surprised that they all fell on level A. That provided me with a lot of information.
• I have never changed my lesson plans if a students behavior was not where it should be. I have excluded students from fun things.
• Again, as I thought for the fine motor construct, it is a great resource for those few children whose behaviors need further classroom supports, but way too overhwelming to conceive of doing for the whole class.
• Made me realize I need more tools to help kids work on self-regulation. • Would be good to do at the beginning of the year with check-in activities with kids.
Emotion Regulation
• I continue to be impressed with the clear and concise language used in the materials. I was especially impressed with the Emotion Regulation language as this is such a critical area in a kindergarten classroom that can negatively or positively impact everything the teacher is trying to accomplish on a daily basis. With today's society being mainly that of both parents working, I have found that many children enter my classroom ill equipped to handle their emotions due to a lack of one-on-one time. The strategies on the construct progression allow for even a brand new teacher to understand how to handle to varied emotions of a kindergarten student who can vary in age from 4-6 years old and therefore be in very different places emotionally.
• For most students, this information is known in teacher's head based upon everyday observations and interactions and does not need to be placed on paper, just adds more work. Good resource for RTI purposes or parent conferences when difficulties arise.
Emotional Literacy
• I did not look at the situations ahead of time, but as I went back through to offer the feedback on this survey, the situations are exactly how I taught and assessed. The situations are spot on to how I feel most teachers would approach this topic.
• There is a lot of information for this construct. By this, I mean, I was surprised by the amount of skill levels. I was a little overwhelmed at first when I read through it.
EAG K-3 Formative Assessment: Final Report September 2017 B-27
Following Directions
• I learned a lot about one little boy. Prior to these observations, I thought he was going to have a very difficult time following a 2 or 3 step direction. However, he was able to follow them all without support which surprised me.
• I teach grade 3 and I feel that at this age my students should be able to complete the following directions construct. I feel most of the constructs would be great and very beneficial to to students in grades 1& 2, but by grade 3 these constructs should be the norm.
• This was great because I did not know how students really only listen to the last thing out of my mouth for following directions.
• The Following Directions progression really helped me to take a closer look at some other students in my class. I realized that some students that I thought could easily follow multi-step directions could not.
Grip & Manipulation
• The descriptors were helpful because they used real day-to-day scenarios that we use in our classroom (ex: holding a paintbrush with two fingers using precise strokes)
• The majority of students come to school able to manipulate pencils, scissors, etc... the few that do not, are provided with instruction on proper use and interventions when needed to correct grips. This is so individualized having a standardized progression is not necessarily needed.
• The information gathered through this construct was useful, but the anecdotal notes on this did not help me. At the beginning of the year, I observe students and work with them through their struggles…
• I think that the information that could be gleaned from this assessment is valuable and vital information, however, if school districts are unwilling to allow time in the classroom (especially kindergarten) when students are freely choosing activities, and those that would help exhibit the constructs, then it is going to be extremely difficult to implement. In my classroom, I am working with a small group or individuals during reading and writing time, so it is possible at times to observe students writing with a pencil, or coloring, but not for an extended period of time. Also, we rarely cut, and never paint.
• Can one child score more than one "Skill" score? I feel the rubric isn't complete yet. For example: a child might have an amazing pencil grip but struggle with scissors.
• If Kindergarten teachers would follow this it would help the first grade teachers so much. It it a harder to start this in 1st grade. They all ready have their habits and it is harder to change.
• The Construct Progression is so clearly and concisely written that even a new teacher could easily follow it. There can be such a large and varied skill base with fine motor skills in kindergarten students. Even with the higher academically
EAG K-3 Formative Assessment: Final Report September 2017 B-28
developed students can be low on their skill level for fine motor tasks. I have found it both difficult and challenging to break students of poor fine motor skill habits that they have entered kindergarten with. I found this to be very valuable to my teaching and I look forward to having the skill construct to use at the start of my next school year as it will be a valuable tool to use when discussing a child's fine motor progression with parents.
• It was easy to observe through our daily activities. I like that it doesn't take time away from our teaching time like other assessments do.
• I do feel that more training on the various grasps would have been helpful. I did use the materials (pictures & descriptions) given, but I was uncertain as to the proper name for each hold of the pencil as some of my students had different holds than the pictures. I did not feel 100% accurate at giving a clear label.
Gross Motor
• Most of my third grade students are too advanced for this construct • I needed to provide specific situations in order to observe skipping and galloping. I
could easily assess running and walking while at recess • I think it would also be beneficial to have other motor skills included in the "skills
documentation lists", like bouncing or catching a ball. Many young children have trouble with this skill, unless they are involved in sports outside of school.
• Liked being able to see where my kids were physically. • Like that it's about the whole child. • A lot of kids didn't know what gallop meant.
Hand Dominance
• I feel like this construct is established before they come to kindergarten, there is not a big need for direct instruction. The majority of my kids are already on level C, the couple on level B needed a couple reminders to stabilize, and the one friend on A is being evaluated for special needs.
• students at this age are still trying to decide their dominate hand • Probably more beneficial to prek.
Letter Naming
• Letter naming is such a huge part of the kindergarten curriculum and kinders are assessed to excess on this particular skill. This was quick and easy to apply to the construct since it is something done frequently already.
• We work with letter naming from day 1 in Kindergarten so this is something that we would plan for without knowledge of progression.
• Excited because this assessment is really developmental.
EAG K-3 Formative Assessment: Final Report September 2017 B-29
Object Counting
• Using to inform instruction: For example ,a student was able to sequence and count, but after looking at the numbers observation, the student still had difficulty with conservation/cardinality and was given some more opportunities to practice those skills as not to have a gap later on.
• I would like to see the content/progression become a little more in depth for 3rd grade core skills.
• It is good for the beginning and middle of Kindergarten. At the end of Kindergarten I found that my students were at the end of object counting.
• The literacy and math assessments were very repetitive with what I already do. They did not offer me new information on my students.
Perseverance
• I would know how to do this without the progressions • I observed the children during 3 different activities. One was an easy math fact
paper, the second was a moderately challenging reading activi ty, and the third a very challenging math activity. It was interesting to note that my more advanced students did better on the challenging activity than the rote math facts, scoring at least one skill level better. My less advanced students did about the same on alll three activities. It was a very interesting construct to use. I was over all pleased with how well most of my students did. I found the "watch for" notes on the documentation form very helpful. Thank you for the opportunity to work with these constucts. I have found them to be very valuable and I will add them to my "tool box."
• Perseverance became a classroom goal. Life long lessons. • Got a lot out of this. Will play in the back of my mind when I work with students.
Print Awareness
• I felt as if both of these professions were too easy for my classroom and although I only "assessed" 3 students, all of my students would fall of the highest profession. Especially at this time of the year, all kindergarteners should know how to easily handle their reading books. If a child cannot do this it is usually handled on a one-to-one basis. Again, I think this would be great information to obtain in a preschool setting, not so much a middle/end of the year kindergarten assessment.
Problem Solving
• Skills I and J could use more delineation between the two. Other domains/progressions
are integral parts of the problem solving process. I understand that the K-3 assessment process is intended to look at the whole child and covers perseverance and
EAG K-3 Formative Assessment: Final Report September 2017 B-30
collaboration in other components but I felt they needed to be addressed within the problem-solving section as well.
• One of the biggest focuses in my classroom as been to teach my children to become problem solvers. What I have noticed through this process is that a child's ability to problem solve varies greatly from one academic problem to the next depending on many variables such as interest, content, mood, engagement, etc. I collected evidence in 3 different content settings - writing, math and a STEM station. One of my students varied tremendously on the progression because he dislikes anything that requires writing so therefore he puts forth very little effort and shuts down easily. He is middle of the road in math so he placed a little higher but he a very scientific mind and is a kinesthetic learner so he placed the highest on the building a bridge/ raft activity. It almost seems as you need a problem solving construct for each academic area.
• Sometimes it was hard to try to figure out what kids were thinking when they weren’t verbalizing it themselves
Reading Comprehension
• I utilize our curriculums' reading assessments to plan my instruction • I really enjoyed working with this activity. After I did the 6 for the study, I put the
rest of the class on the scale. It was easy to use and I was able to easily put students on the progression. The difference between the skill levels was well defined and skills needed to move up the progression was easier than the writing one. I had the students read a story orally and tested them and then had them read and answer question after silently reading a similar story. I was interested to find that some did better orally and some did better silently. Good activity!
• This is a great formative assessment tool for measuring how students are doing with reading strategies that are being taught in the classroom and where they are at in terms of using them to improve understanding of reading independently. It also gave great insight into which strategies are most effective for children of this age.
• I feel that I gather the same if not more information through what I am already doing.
• I am the Title 1 teacher at my school, so most of my work is related to reading fluency and comprehension. I found the progression easy to use and accurate as to what I see in student's reading development.
Vocabulary
• It's hard to connect the curriculum we are expected to teach to the progressions. I think it's a different way to look at assessment. We currently assess if a child understands the vocabulary we teach, not necessarily where they are on a progression.
• Vocabulary instruction is something I am already focusing on in my instruction. The skills don't always match what my students can do or are working on. I also have
EAG K-3 Formative Assessment: Final Report September 2017 B-31
other forms of assessment that I am already doing that provide more information and data.
• It was helpful to see where the children fell in terms of understanding of vocabulary; however feel like I need more info. to actually decide what to teach next, where it fits in to CCSS, etc.
• Most 3rd grade students are too advanced for this progression. • Had to set up specific situations, word sorts, writing to measure this progression
[Regarding gathering info from routine activities] • I like the clarity of the progression and found the details of the progression
(understanding, skills, performance descriptors and examples) they really helped to fine tune what I know about a child's language and vocabulary and helped me place them accurately (I think!) on the progression.
• this weeks words were related to a story that was a play... rehearsal, lines, characters etc. I used the word lines for the writing situation.. (get in a line, draw a line and learn your lines) The construct made me more aware of making sure I use the weekly vocabulary words in teaching and having the children use them..not just telling them what they mean!
• This was a helpful construct tool. I was able to share the information with parents at conference time. It gave parents some insight as to what would be helpful at home when giving directions to their child. Also, by making thoughtful observations on where some students were on the scale led to more thoughtful direction planning and implementing.
Writing
• In 3rd grade we write a lot of narrative and informational text and I felt like the performance descriptors weren't all that relevant to the genre of writing we do.
• Our curriculum, Common Core, expects all children to be able to do everything on the progression at all grade levels - i.e. revise own writing.
• I feel like the expectations for my students are more complex than what this construct outlines.
• Writing instruction isn't linear like this - I have students who can revise, but not be genre specific. Also, a student could be very purposeful in their writing, but not be able to form words.
• I liked the organization of using the progression. It blended well with the writing project and conferencing that I was doing with this second grade group. It didn't feel strange or like extra work. As we use a Common Core aligned writing progression, I did find a couple of things that were slightly different, but not so much so that it was a distraction.
• More information on how to move students to the next level would be great. As I have a large class,(22) most of the work except students selection was done in small group.
EAG K-3 Formative Assessment: Final Report September 2017 B-32
• It was very difficult not to do spelling issues. We have been taught to take a moment to do a lesson taken from their writing. This gives a one-on-one time to teach that individual student.
EAG K-3 Formative Assessment: Final Report September 2017 B-33
Section 3: What Changes Were Made in Response to Cognitive Labs Feedback (As Well as Pilot Feedback)?
Feedback from cognitive labs and the pilot was used to evaluate the construct
progressions and assessment materials to determine if changes were needed. Most constructs remained the same (N = 8), some constructs had changes to the assessment materials to clarify specific points (N = 7), and two constructs had changes to the progression. Most Constructs Remained the Same
The following construct progressions and assessment materials remained the same (except for minor formatting changes): book orientation, crossing midline, grip & manipulation, hand dominance, letter naming, object counting, perseverance, and print awareness. Some Constructs Had Changes to the Assessment Materials to Clarify Specific Points
The following construct progressions had changes to the assessment materials based on feedback from teachers: emotion expression, emotion regulation, following directions, gross motor, problem solving, reading comprehension, and writing. The changes made are presented for each construct.
Emotion expression. Additional background information was added to the progression document. The assessment means form was modified to include additional observation instructions related to emotion-eliciting events.
Emotion regulation. Additional background information was added to the progression document. The assessment means form was modified to include additional information on children's use of emotion regulation strategies, additional examples of strategies and information about strategies, expanded teacher instructions and information around "hard-to-assess" children, and revised skill-level teacher instructions.
Following directions. The assessment means form was modified to include additional information about eliciting a child’s attention before providing a direction, to avoid the need to repeat directions, since that is considered a support.
Gross motor. The assessment means form was modified to include additional guidance on what to do when a child can skip but not gallop. Additional guidance was also added about modeling skills when children do not know what terms such as “gallop” mean.
Problem solving. The assessment means form was modified to clarify the need to observe across content areas to determine consistent performance; importance of asking open-ended, non-leading questions; and scope of academic problem.
EAG K-3 Formative Assessment: Final Report September 2017 B-34
Reading comprehension. The assessment means form was modified by revising the "Specific Observation Instructions" to include information about skills being cumulative and using text slightly above the child's reading level. Writing. Additional background information was added to the progression document. A minor grammatical error in Skill G was corrected in all documents.
Two Constructs Had Changes to the Progressions
The following construct progressions had changes to the construct progression and assessment materials based on feedback from teachers: emotional literacy and vocabulary. The changes made are presented for each construct.
Emotional literacy. The skills were reduced and revised (from 14 to 9 skills), which necessitated also changing performance descriptors and examples. Additional background information was added to the progression document. The assessment means form was modified to include additional information about placing children on the progression. Vocabulary. Skills B and C were combined which also necessitated changing performance descriptors and examples. One of the examples for Skill D (new Skill C) was revised.
EAG K-3 Formative Assessment: Final Report September 2017 B-35
Appendix C Pilot Test Internal Report EAG K-3 Formative Assessment Pilot Study Report
EAG K-3 Formative Assessment: Final Report September 2017 C-1
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
K-3 Formative Assessment: 2015-16 Pilot Study
Sara Thayer and Suzanne Raber
SRI Education
Author Note
This brief is an internal report of pilot activities conducted during the 2015-16 school year as
part of the Enhanced Assessment Grant awarded to the North Carolina Department of Public
Instruction and a consortium of ten states.
EAG K-3 Formative Assessment: Final Report September 2017 C-2
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Executive Summary The full K-3 Formative Assessment was piloted with approximately 80 K–3 teachers and 800
children in four states (Arizona, Iowa, Maine, Rhode Island) during the 2015–16 school year.
Each teacher assessed a sample of 10 children on all 17 constructs.1 Teachers’ use of the
assessment was supported by a technology platform. The teachers entered evidence (such as
anecdotal notes, photographs, or videos) from their observations for each construct and child in
the sample. They then determined where on the progression each piece of evidence fell
(preliminary learning status) and, across multiple pieces of evidence over multiple observations,
where to place the child on the progression (status summary). All the assessment data
(evidence, learning statuses, status summaries) could be entered into the platform for easy
review, reporting, and analysis. This process was further supported by an application that
allowed teachers to upload directly to the platform evidence (their notes, photos, and videos)
from a tablet or smart phone to assist in selecting a status summary.
Teachers provided feedback on the manageability and usability of the assessment to inform
instruction through online surveys, focus groups, and interviews. They appreciated the
assessment’s focus on the whole child and coverage of multiple domains of learning and
development. They also appreciated being able to store and retrieve the evidence they collected
in the technology platform. Teachers’ other feedback and the response to it were as follows:
• The assessment process was time consuming, and documenting and interpreting multiple
pieces of evidence to determine where a child fell on each progression was cumbersome.
Consequently, the field test training was modified to further demonstrate how formative
assessment observations can fit into daily instructional routines.
• The social-emotional constructs were particularly difficult to assess. Accordingly, the
Emotional Literacy progression was simplified, the assessment materials for Emotion
Expression and Emotional Regulation Strategies were clarified, and more guidance on how
to assess these constructs was added.
• The Vocabulary construct was challenging to assess. Thus, two progression skill levels that
were difficult to observe in the classroom were combined (using vocabulary in different
contexts) and the assessment materials clarified to emphasize that the construct is about
vocabulary concepts rather than vocabulary word lists.
1 Mathematical Patterns was not included because it was developed during the pilot year.
EAG K-3 Formative Assessment: Final Report September 2017 C-3
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
K-3 Formative Assessment: 2015-16 Pilot Study The pilot study was designed to gather information about the K-3 Formative Assessment from
teachers in four states (Arizona, Iowa, Maine and Rhode Island) who used the full complement
of the 17 constructs developed thus far to assess 10 children in each of their classes. Input
from the teachers on the training, the content of the assessment and it use and manageability,
was then used to make modifications in the assessment content and process for a larger scale
field test conducted in the 2016-17 school year.
Methodology Each of the four consortium member states that had volunteered to be part of the pilot study
recruited K-3 teachers who were interested in using formative assessment in their classrooms,
at K entry and then throughout the school year in grades K-3. Kindergarten teachers were
recruited to participate in the fall 2015 pilot study and then additional K-3 teachers were asked
to pilot the assessment in the winter/spring of the 2015-16 school year. Pilot teachers
participated in a 2-day training prior to the fall and winter data collection periods, conducted by
staff from SRI Education who led the assessment enhancement efforts and staff from the
technology vendor who supported the assessment.
The full K-3 Formative Assessment was piloted with approximately 80 K–3 teachers and 800
children in four states during the 2015–16 school year. Each teacher assessed a sample of 10
children on all 17 constructs.2 The fall pilot sample is described in Exhibits 1 and 3, the winter
pilot sample in Exhibit 2.
Teachers’ use of the assessment was supported by a technology platform. The teachers
entered evidence (such as anecdotal notes, photographs, or videos) from their observations for
each construct and child in the sample. They then determined where on the progression each
piece of evidence fell (preliminary learning status) and, across multiple pieces of evidence over
multiple observations, where to place the child on the progression (status summary). All the
assessment data (evidence, learning statuses, status summaries) was entered into the platform
for easy review, reporting, and analysis. This process was further supported by an application
that allowed teachers to upload directly to the platform evidence (their notes, photos, and
videos) from a tablet or smart phone to assist in selecting a status summary.
2 Mathematical Patterns was not included because it was developed after the pilot
EAG K-3 Formative Assessment: Final Report September 2017 C-4
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Teachers provided feedback on the content, manageability and usability of the assessment to
inform instruction through online surveys, focus groups, and interviews. Surveys were sent to
teachers after the training and at the end of the fall and winter data collection periods. Teachers
also participated in focus groups or individual interviews at the end of the fall and winter data
collection windows.
Findings Findings are presented in four areas:
• Teacher feedback on the training and professional development they received;
• Implementation of the assessment;
• Manageability of the assessment materials and process; and
• Overall usability of the assessment materials and process
Training/professional Development Teacher Background and Education. In both the fall and winter samples, the majority of
teachers were white and female (Table #). Over half of the teachers in both the fall and winter
sample had at least a master’s degree and over three-fourths had earned the highest certificate
available in their state. Teachers had about 14 years of teaching experience, and over one-
quarter of teachers had experience teaching preschool.
Preparation for implementing the K-3 Assessment. Teachers received in-person training on
the formative assessment process, content of the assessment (i.e., construct progressions and
assessment means), and opportunities for practicing with the assessment, including how to use
the technology as part of the process. Embedded in the training were construct modules with
video content for “live” coding. Training materials provided to the teachers (i.e., reference
binders and handouts) as well as additional resources developed over the course of the pilot
were made available through a website. The majority of teachers felt very prepared or
adequately prepared to implement the assessment in both the fall (81%) and winter (86%).
Aspects of the fall training that were of particular help in preparing teachers to use the
assessment were overviews of the construct progressions (72%), assistance provided by the
trainers in organizing or “clustering” the constructs (53%), demonstration of the technology
(53%), and practice observing and documenting evidence (41%).
Based on feedback from the fall, changes were made to the winter training, including less time
spent on background information on formative assessment and the vision for the assessment,
and more hands-on opportunities to practice using the assessment materials. The background
EAG K-3 Formative Assessment: Final Report September 2017 C-5
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
and overview of the assessment were instead covered in pre-training modules rather than the
in-person training. Other content added to the training covered characteristics of high-quality
evidence, tips for documenting evidence, and fitting the assessment into the instructional day.
After the winter training, over 50% of teachers found the majority of the practice activities to be
helpful (i.e., planning, summarizing the status, looking for and gathering evidence, and
assigning learning statuses). Teachers in the fall sample who returned for the winter pilot were
provided the opportunity to attend the winter training and had access to the revised training
materials.
Implementation of Assessment Process We were able to learn a great deal about how teachers implemented the assessment from what
they documented on the technology platform. The teachers entered evidence (such as
anecdotal notes, photographs, or videos) from their observations for each construct and child in
the sample. They then determined where on the progression each piece of evidence fell
(preliminary learning status) and, across multiple pieces of evidence over multiple observations,
where to place the child on the progression (status summary). All the assessment data
(evidence, learning statuses, status summaries) could be entered into the platform for easy
review, reporting, and analysis. This process was further supported by an application that
allowed teachers to upload directly to the platform evidence (their notes, photos, and videos)
from a tablet or smart phone to assist in selecting a status summary.
Based on the evidences and status summaries documented on the platform, teachers were able
to assess most of the constructs and followed the process with some fidelity, documenting
multiple pieces of evidence, assigning preliminary learning status and determining a status
summary for each child and each construct. This varied by construct with teachers more likely
to complete the assessment of those constructs introduced in the earlier clusters and less likely
to complete the assessment of constructs they found more challenging. For example, in the fall
pilot less than 70 percent of the children in the sample were assessed on the social-emotion
constructs, Vocabulary and Perseverance.
From the information in the platform, we learned the following about teachers’ adherence to the
steps in the assessment process:
• Assigned child a status summary (required). Teachers assigned a status summary for at
least one construct for 372 children in the fall. Across constructs, 57 to 93 percent of the
children in the pilot sample were assessed in the fall. The constructs least frequently
EAG K-3 Formative Assessment: Final Report September 2017 C-6
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
assessed were Emotional Literacy (57 percent of the children), Emotional Regulation
Strategies (61 percent), Emotion Expression (63%), Vocabulary (64%) and Perseverance
(67%).
• Documented multiple pieces of evidence (required but might not be in the platform).
Teachers did not consistently document more than one piece of evidence for determining a
child’s placement on a progression in the technology platform. Across constructs, 39 to 78
percent of the status summaries were based on multiple pieces of evidence in the fall.
Constructs least often documented with multiple pieces of evidence were Reading
Comprehension (39%), Perseverance (40%), Vocabulary (43%), and Object Counting
(44%).
• Assigned evidence a preliminary learning status (optional). Teachers were encouraged
to assign each piece of evidence a preliminary learning status to help them interpret multiple
pieces of evidence and determine the child’s status summary. Teachers followed this
optional step for the majority of all the evidence records entered on the platform. When
learning status was tabulated by construct and child, some construct variation was
observed. Learning status was assigned least often for Emotional Literacy (49%), Emotion
Regulation Strategies (50%) and Vocabulary (51%).
• Looking across the three steps in the assessment process, we observed that teachers were
least likely to collect multiple pieces of evidence, assign the evidence with a preliminary
learning status and determine a status summary for the three constructs in the social-
emotional domain and Vocabulary. Based on this finding and the survey findings from
teachers described below, we recommended additional changes and supports for these
constructs.
Manageability of the Assessment A goal of the fall and winter pilot studies was to understand the manageability of implementing
the assessment in kindergarten, first, second, and third grades, with an emphasis on
kindergarten in the fall. To determine manageability of the assessment, teachers responded to
survey and interview questions about how teachers incorporated the assessment into their
instructional practice, their perceived difficulty in observing and understanding constructs, the
compatibility of the K-3 Assessment with other assessments administered, use of technology to
support the implementation of the assessment, and the challenges and support they received.
Teachers were provided multiple opportunities to reflect on implementation of the assessment
both during and after the assessment.
EAG K-3 Formative Assessment: Final Report September 2017 C-7
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Use of Other Assessments. The intent of the K-3 Formative Assessment is to complement
other assessments a teacher might use as part of a balanced comprehensive assessment
system. Thus, it was expected and confirmed that teachers would be able to use information
from other assessments to inform the K-3 Assessment, which in turn should make administering
the assessment more manageable. Over 85 percent of teachers in the fall and winter (85% and
88%, respectively) reported that they used information from other assessments as evidence for
the K-3 Assessment. These other assessments included district-mandated assessments and
teacher-created assessments in reading and math, running records and writing prompts, and
formal benchmark tests (e.g., DIBELS, FAST, and Fountas and Pinnell’s Benchmark
Assessment System). Teachers often mentioned using information from other assessments for
the literacy and mathematics constructs, including Letter Naming, Reading Comprehension,
Writing, and Object Counting.
Difficulty Assessing Constructs. Teachers reported that some constructs were more difficult
to assess than others, namely, the constructs in the social-emotional domain, whereas the more
“Kindergarten” oriented constructs tended to be nominated as difficult less than the other
constructs or not at all.
• In both the fall and winter, teachers reported that the social-emotional constructs were the
most difficult to assess. When asked to indicate any construct they found to be difficult at the
end of the winter pilot, 76% of teachers reported at least one emotion construct was difficult,
and 42% of teachers reported all three emotion constructs as being difficult. Some teachers
noted inconsistencies in children’s behavior from day to day as a reason for the social-
emotional constructs being difficult. Two insightful comments from teachers:
– All of the emotional pieces were difficult because my students are 5 and 6…and
their emotions are SO drastically changing from minute to minute sometimes. It's
exhausting to try and keep up with and to try to "analyze" or figure things out enough
to place them on the chart was difficult.
– Without the time for play to work through these emotions and discuss and such a
high academic rigor expectation, it is harder to document throughout the day.
• At least 25 percent of teachers also reported Vocabulary and Perseverance as being difficult
to assess. Some teachers felt that the Perseverance construct was more open to
interpretation, which made it difficult to assess,
EAG K-3 Formative Assessment: Final Report September 2017 C-8
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
– I feel this was challenging because I can give my students difficult tasks, but I never
felt clear if it was super challenging for them and I am not going to give them
something they will fail at to see if they will stick with it.”
For Vocabulary, teachers reported that it was difficult to look for during the day,
• Unless given specific situations, it was hard to catch and listen to students using “school
related” language.
• Across constructs, teachers’ difficulties in assessing constructs often stemmed from inability
to observe the construct during the day and/or the construct took too much time to assess.
• Constructs not nominated as being difficult to assess by any teachers included Book
Orientation, Hand Dominance, Letter Naming, Grip and Manipulation, Print Awareness,
Object Counting and Writing.
Incorporating Assessment into Instructional Activities. The assessment is designed such
that constructs can be observed during typical classroom activities, however 27 percent of fall
teachers and 38 percent of winter teachers found one or more constructs did not fit into daily
instruction.
• Some fall (21%) and winter (13%) teachers found Crossing Midline to not fit into daily
instruction.
• Both fall (18%) and winter (27%) teachers found Gross Motor to be difficult to assess during
daily instruction.
• Some teachers also found that the emotion constructs did not fit into daily instruction (18%
in the fall for Emotion Expression, Emotion Regulation strategies, and Emotional Literacy).
Fewer teachers in the winter reported the Emotion Expression, Emotion Regulation
Strategies, and Emotional Literacy constructs to not fit into daily instruction (7%, 9%, and
11%, respectively).
• Notably, teacher found some constructs easier to fit into daily instruction in the winter. This
may be attributed to increased comfort with the assessment, and more focused training in
the winter, particularly for constructs reported as difficult to assess in the fall (e.g., social-
emotional constructs).
Use of Materials. Teachers were provided multiple resources to support the constructs,
information about the constructs and documentation forms, and provided feedback on the ease
of use and helpfulness of each resource:
EAG K-3 Formative Assessment: Final Report September 2017 C-9
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
• The Construct Progression document provided a rationale and background information
about the construct, skills on the progression for which to observe the children, performance
descriptors for each skill, and examples for each skill. Over half of the teachers in the fall
and winter reported the construct progressions as easy to use (72% and 66%) and/or helpful
(67% and 64%) during implementation.
• The Assessment Means Form also included the construct skills, performance descriptors
and selected examples, as well as information on how to assess a child for that construct.
For example, the Assessment Means Forms contain sections on key terminology used in the
Construct Progression, potential settings and materials, general teacher instructions, skill-
specific teacher instructions, and situations and tasks one may construct in order to better
elicit the information needed. Over one-quarter of teachers found the assessment means
forms easy to use in the fall and winter (25% and 31%), and even more teachers also found
them helpful (40% and 35%).
• Construct Modules were online tutorials for each construct that provided an overview of the
construct, detailed explanation of the construct progression skills, and videos illustrating
children at different levels on the progression. Over one-third of teachers in the fall and
winter found the construct modules easy to use (38% and 45%), and even more teachers
found them helpful (40% and 47%).
• The single child and multiple children Documentation Forms were designed as an
alternative to the tech platform for teachers to document evidence and determine status
summaries for a single child over time, or multiple children on one form. About one third of
teachers in the fall and winter found the single child forms easy to use (38% and 31%)
and/or helpful (33% and 33%). Just under half of the teachers in the fall and winter found the
multiple children documentation form to be easy to use (47% and 50%) and helpful (43%
and 45%).
• The Teacher Site was a web-based repository for electronic copies of all construct
progressions, assessment means forms, documentation forms, modules; PowerPoints and
other training materials; calendars and timelines for completion of the pilot; and an optional
discussion board for each state and construct. Few teachers found this teacher resource
website easy to use (9% in the fall and 16% in the winter). More teachers found the
Teacher Site helpful (20% in the fall, and 24% in the winter).
EAG K-3 Formative Assessment: Final Report September 2017 C-10
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Overall the materials and resources were well received, particularly those materials that were
critical to implementing the assessment (i.e., Construct Progressions). Few teachers reported
any of the materials as being difficult to use.
Use and Perceptions of Technology. The technology platform and K-3 evidence application
were designed to provide a more expedient way of collecting and uploading evidence; organize
the evidence collected during their observations; and help teachers summarize, track, and
report information. Thus, the use of technology in this assessment was critical to making the
assessment process more manageable and less burdensome by providing alternatives to
traditional note-taking and record keeping. All teachers in the fall sample and nearly all teachers
in the winter sample (95%) reported having access to a computer in the classroom, and nearly
80% reported having access to a tablet or mobile device at school. All teachers had access to
the internet, however some teachers reported that their internet access was “spotty” in the fall
(14%) and the winter (21%).
Overall, both the technology platform and the K-3 evidence application for mobile devices and
tablets were reported to be easy to use and helpful.3
• Fifty-six percent of teachers in the fall and 60% of teachers in the winter found the
technology platform to be easy to use.
• Teachers described the platform overall as being “user-friendly” and easy to use during
interviews, though several teachers acknowledged that using the technology platform was a
steep learning curve at first.
• The one aspect of the technology platform that appealed to teachers the most was the
differing ways in which information could be viewed. For example, teachers liked being able
to view the constructs and examples on the platform as they were assigning learning and
status summaries, rather than having to refer back to hard copy or other soft copy materials.
Teachers also appreciated the different ways in which information could be accessed, such
as viewing status summaries by the class or by individual children. Teachers reported using
the individual and class reports for viewing progress in data collection, either for completion
by child and/or by the amount of data collected across constructs.
• More teachers found the technology platform helpful in the winter (66%) than in the fall
(41%). The increase in teachers reporting the technology as easy and helpful from fall to
3 Some states and districts provided tablets for teachers to use, although these devices were not always available at the beginning of data collection. Teachers had the option of using personal devices if none were provided.
EAG K-3 Formative Assessment: Final Report September 2017 C-11
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
winter is likely due to increased use (for returning teachers), improvements in training for the
winter pilot, additional support materials (e.g., application tutorials), and the fact that by
winter more teachers had been provided with tablets.
Teachers’ views about the K-3 mobile application were less straightforward than feedback on
the technology platform.
• In the fall, only about one-quarter of teachers found the application easy or helpful (28% and
25%, respectively). In the winter, more teachers found the application to be helpful than
easy, with increases in both ease of use and helpfulness from fall to winter (48% and 55%,
respectively).
• Teachers who liked the application reported in interviews that they found it made the
assessment process easier and less time consuming. Aspects of the application that were
particularly helpful were seeing the skills on the application, using the device to take and
upload videos and pictures, and using the speech-to-text feature.
• Teachers who did not report favorably on the application either did not use the application
due to issues with Wi-Fi in their school or classroom, prohibition of mobile devices in the
classroom, or because of problems they experienced while using the application that did not
get resolved. As mentioned, some teachers did not have consistent access to the internet,
and had to upload evidence and work onto the platform outside of school.
• Teachers also mentioned that it was at times hard to use “in the moment” because of
privacy features, such as having to re-enter the password every time the application
opened, “The password we always had to enter was time consuming…Took forever to get in
and [I] lost opportunities for collecting evidence.”
In open ended survey and interview questions regarding the technology, teachers identified
three primary ways in which the technology platform and K-3 application could be improved:
• Provide additional training, either on the platform, the application, or as a booster training
once data collection was underway;
• Organize information on the platform so information, particularly student evidence, is easy to
locate; and
• Provide easier access into the mobile application, including bypassing the password feature
when phone or tablet has briefly been in stand-by mode.
EAG K-3 Formative Assessment: Final Report September 2017 C-12
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Challenges. The fall pilot was the first opportunity for any participating teacher to implement the
full assessment, and for some teachers, the winter pilot was their first opportunity to work with
the assessment materials. It was expected that there would be both challenges to implementing
the assessment.
• Asked about challenges part-way through the data collection, more than half of the teachers
identified that time to implement the assessment was the biggest challenge in both the fall
(63%) and winter (54%). Teachers also identified time spent implementing the assessment
as the greatest challenge at the conclusion of the winter pilot (39%). Teachers specified that
it was difficult to find time to observe the constructs and to upload data to the technology
platform, particularly during the work day.
• Additional challenges identified at the conclusion of the winter pilot included not enough
support in the classroom (42%), not enough staff in the school trained on the K-3
assessment (38%), not enough time to practice using the technology (33%), and perceived
lack of alignment with state standards (33%).
Despite challenges with implementing the assessment, teachers found certain aspects of the
assessment that facilitated it implementation, or discovered their own ways to make the process
more manageable.
• When asked what went well when implementing the assessment at the conclusion of the
winter pilot, 40 percent of teachers again mentioned use of the application on a tablet or
mobile device. One teacher commented, “The K-3 app was a wonderful and useful tool to
use. It allowed me to capture organic situations that were happening in my classroom. I
found by using this tool I did not have to set up any situations to capture evidence.”
• Another positive aspect of the pilot that teachers mentioned was the availability of and
support of project staff (9%).4
• Teachers also reported during interviews that they found ways to manage the
implementation of the assessment such as creating their own checklists for monitoring data
collection, planning which constructs would be observed in a day, and being flexible in their
approach to collecting data (i.e., capitalizing on opportunities to observe a construct).
• Finally, some teachers were able to draw on other staff to support the collection of data.
About 20 percent of teachers in the fall and winter reported working with other educators to
4 Each state had an assigned “research liaison” and a dedicated technology platform/application contact who were available to answer questions and provide one-to-one support.
EAG K-3 Formative Assessment: Final Report September 2017 C-13
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
help collect data from the assessment. And nearly 50 percent or more teachers in the fall
and winter (64% and 48%) reported having support staff in the classroom, which suggests
that support staff may be have been an under-utilized potential resource.
Usability of Overall Assessment Teachers reported that the assessment helped them understand the student skills on each
progression, where their students fell and their next steps in learning.
• The majority of teachers in the fall and winter reported that they were able to use information
from the assessment (72% and 60%, respectively).
• Of the teachers who used the information from the assessment, more than half in the fall
and winter reported that they used the information to identify areas of individual or group
needs or to generally inform instruction (61% and 69%, respectively). One teacher noted,
“[The information from the assessment] is helping to group students for differentiated work
and also helping to hone in on specific skills and create a dialogue with the specialists in the
building more knowledgeable in those areas.”
• Teachers most often described using data from the Language and Communication Domain,
the Cognitive Domain, and the Physical/Motor Domain. One teacher provided an example of
working with a student on crossing midline, “Working with a specific student who was
struggling in all academic areas, I assessed for crossing the midline and found that he could
NOT cross the midline even with assistance. This shows that the two parts of his brain are
not working in sync. Once we increased the crossing midline activities, he started to make
progress elsewhere.”
• Teachers who did not use information from the assessment indicated that they already knew
this information about their children, the information did not align with what they were
teaching, or the information was not as relevant at the time of assessment.
Use of the Assessment with Special Populations. More teachers were able to implement the
assessment with children with developmental delays or disabilities than English Language
Learners, though the majority that did implement the assessment with these populations found
the assessment to be useful or could be useful, either with or without additional modifications or
supports.
• More than half of the teachers were able to implement the assessment with children with
developmental delays or disabilities in the fall (63%) and winter (52%). The majority of these
teachers felt that the skills were or would be effective in capturing their abilities in the fall
EAG K-3 Formative Assessment: Final Report September 2017 C-14
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
and winter (92%a and 100%, respectively). Teachers noted that the kinds of supports that
would be helpful for working with these students would be more assistance from other staff
(e.g., OT, PT, team members), more time spent with the students, and additional guidance
for working with these students.
• Less than half of the teachers were able to implement the assessment with English
Language Learners in the fall (38%) and winter (13%). The majority of these teachers felt
that the skills were or would be effective in capturing their abilities in the fall and winter (75%
and 100%, respectively). Teachers noted that the supports that would be helpful might
include additional guidance for working with these students (particularly assessment in the
child’s native language) and use of visual materials.
Overall Perspectives on Using the Assessment. The majority of teachers in the pilot study
found the K-3 Formative Assessment to be a useful resource for instruction (70% and 61% of
teachers in the fall and winter, respectively). Teachers often cited the assessment as providing
a new or different perspective on students, providing a look at the whole-child, helping to drive
instruction, and providing information for speaking with parents. One teacher commented, “I've
really been able to get an in-depth picture of my kids and the ability of sharing that with the
parents has been great. There was overwhelming support from the parents and it's allowed me
to teach in the way that I know to be good for kids.”
There was less consistency in teacher’s responses for why the assessment was useful for their
instruction. Some teachers that did not find the K-3 Assessment as a resource for instruction
noted that much of the information they already knew at the time of the assessment, suggesting
a mismatch in timing of the pilot data collection windows and where they were in their course of
instruction. Other teachers felt that some of the information was already collected using other
instruments and thus duplicative in nature. Some teachers reported that the information could
have been useful with more guidance about how to use the information to support instruction or
had the assessment been conducted with all of the students in their classes.
Recommendations The overall objective for the pilot was to gather teacher perspectives on the assessment content
and process as well as the materials and training activities in order to inform the 2016-17 field
test. To this end, here is a summary of the teachers’ feedback and the recommendations for
changes that were implemented in the field test.
EAG K-3 Formative Assessment: Final Report September 2017 C-15
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
• The assessment process was time consuming, and documenting and interpreting multiple
pieces of evidence to determine where a child fell on each progression was cumbersome.
Consequently, the field test training was modified to further demonstrate how formative
assessment observations can fit into daily instructional routines.
• The social-emotional constructs were particularly difficult to assess. Accordingly, the
Emotional Literacy progression was shortened and simplified, the assessment materials for
Emotion Expression and Emotional Regulation Strategies were clarified, and more guidance
on how to assess these constructs was developed (see Assessing the Social - Emotional
Constructs: Overview).
• The Vocabulary construct was challenging to assess. Thus, two progression skill levels that
were difficult to observe in the classroom were combined (using vocabulary in different
contexts) and the assessment materials clarified to emphasize that the construct is about
vocabulary concepts rather than vocabulary word lists.
• The progression placements for each construct at each grade level were reviewed and
decisions made to include additional grade levels where appropriate in the field test to
ensure that we collected the full range of data necessary to validate each progression.
EAG K-3 Formative Assessment: Final Report September 2017 C-16
EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT
Exhibits
Exhibit 1. Pilot Study Fall Sample (All K)
State N of Teachers N of Children Arizona 11 131 Iowa 8 88 Maine 8 89 Rhode Island 6 64 Total 33 372
Exhibit 2. Pilot Study Winter Sample (K-3)
Grade N of Teachers N of Children Kindergarten 37 352 Grade 1 13 114 Grade 2 8 67 Grade 3 7 51 Total 65 584
Exhibit 3. Pilot Study Fall Sample: Child Characteristics
Characteristics N of Children % of Children Gender (Female) 159 52.4% English Learner 28 7.5% Has IEP 30 8.1% Race/Ethnicity Asian/Pacific Islander 5 1.3% Black 9 2.4% Hispanic 56 15.1% White 256 68.8% Other 12 3.2% Unknown 34 9.1%
EAG K-3 Formative Assessment: Final Report September 2017 C-17
Appendix D Field Test: Supporting Exhibits Exhibit D.1.1. Teacher survey: Background and training
Exhibit D.1.2. Teacher survey: End of fall
Exhibit D.1.3. Teacher survey: End of field test
Exhibit D.1.4. Teacher focus group protocol
Exhibit D.2.1. Fall K progression placement distributions by distribution type
Exhibit D.2.2. Assessment results for 18 constructs by grade level and administration (fall,
winter)
Exhibit D.2.3 School readiness scores by construct and skill level: Mean total scores (K only)
and mean IRT factor scores (K-3)
Exhibit D.2.4. Teacher survey: Kindergarten entry performance levels
Exhibit D.2.5. Kindergarten entry performance levels: Sample domain expert questionnaire
Exhibit D.2.6. Final K entry performance levels compared to teacher survey responses and K
fall assessment data
Exhibit D.2.7. Percentage of kindergarten students developing and demonstrating expected
skill levels in fall based on expert recommended performance levels
Exhibit D.2.8. Mean progression placements for children assessed in fall and winter
Exhibit D.2.9. Mean progression placements for children assessed in fall by grade level
Exhibit D.2.10. Mean progression placements for children assessed in winter by grade level
Exhibit D.2.11. Percent of kindergarten children at higher, same, or lower levels on progression
from fall to winter administrations
Exhibit D.2.12. Percent of grade 1 children at higher, same, or lower levels on progression from
fall to winter administrations
Exhibit D.2.13. Percent of grade 2 children at higher, same, or lower levels on progression from
fall to winter administrations
EAG K-3 Formative Assessment: Final Report September 2017 D-1
Exhibit D.2.14. Percent of grade 3 children at higher, same, or lower levels on progression from
fall to winter administrations
Exhibit D.2.15. Progression placements for kindergarten children with and without IEPs
Exhibit D.2.16. Progression placements for kindergarten ELLs and non-ELLs
Exhibit D.2.17. Gender differences in construct performance by grade and test administration
Exhibit D.2.18. Racial/ethnic differences* in construct performance by grade and test
administration
Exhibit D.3.1. Coding protocol for evidence analysis
Exhibit D.3.2. Internal reliability coefficients for evidence analysis
Exhibit D.4.1. A profile of teachers participating in the field test
Exhibit D.5.1. Training guidance for administrators
Exhibit D.7.1. Construct performance for each validity claim
EAG K-3 Formative Assessment: Final Report September 2017 D-2
Exhibit D.1.1. Teacher survey: Background and training
K-3 Formative Assessment: Background & Post-Training Teacher Survey 2016–17
Section 1: General Background
1. What grade level are you teaching this year? � Kindergarten � 1st Grade � 2nd Grade � 3rd Grade � Mixed/Multi-Grade (please identify grades): ________ � Other (please describe): ____________________
If you are teaching kindergarten, 1b. Do you teach full- or half-day classes?
� Full Day � Half Day
2. How many hours per day does your class normally meet? (Enter a whole number only.) Hours: ________
3. Including this year, how many years have you been teaching? (Enter a whole number only, leave as "0" if this is your first year teaching.) Years: ________
4. Counting this school year, how many years have you taught each of the following grades and programs? (Enter the number of years to the nearest half-year. Please include part-time teaching. Leave the entry blank if you have never taught the grade or program listed.) a. Preschool or Head Start: ________ b. Kindergarten: (including Transitional/Readiness Kindergarten and Transitional/ pre-1st
grade): ________ c. First grade: ________ d. Second grade: ________ e. Third grade: ________ f. Fourth grade or higher: ________ g. English as a Second Language (ESL) program: ________ h. Bilingual education program: ________ i. Special education program: ________ j. Physical education program: ________ k. Art or music program: ________
EAG K-3 Formative Assessment: Final Report September 2017 D-3
5. Which of the following best describes the teaching certificate you currently hold in THIS state? (Mark one response.) � Regular or standard state certificate or advanced professional certificate � Certificate issued after satisfying all requirements except the completion of a
probationary period � Certificate that requires some additional coursework, student teaching, or passage of a
test � Certificate issued to persons who must complete a certification program in order to
continue � I do not hold any of the above certifications in THIS state.
If you hold a teaching certificate in THIS state: 5b. In what areas are you certified? (Select all that apply.)
� Elementary education � Early childhood education � Special education � English as a Second Language (ESL) or instruction for English language learners or
bilingual education � Other (please specify): ____________________
6. Do you have paid paraprofessionals/aides or volunteers working with children in your classroom? � Yes � No
If you selected yes to question 6: 6b. How many hours a week do different types of paid paraprofessionals/aides and/or
volunteers usually work directly with children on instructional tasks either in your classroom or in a pullout setting? (Enter a whole number of hours per week. Leave the entry as blank if none.) a. General paraprofessionals/aides: ________ b. Special education paraprofessionals/aides: ________ c. ESL or bilingual education paraprofessionals/aides: ________ d. Volunteers (e.g., parents, high school students, community members): ________
7. Do you have a computer in your classroom? � Yes � No
8. Do you have access to a tablet/mobile device at school? � Yes � No
EAG K-3 Formative Assessment: Final Report September 2017 D-4
9. Do you have internet access in your classroom? � Yes, good access � Yes, spotty access � No
10. Do you have access to internet somewhere in your school? � Yes, good access � Yes, spotty access � No
11. Have you had professional development on formative assessment (outside of this project)? � Yes � No
If you selected yes to question 11: 11b. Approximately how many hours? (Enter a whole number only.)
_______
12. Were you already using formative assessment in your classroom before this project? � Yes � No
Section 2: Demographics
13. What is your gender? � Female � Male
14. Are you Hispanic/Latino? � Yes � No
15. Which best describes your race? (Mark one or more responses to indicate what you consider yourself to be.) � American Indian or Alaska Native � Asian � Black or African American � Native Hawaiian or Other Pacific Islander � White
EAG K-3 Formative Assessment: Final Report September 2017 D-5
16. Which languages other than English do you speak fluently? (Select all that apply.) � No language other than English � Spanish � French � Arabic or Amharic � Vietnamese � A Chinese language � Japanese � Korean � Other language (please specify): ____________________
17. What is the highest level of education you have completed? � High school diploma or GED � Associate's degree � Bachelor's degree � At least one year of course work beyond a Bachelor's but not a graduate degree � Master's degree � Education specialist or professional diploma based on at least one year of course work
past a Master's degree level � Doctorate � Other (please specify): ____________________
Section 3: Post-Training Questions
18. How prepared do you feel to implement the assessment after the field test training? � Very prepared � Adequately prepared � Fairly prepared � Not prepared at all
19. Did you participate in the K-3 Formative Assessment Pilot in 2015–16? � Yes, in the Fall K Pilot and/or Winter K-3 Pilot � No, the Field Test is my first time using this assessment
If you selected yes to question 19: 19b. What aspect(s) of the training have been helpful when using the assessment? (Select all
that apply.)
� Updates to the K-3 Formative Assessment � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-6
If you selected yes to question 19: 19c. Were there any aspects of the training not helpful? (Select all that apply.)
� Updates to the K-3 Formative Assessment � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________
If you selected no to question 19: 19d. What aspect(s) of the training have been helpful to you when using the assessment?
(Select all that apply.)
� Overviews of five domains � Overview of construct structure � Overview of formative assessment � Magnifying glass advance organizer � Practice “Getting a plan” � Practice “Looking for and gathering evidence” � Practice “Assigning a learning status” � Practice “Summarizing the status” � Practice “Systematically using data” � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________
If you selected no to question 19: 19e. Were any aspects of the training not helpful? (Select all that apply.)
� Overviews of five domains � Overview of construct structure � Overview of formative assessment � Magnifying glass advance organizer � Practice “Getting a plan” � Practice “Looking for and gathering evidence” � Practice “Assigning a learning status” � Practice “Summarizing the status” � Practice “Systematically using data” � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-7
20. What else would have been helpful to include in the training that would support your use of the assessment? (e.g., additional information or resources)
___________________________________________________________________
Thank you for your participation in this survey!
EAG K-3 Formative Assessment: Final Report September 2017 D-8
Exhibit D.1.2. Teacher survey: End of fall
K-3 Formative Assessment: Fall Teacher Survey 2016-17
Section 1: Assessment Materials
1. Please rate your ease of use for each of the following assessment materials/resources: (Select one response for each row.)
Very easy to use
Easy to use
Difficult/ confusing to
use
Very difficult/ confusing to
use
Did not use this material/
resource
Construct Progression □ □ □ □ □
Assessment Means Form □ □ □ □ □
Single Child Documentation Form □ □ □ □ □
Multiple Children Documentation Form □ □ □ □ □
K-Entry Checklist □ □ □ □ □
Online Construct Modules □ □ □ □ □
K-3 Technology Platform □ □ □ □ □
K-3 Evidence App □ □ □ □ □
K-3 Teacher Site □ □ □ □ □
EAG K-3 Formative Assessment: Final Report September 2017 D-9
2. How helpful have you found each of the following materials/resources in implementing the assessment so far? (Select one response for each row.)
Very helpful
Somewhat helpful
Not helpful
Not at all helpful
Did not use this material/
resource
Construct Progression □ □ □ □ □
Assessment Means Form □ □ □ □ □
Single Child Documentation Form □ □ □ □ □
Multiple Children Documentation Form □ □ □ □ □
K-Entry Checklist □ □ □ □ □
K-3 Technology Platform □ □ □ □ □
K-3 Evidence App □ □ □ □ □
K-3 Teacher Site □ □ □ □ □
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Construct Progression helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Assessment Means Form helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Single Child Documentation Form helpful? ________________________________________________________________________________________________________________________
EAG K-3 Formative Assessment: Final Report September 2017 D-10
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Multiple Children Documentation Form helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-Entry Checklist helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Technology Platform helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Evidence App helpful? ________________________________________________________________________________________________________________________
If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Teacher Site helpful? ________________________________________________________________________________________________________________________
EAG K-3 Formative Assessment: Final Report September 2017 D-11
3. Did you watch any of the online construct modules? � Yes � No
If you answered “yes” to question 3: Which construct modules did you watch? (Select all that apply.)
� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies’ � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary
If you answered “yes” to question 3: For each construct module watched, please rate its helpfulness. (Select one response for each row.)
Very helpful
Somewhat helpful
Not helpful
Not at all helpful
Book Orientation □ □ □ □
Crossing Midline □ □ □ □
Emotion Expression □ □ □ □
Emotion Regulation Strategies □ □ □ □
Emotional Literacy □ □ □ □ Fine Motor: Grip and Manipulation □ □ □ □
Fine Motor: Hand Dominance □ □ □ □
Following Directions □ □ □ □
Gross Motor □ □ □ □
Letter Naming □ □ □ □
Mathematical Patterns □ □ □ □
EAG K-3 Formative Assessment: Final Report September 2017 D-12
Very helpful
Somewhat helpful
Not helpful
Not at all helpful
Object Counting □ □ □ □
Perseverance □ □ □ □
Print Awareness □ □ □ □
Problem Solving □ □ □ □
Reading Comprehension □ □ □ □
Writing □ □ □ □
Vocabulary □ □ □ □
If you answered “no” to question 3: Why didn’t you watch any of the modules? ________________________________________________________________________________________________________________________
EAG K-3 Formative Assessment: Final Report September 2017 D-13
Section 2: Assessment Content and Process
4. Did you cluster the constructs when implementing the K-3 Assessment in your classroom this fall? � Yes � No
If you answered “no” to question 4: Why not? ________________________________________________________________________________________________________________________
If you answered “yes” to question 4: How helpful has the clustering been in implementing the assessment?
� Very helpful � Somewhat helpful � Not helpful � Not at all helpful
EAG K-3 Formative Assessment: Final Report September 2017 D-14
5. Please rate your ease in assessing each of the following constructs: (Select one response for each row.)
Did not assess
Very easy to assess
Easy to assess
Difficult to assess
Very difficult to assess
Book Orientation □ □ □ □ □
Crossing Midline □ □ □ □ □
Emotion Expression □ □ □ □ □
Emotion Regulation Strategies □ □ □ □ □
Emotional Literacy □ □ □ □ □
Fine Motor: Grip and Manipulation □ □ □ □ □
Fine Motor: Hand Dominance □ □ □ □ □
Following Directions □ □ □ □ □
Gross Motor □ □ □ □ □
Letter Naming □ □ □ □ □
Mathematical Patterns □ □ □ □ □
Object Counting □ □ □ □ □
Perseverance □ □ □ □ □
Print Awareness □ □ □ □ □
Problem Solving □ □ □ □ □
Reading Comprehension □ □ □ □ □
Writing □ □ □ □ □
Vocabulary □ □ □ □ □
If you answered “difficult to assess” or “very difficult to assess” for Book Orientation in question 5: Why was the Book Orientation construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-15
If you answered “difficult to assess” or “very difficult to assess” for Crossing Midline in question 5: Why was the Crossing Midline construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Emotion Expression in question 5: Why was the Emotion Expression construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Emotion Regulation Strategies in question 5: Why was the Emotion Regulation Strategies construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Emotional Literacy in question 5: Why was the Emotional Literacy construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Fine Motor: Grip and Manipulation in question 5: Why was the Fine Motor: Grip and Manipulation construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-16
If you answered “difficult to assess” or “very difficult to assess” for Fine Motor: Hand Dominance in question 5: Why was the Fine Motor: Hand Dominance construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Following Directions in question 5: Why was the Following Directions construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Gross Motor in question 5: Why was the Gross Motor construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Letter Naming in question 5: Why was the Letter Naming construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Mathematical Patterns in question 5: Why was the Mathematical Patterns construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-17
If you answered “difficult to assess” or “very difficult to assess” for Object Counting in question 5: Why was the Object Counting construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Perseverance in question 5: Why was the Perseverance construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Print Awareness in question 5: Why was the Print Awareness construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Problem Solving in question 5: Why was the Problem Solving construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Reading Comprehension in question 5: Why was the Reading Comprehension construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-18
If you answered “difficult to assess” or “very difficult to assess” for Writing question 5: Why was the Writing construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
If you answered “difficult to assess” or “very difficult to assess” for Vocabulary in question 5: Why was the Vocabulary construct difficult to assess? (Select all that apply.)
� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________
Section 3: Current Assessment Practices
6. How many other assessments are you required to use this school year? (Enter a whole number only, leave as “0” if none.) Number of assessments:
_______
If you have 1 or more assessments this school year: What domains/areas of development do these assessments capture? (Select all that apply.)
� Literacy/Reading � Math � Writing � Health � Other (Please specify): ____________________
If you have 1 or more assessments this school year: Consider all the time you spend assessing one child with all other required assessments. What is your estimate of total time per child? (Enter a whole number of minutes only, leave as “0” if none.)
Number of minutes: _______
7. Did you use data/information from other assessments as evidence for determining learning status(es) on any of the construct progressions in this assessment? � Yes � No
EAG K-3 Formative Assessment: Final Report September 2017 D-19
If you answered “yes” to question 7: What other assessments did you use to inform learning status(es) on the K-3 Formative Assessment? ________________________________________________________________________________________________________________________
8. To what extent are you using information from the K-3 Formative Assessment (e.g., evidence collected, learning status) to inform your instruction? � A great deal � Somewhat � A little � Not at all
9. What supports would be helpful to successfully use this assessment? (Select all that apply.) � Professional Learning Communities (PLCs) � Time allocated for planning � Team meetings to discuss � Good internet access � Access to computer/laptop in classroom � Other (Please specify): ____________________
EAG K-3 Formative Assessment: Final Report September 2017 D-20
Section 4: Tell us about your class
10. In a typical day, how much time does a child in your class spend in the following types of activities? (Select one response for each row.)
No time
Half hour or less
About one hour
About two hours
About three hours
Four hours or more
Working independently □ □ □ □ □ □
Working on individual tasks under teacher direction
□ □ □ □ □ □
Working with peers under teacher direction
□ □ □ □ □ □
Working in small groups with teacher
□ □ □ □ □ □
Teacher lecture with large group and/or large group discussion led by teacher
□ □ □ □ □ □
11. During a typical day, how much time per day would you estimate that you spend on classroom discipline and handling disruptive behavior? � Less than 15 minutes a day � 15 minutes to less than 30 minutes a day � 30 minutes to less than 45 minutes a day � 45 minutes to less than 1 hour a day � 1 hour to less than 2 hours a day � 2 hours or more a day
12. At this point in the school year, how would you rate the behavior of the children in your class? � Group misbehaves very frequently and is almost always difficult to handle. � Group misbehaves frequently and is often difficult to handle. � Group misbehaves occasionally. � Group behaves well. � Group behaves exceptionally well.
EAG K-3 Formative Assessment: Final Report September 2017 D-21
13. How often are the following materials or resources used in your class? (Select one response for each row.)
Not available Never
Once a month or
less
Two or three times
a month
Once or twice a week
Three or four times
a week Daily
Art materials □ □ □ □ □ □ □
Musical Instruments □ □ □ □ □ □ □
Costumes for creative dramatics/theater
□ □ □ □ □ □ □
Cooking or food related items □ □ □ □ □ □ □
Science equipment (for example, magnifying glass, scales, thermometers)
□ □ □ □ □ □ □
Thank you for your participation in this survey!
EAG K-3 Formative Assessment: Final Report September 2017 D-22
Exhibit D.1.3. Teacher survey: End of field test
K-3 Formative Assessment Final 2016-17 Teacher Survey
Section 1: Assessment Process and Materials
1. Which of the constructs did you administer? (Select all that apply.) � Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary
2. How did you collect evidence for this assessment? (Select all that apply) � Observation of classroom activities � Situation � Task � Other (please specify): ____________________
3. Did you use any of the (optional) Situations available on the assessment means form? � Yes � No
EAG K-3 Formative Assessment: Final Report September 2017 D-23
If you answered “yes” to question 3 Please rate the usefulness of the Situations you used. (Select one response for each row.)
Did not use
Very useful
Somewhat useful
Not useful
Not at all useful
a. Book Orientation – Help Me Read this Book □ □ □ □ □
b. Print Awareness – Help Me Read this Book □ □ □ □ □
c. Crossing Midline – All About Shapes □ □ □ □ □ d. Emotional Literacy – How Does the
Character Feel? □ □ □ □ □
e. Emotional Literacy – Why Do You Feel…? □ □ □ □ □ f. Emotional Literacy – Feelings in Social
Context □ □ □ □ □
g. Following Directions – Drawing Animals □ □ □ □ □ h. Grip and Manipulation – Friendship Wreath
Activity □ □ □ □ □ i. Hand Dominance – Friendship Wreath
Activity □ □ □ □ □
j. Gross Motor – Simon Says □ □ □ □ □
k. Letter Naming – Letter Naming Games □ □ □ □ □ l. Mathematical Patterns – Pattern
Calisthenics □ □ □ □ □
m. Mathematical Patterns – Pattern Game □ □ □ □ □ n. Object Counting – Hey! Let’s Go to the
Store! □ □ □ □ □
o. Problem Solving – Deserted Island □ □ □ □ □
p. Problem Solving – Building a Bridge □ □ □ □ □ q. Reading Comprehension – Individual
Reading Time □ □ □ □ □
r. Vocabulary – Group Discussion □ □ □ □ □
s. Vocabulary – Writing □ □ □ □ □
t. Vocabulary – Word Sort □ □ □ □ □
u. Writing – Free Writing Day □ □ □ □ □
EAG K-3 Formative Assessment: Final Report September 2017 D-24
4. Did you use any of the (optional) Tasks available on the assessment means form? � Yes � No
If you answered “yes” to question 4: Please rate the usefulness of the Tasks you used:
Did not use
Very useful
Somewhat useful
Not useful
Not at all useful
a. Crossing Midline Task □ □ □ □ □
b. Gross Motor Task □ □ □ □ □
c. Object Counting Task □ □ □ □ □
5. What did you use to document the evidence you collected? (Select all that apply.) � K-3 Technology platform � K-3 Evidence App � Single Child Documentation Form � Multiple Children Documentation Form � K-Entry Construct Checklist � Self-created form � Other (please specify): ____________________
6. Please rate the quality of the following assessment materials/resources:
High quality
Moderate quality
Low quality
Very low quality
a. Construct Progression Forms □ □ □ □
b. Assessment Means Forms □ □ □ □
c. Construct Modules □ □ □ □
d. K-3 Teacher Site □ □ □ □
e. K-3 Evidence App □ □ □ □
f. K-3 Technology Platform □ □ □ □
7. Thinking of all the constructs you assessed, what proportion of them were you able to assess as part of your everyday classroom activities? � All of them � Most of them � Some of them � None of them
EAG K-3 Formative Assessment: Final Report September 2017 D-25
If you selected “most of them” or “some of them” to question 7: Of the constructs you administered, which were you not able to assess as part of your everyday classroom activities? (Select all that apply.)
� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary
8. Did the order of the levels seem right for all of the progressions you assessed (that is, the construct progressed from least difficult to most difficult skill)? � Yes � No
EAG K-3 Formative Assessment: Final Report September 2017 D-26
If you answered “no” to question 8: Please identify the progressions for which the levels were not in the right order: (Select all that apply.)
� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary
EAG K-3 Formative Assessment: Final Report September 2017 D-27
9. For each of the constructs you administered, indicate how well the progression was able to capture children’s progress and development over time: (Select one response for each row.)
Very well
Somewhat well
Not well
Not at all well
Book Orientation □ □ □ □
Crossing Midline □ □ □ □
Emotion Expression □ □ □ □
Emotion Regulation Strategies □ □ □ □
Emotional Literacy □ □ □ □
Fine Motor: Grip and Manipulation □ □ □ □
Fine Motor: Hand Dominance □ □ □ □
Following Directions □ □ □ □
Gross Motor □ □ □ □
Letter Naming □ □ □ □
Mathematical Patterns □ □ □ □
Object Counting □ □ □ □
Perseverance □ □ □ □
Print Awareness □ □ □ □
Problem Solving □ □ □ □
Reading Comprehension □ □ □ □
Writing □ □ □ □
Vocabulary □ □ □ □
EAG K-3 Formative Assessment: Final Report September 2017 D-28
10. For each of the constructs you administered, please rate your level of confidence in being able to accurately place children on each of the progressions you assessed: (Select one response for each row.)
Very confident
Somewhat confident
Not confident
Not at all confident
Book Orientation □ □ □ □
Crossing Midline □ □ □ □
Emotion Expression □ □ □ □
Emotion Regulation Strategies □ □ □ □
Emotional Literacy □ □ □ □
Fine Motor: Grip and Manipulation □ □ □ □
Fine Motor: Hand Dominance □ □ □ □
Following Directions □ □ □ □
Gross Motor □ □ □ □
Letter Naming □ □ □ □
Mathematical Patterns □ □ □ □
Object Counting □ □ □ □
Perseverance □ □ □ □
Print Awareness □ □ □ □
Problem Solving □ □ □ □
Reading Comprehension □ □ □ □
Writing □ □ □ □
Vocabulary □ □ □ □
EAG K-3 Formative Assessment: Final Report September 2017 D-29
11. For each of the constructs you administered, how helpful was each of these constructs to your instruction? (Select one response for each row.)
Very helpful
Somewhat helpful
Not helpful
Not at all helpful
Book Orientation □ □ □ □
Crossing Midline □ □ □ □
Emotion Expression □ □ □ □
Emotion Regulation Strategies □ □ □ □
Emotional Literacy □ □ □ □
Fine Motor: Grip and Manipulation □ □ □ □
Fine Motor: Hand Dominance □ □ □ □
Following Directions □ □ □ □
Gross Motor □ □ □ □
Letter Naming □ □ □ □
Mathematical Patterns □ □ □ □
Object Counting □ □ □ □
Perseverance □ □ □ □
Print Awareness □ □ □ □
Problem Solving □ □ □ □
Reading Comprehension □ □ □ □
Writing □ □ □ □
Vocabulary □ □ □ □
If you answered “not helpful” or “not at all helpful” to constructs in question 11: Why wasn’t this construct helpful to your instruction: Book Orientation ________________________________________________________________________________________________________________________
Crossing Midline ________________________________________________________________________________________________________________________
EAG K-3 Formative Assessment: Final Report September 2017 D-30
Emotion Expression ________________________________________________________________________________________________________________________
Emotion Regulation Strategies ________________________________________________________________________________________________________________________
Emotional Literacy ________________________________________________________________________________________________________________________
Fine Motor: Grip and Manipulation ________________________________________________________________________________________________________________________
Fine Motor: Hand Dominance ________________________________________________________________________________________________________________________
Following Directions ________________________________________________________________________________________________________________________
Gross Motor ________________________________________________________________________________________________________________________
Letter Naming ________________________________________________________________________________________________________________________
Mathematical Patterns ________________________________________________________________________________________________________________________
EAG K-3 Formative Assessment: Final Report September 2017 D-31
Object Counting ____________________________________________________________
____________________________________________________________
Perseverance ________________________________________________________________________________________________________________________
Print Awareness ________________________________________________________________________________________________________________________
Problem Solving ________________________________________________________________________________________________________________________
Reading Comprehension ________________________________________________________________________________________________________________________
Writing ________________________________________________________________________________________________________________________
Vocabulary ________________________________________________________________________________________________________________________
12. During the K-3 assessment process, did you implement the assessment with any children with IEPs or 504 Plans? � Yes � No
If you answered “yes” to question 12: How many? (Enter a whole number only.) _______
EAG K-3 Formative Assessment: Final Report September 2017 D-32
If you answered “yes” to question 12: How difficult was it for you to assess this child or these children? Compared to using the assessment with the other children in the class, it was...
� Much more difficult � A little more difficult � Not any more difficult
If you answered “yes” to question 12: How useful was the information for informing his/her/their instruction?
� Very useful � Somewhat useful � Not useful � Not at all useful
13. During the K-3 assessment process, did you implement the assessment with any children who were English language learners? � Yes � No
If you answered “yes” to question 13: How many? (Enter a whole number only.) _______
If you answered “yes” to question 13: How difficult was it for you to assess this child or these children? Compared to using the assessment with the other children in the class, it was...
� Much more difficult � A little more difficult � Not any more difficult
If you answered “yes” to question 13: How useful was the information for informing his/her/their instruction?
� Very useful � Somewhat useful � Not useful � Not at all useful
14. Do you think that most teachers teaching at your grade level would be able to incorporate this assessment into their classroom routines? � Yes � No
EAG K-3 Formative Assessment: Final Report September 2017 D-33
If you answered “no” to question 14: Why not? ____________________________________________________________
____________________________________________________________
Section 2: Implementation Supports
15. What ways, if any, have you communicated or collaborated with other professionals about the K-3 Formative Assessment? (Check all that apply) � Professional Learning Communities (PLCs) � Grade-level meetings (e.g., K meeting or 3rd grade meeting) � Teacher-principal meetings � Informal discussions with other teachers � Other (please specify): ____________________ � I did not communicate or collaborate with others about this assessment.
16. For each of the communication and collaboration strategies you used, please tell us how helpful it was in supporting the implementation of the assessment. (Select one response for each strategy.)
Very helpful
Somewhat helpful
Not helpful
Not sure
Professional Learning Communities (PLCs) □ □ □ □ Grade-level meetings (e.g., K meeting or 3rd grade meeting) □ □ □ □
Teacher-principal meetings □ □ □ □
Informal discussions with other teachers □ □ □ □
Other (please specify): ______________ □ □ □ □
17. Did any other school staff provide data/evidence for you during the assessment process? (Select all that apply). � Paraprofessional or aide � Co-teacher � Special education teacher � Art, music or PE teacher � Therapist (speech, physical therapy, etc.) � Other (please specify): ____________________ � No other school staff provided data or evidence during the assessment process.
EAG K-3 Formative Assessment: Final Report September 2017 D-34
18. Did your school/administration provide you any of the following specifically to support the administration of the K-3 Assessment? (Select all that apply). � PLCs (professional learning community) � Increased time allocated for planning � Team meetings to discuss � Improved internet access/connectivity � Access to computer/laptop in classroom � Other (please specify): ____________________ � No, the school/administration provided none of this to support administration of the
assessments.
19. Would any of these supports have been helpful to you? (Select all that apply). � PLCs (professional learning community) � Increased time allocated for planning � Team meetings to discuss � Improved internet access/connectivity � Access to computer/laptop in classroom � Other (please specify): ____________________
20. Other than the supports you’ve already mentioned, what else would be helpful to support implementation of the K-3 Formative Assessment in the future?
___________________________________________________________________
Section 3: Opinions and Final Reflections
21. Based on your experience with the K-3 assessment process this school year, what challenges did you face in implementing this assessment? (Select all that apply) � Not enough information about the content of the assessment (e.g., uncertainty about the
purpose or use of the assessment, individual constructs) � Not enough information about expectations for implementing the assessment (e.g., the
cluster scheduling or dates for uploading evidence) � Not enough staff in the school were trained (e.g., it would have been helpful for
administrators to be trained; it would have been helpful to have more teachers in my school using the assessment).
� Not enough support in the classroom to collect the evidence (e.g., it would have been helpful for aides to help collect evidence)
� Lack of support and/or buy-in from my school administrators � Not enough time to implement � Lack of alignment with state standards � Not enough practice with the technology platform or app prior to using it in the classroom � Technical difficulties accessing the technology platform or app (e.g., accessing the
internet, downloading the app, internal server errors) � Problems using the technology platform or app (e.g., understanding how to use, locating
items on the platform/app, lack of support from technology provider) � Other (please specify): ____________________ � No challenges
EAG K-3 Formative Assessment: Final Report September 2017 D-35
22. What benefits have you experienced in relation to implementing the new assessment in your classroom? (Select all that apply) � Having deeper understanding of individual child’s skills and developmental needs � Having better understanding of the whole child, of development in multiple domains � Being better equipped to respond to individual student needs � Being better prepared to plan instruction that is developmentally appropriate � Having data all in one place to review for instructional planning � Having readily available data to share with support staff, administrators, and families � Having increased knowledge of how technology can be used for data collection and for
guiding instruction � Other (please specify): ____________________ � No real benefits
23. Overall, how helpful has the K-3 Formative Assessment been as a resource for informing instruction in your classroom? � Very helpful � Somewhat helpful � Not helpful � Not at all helpful
If you answered “not helpful” or “not at all helpful” to question 23: Why wasn’t this assessment helpful for informing your instruction? ____________________________________________________________
____________________________________________________________
Section 4: Tell Us About Your Class
24. As of today’s date... (Enter whole numbers only. Leave as zero if none.)
_______
25. How many children are currently enrolled in your class? _______
26. How many children in your class have a diagnosed disability? _______
27. How many English language learners (ELL) do you have in your class?
_______
Thank you for your participation in this survey!
EAG K-3 Formative Assessment: Final Report September 2017 D-36
Rev. 11/09/16
Exhibit D.1.4. Teacher focus group protocol
K-3 Formative Assessment Teacher Focus Group/Interview Protocol Winter/Spring 2017
Assessment Process 1. Let’s begin by talking about how you collected information on children for the assessment.
a. What did you do?
b. What worked well?
c. What didn’t work?
d. How could you improve your data collection process for the future?
2. Tell us about your planning process for including the assessment in your regular classroom
routine.
Technology Platform 3. What did you think about the technology (e.g., tech platform, evidence app)?
a. What did you like about it?
b. What didn’t you like about it?
4. Did you use the reports available on the tech platform?
a. (If yes/for those who did), what did you think about the reports? Were they useful?
b. Which of the reports, if any, were helpful for implementing this assessment and/or for
informing instruction?
Assessment and Instruction 5. Let’s talk about using the information from the assessment.
a. Were you able to use the information (e.g., evidence collected, learning status) to inform
your instruction?
b. How? Can you give us an example?
6. How would you envision sharing the information from this assessment with parents?
7. Do you feel the K-3 formative assessment construct progressions and learning statuses
provided you with meaningful information about your students’ abilities?
a. Can you share some examples?
EAG K-3 Formative Assessment: Final Report September 2017 D-37
Rev. 11/09/16
8. What other assessments did you use this school year?
a. Did you use other assessments in conjunction with this assessment?
i. (If so/for those who did), what did you use?ii. (If so/for those who did), how did you use it?
b. Possible probes
iii. Were they aligned with the K-3 formative assessment?iv. Did you use them as evidence for the K-3 assessment?
Implementation Supports 9. What extra resources and support, if any, did you receive for implementing the K-3
Assessment from your school, district or state?
Probe for:
a. Additional teachers/paraprofessionals in the classroom
b. Time allocated for planning
c. Internet access
d. Access to computer/laptop in classroom
10. Were there any other types of formal or informal help or assistance that you received or
provided to others that we have not yet discussed?
Probe for:
a. Help collecting evidences from other teachers/paraprofessionals
b. Professional Learning Communities (PLCs)
c. Grade-level meetings
d. Teacher-principal meeting
e. Informal discussions with other teachers
11. Is there anything that the leadership either in your school or district could have done to
make your experience with the assessment more successful?
EAG K-3 Formative Assessment: Final Report September 2017 D-38
Rev. 11/09/16
Opinions and Final Reflections
12. Now that you have gone through the entire assessment process, are there any changes to
the training or the materials that would help a teacher more effectively implement this
assessment?
Probe for:
a. Additional training
b. Additional materials or resources
c. Changes in the construct progressions
13. Did the implementation of the assessment positively or negatively affect other activities in
your classroom?
a. How so?
14. Is there anything else you would like to share about your experience implementing this
assessment?
EAG K-3 Formative Assessment: Final Report September 2017 D-39
Exhibit D.2.1. Fall K progression placement distributions by distribution type
Unimodal distribution with smooth tails
2%
15%18%
28% 26%
10%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E
Emotion Regulation Strategies - Grade K Fall (n = 787)
1%5%
9%
18%
37%
12%6% 6% 5%
1%0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I
Emotional Literacy - Grade K Fall (n = 803)
1% 3%
27%
51%
13%
4%0% 0%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F GWriting - Grade K (n = 826)
EAG K-3 Formative Assessment: Final Report September 2017 D-40
Unimodal distribution with smooth tail, skewed to higher end
1% 0% 3%11%
86%
0%
20%
40%
60%
80%
100%
Emerging A B C D
Book Orientation - Grade K Fall (n = 837)
0%5%
20%
75%
0%
20%
40%
60%
80%
100%
Emerging A B C
Crossing Midline - Grade K Fall (n = 829)
1%4% 5%
11%16% 18%
45%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E FFollowing Directions - Fall Grade K (n = 827)
EAG K-3 Formative Assessment: Final Report September 2017 D-41
Unimodal distribution with smooth tail, skewed to higher end (continued)
0%5%
22%
30%
42%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D
Grip & Manipulation: Grade K Fall (n = 833)
2%
23%
75%
0%
20%
40%
60%
80%
100%
Emerging A B C
Hand Dominance - Grade K Fall (n = 835)
0% 1% 1% 1%
9%
28% 29% 31%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G
Letter Naming - Grade K Fall (n = 835)
EAG K-3 Formative Assessment: Final Report September 2017 D-42
Unimodal distribution with smooth tail, skewed to lower end
Unimodal distribution with uneven tail(s)
2%
13%
29%34%
11%8%
2% 2% 1%0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H
Vocabulary - Grade K - Fall (n = 796)
0%
12%16% 15%
20%24%
7% 7%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G
Emotion Expression Grade K Fall (n = 805)
0% 0% 0% 1%6% 3%
14% 16%13%
46%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I
Gross Motor - Grade K Fall (n = 832)
EAG K-3 Formative Assessment: Final Report September 2017 D-43
Unimodal with uneven tail(s) (continued)
4%9%
3%
18%
28%
18%
10%6% 3%
0% 0% 0% 0% 0%0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I J K L M
Mathematical Patterns - Grade K Fall (n = 797)
0% 3% 4%6% 4% 5%
29%
21%
10%6%
12%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I J
Object Counting - Grade K Fall (n = 826)
1%
7%13%
10%
24%
11%14% 12%
9%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H
Perseverance - Grade K Fall (n = 795)
EAG K-3 Formative Assessment: Final Report September 2017 D-44
Unimodal with uneven tail(s) (continued)
0%4%
10%5%
12%16%
11%
41%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G
Print Awareness - Grade K Fall (n = 843)
1%6%
14%20%
28%
11%
3%10%
3% 2% 1%0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H I J
Problem Solving - Grade K Fall (n = 797)
2%8%
16%
26%
11%8% 9%
12%8%
0%
10%
20%
30%
40%
50%
60%
Emerging A B C D E F G H
Reading Comprehension - Grade K Fall (n = 784)
EAG K-3 Formative Assessment: Final Report September 2017 D-45
Exhibit D.2.2. Assessment results for 18 constructs by grade level and administration (fall, winter)
Construct, Grade Level and Administration
Children Assessed
Most Frequent Skill Level
Range of Skill Level
Placements Numeric Average
Standard Deviation
Book Orientation (A-D)
K-Fall 837 D-86% Emerging-D 4.8 0.5 K-Winter 733 D-98% Emerging-D 5.0 0.2
Hand Dominance (A-C) K-Fall 835 C-75% A-C 3.7 0.5 K-Winter 783 C-93% A-C 3.9 0.3
Crossing Midline (A-C) K-Fall 829 C-75% Emerging-C 3.7 0.6 K-Winter 782 C-87% Emerging-C 3.8 0.4 1-Fall 184 C-77% Emerging-C 3.7 0.6 1-Winter 193 C-85% A-C 3.8 0.4
Grip & Manipulation (A-D) K-Fall 833 D-42% A-D 4.1 0.9 K-Winter 784 D-82% A-D 4.8 0.6 1-Fall 197 D-64% A-D 4.5 0.8 1-Winter 193 D-79% A-D 4.7 0.7
Letter Naming (A-G) K-Fall 835 G-31% Emerging-G 6.7 1.2 K-Winter 783 G-73% B-G 7.6 0.8 1-Fall 196 G-66% E-G 7.6 0.7 1-Winter 192 G-84% E-G 7.8 0.5
Print Awareness (A-G) K-Fall 843 G-41% Emerging-G 6.3 1.9 K-Winter 782 G-84% A-G 7.7 0.9 1-Fall 185 G-94% E-G 7.9 0.3 1-Winter 181 G-99% E-G 8.0 0.2
Object Counting (A-J) K-Fall 826 F-29% Emerging-J 7.4 2.2 K-Winter 782 J-41% Emerging-J 9.4 1.9 1-Fall 185 J-46% A-J 9.7 1.7 1-Winter 193 J-66% A-J 10.3 1.2 2-Fall 126 J-78% C-J 10.6 0.9 2-Winter 115 J-93% D-J 10.9 0.7
Emotion Expression (A-G) K-Fall 805 E-24% Emerging-G 4.7 1.7 K-Winter 763 E-21% Emerging-G 5.8 1.8 1-Fall 195 E-19% A-G 5.4 1.9 1-Winter 193 G-39% A-G 6.5 1.6
EAG K-3 Formative Assessment: Final Report September 2017 D-46
Construct, Grade Level and Administration
Children Assessed
Most Frequent Skill Level
Range of Skill Level
Placements Numeric Average
Standard Deviation
2-Fall 126 G-32% A-G 6.4 1.6 2-Winter 115 G-39% A-G 6.7 1.5 3-Fall 138 E-31% A-G 6.6 1.3 3-Winter 135 G-55% A-G 7.3 1.2
Emotion Regulation Strategies (A-E)
K-Fall 787 C-28% Emerging-E 3.9 1.3 K-Winter 773 E-21% Emerging-E 4.5 1.3 1-Fall 197 C-28% Emerging-E 4.5 1.2 1-Winter 194 E-48% A-E 5.1 1.1 2-Fall 126 E-35% Emerging-E 4.7 1.3 2-Winter 115 E-48% Emerging-E 5.0 1.2 3-Fall 127 E-31% A-E 5.2 1.0 3-Winter 135 E-56% Emerging-E 5.3 1.0
Emotional Literacy (A-I) K-Fall 803 D-37% Emerging-I 5.2 1.8 K-Winter 768 D-27% Emerging-I 6.5 2.1 1-Fall 164 D-22% A-I 6.2 2.0 1-Winter 162 H-22% B-I 7.6 2.0 2-Fall 126 H-29% A-I 7.8 2.0 2-Winter 115 I-43% C-I 8.8 1.5 3-Fall 126 H-28% A-I 8.1 1.8 3-Winter 135 I-53% B-I 9.2 1.3
Following Directions (A-F) K-Fall 827 F-45% Emerging-F 5.7 1.5 K-Winter 782 F-67% A-F 6.3 1.1 1-Fall 194 F-58% A-F 6.0 1.4 1-Winter 193 F-56% A-F 6.2 1.2 2-Fall 126 F-62% B-F 6.3 1.1 2-Winter 115 F-74% B-F 6.5 0.9 3-Fall 138 F-49% A-F 5.8 1.5 3-Winter 135 F-73% C-F 6.6 0.8
Gross Motor (A-I) K-Fall 832 I-46% Emerging-I 8.6 1.7 K-Winter 782 I-68% A-I 9.3 1.3 1-Fall 185 I-51% A-I 8.8 1.6 1-Winter 186 I-69% C-I 9.5 0.9 2-Fall 126 I-84% A-I 9.7 1.0 2-Winter 115 I-95% A-I 9.9 0.8 3-Fall 138 I-83% A-I 9.7 0.9 3-Winter 123 I-93% F-I 9.9 0.4
EAG K-3 Formative Assessment: Final Report September 2017 D-47
Construct, Grade Level and Administration
Children Assessed
Most Frequent Skill Level
Range of Skill Level
Placements Numeric Average
Standard Deviation
Mathematical Patterns (A-M)
K-Fall 797 D-28% Emerging-H 5.0 1.8 K-Winter 770 F-26% Emerging-M 6.9 1.9 1-Fall 184 G-20% Emerging-K 7.3 2.1 1-Winter 193 H-37% C-L 8.7 1.7 2-Fall 126 H-26% A-M 9.0 2.5 2-Winter 115 I-31% B-M 9.6 2.6 3-Fall 127 J-20% B-L 9.0 3.0 3-Winter 124 J-27% Emerging-M 10.4 2.7
Perseverance (A-H) K-Fall 795 D-24% Emerging-H 5.5 2.1 K-Winter 770 H-22% Emerging-H 6.5 2.1 1-Fall 194 D-25% Emerging-H 5.9 1.9 1-Winter 193 G-27% B-H 7.2 1.7 2-Fall 126 G-45% A-H 7.4 1.7 2-Winter 115 G-40% B-H 7.7 1.4 3-Fall 138 G-27% A-H 6.8 1.9 3-Winter 135 H-45% A-H 7.7 1.6
Problem Solving (A-J) K-Fall 797 D-28% Emerging-J 5.0 2.0 K-Winter 766 D-28% Emerging-J 6.4 2.4 1-Fall 181 D-27% Emerging-J 6.1 1.8 1-Winter 193 G-28% A-J 7.7 2.1 2-Fall 126 G-27% A-J 7.9 2.1 2-Winter 115 G-28% C-J 8.9 1.8 3-Fall 138 G-33% B-J 7.4 1.8 3-Winter 127 I-35% A-J 8.9 1.6
Reading Comprehension (A-H)
K-Fall 784 C-26% Emerging-H 5.0 2.2 K-Winter 781 G, H-22% Emerging-H 6.7 2.0 1-Fall 196 H-27% C-H 7.3 1.5 1-Winter 193 H-34% A-H 7.6 1.4 2-Fall 126 H-32% B-H 7.4 1.5 2-Winter 115 H-57% B-H 8.1 1.4 3-Fall 138 H-37% Emerging-H 7.6 1.7 3-Winter 134 H-54% D-H 8.3 1.0
Vocabulary (A-H) K-Fall 796 C-34% Emerging-H 3.8 1.4 K-Winter 772 E-20% Emerging-H 5.2 2.0 1-Fall 184 B-23% Emerging-H 4.9 1.9
EAG K-3 Formative Assessment: Final Report September 2017 D-48
Construct, Grade Level and Administration
Children Assessed
Most Frequent Skill Level
Range of Skill Level
Placements Numeric Average
Standard Deviation
1-Winter 184 D-23% A-H 5.9 1.9 2-Fall 126 F-40% B-H 6.4 1.6 2-Winter 115 F-30% B-H 7.5 1.4 3-Fall 137 F-31% Emerging-H 6.2 1.7 3-Winter 135 F-41% A-H 6.9 1.5
Writing (A-G) K-Fall 826 C-51% Emerging-E 3.9 0.9 K-Winter 780 C-32% Emerging-G 5.1 1.3 1-Fall 195 D-39% Emerging-G 5.3 1.2 1-Winter 193 F-35% Emerging-G 6.0 1.1 2-Fall 126 E-42% Emerging-G 6.0 1.1 2-Winter 115 F-55% A-G 6.6 1.0 3-Fall 138 F-35% B-G 6.5 1.2 3-Winter 134 F-54% Emerging-G 6.9 1.0
EAG K-3 Formative Assessment: Final Report September 2017 D-49
Exhibit D.2.3 School readiness scores by construct and skill level: Mean total scores (K only) and mean IRT factor scores (K-3)
Book Orientation Total Scores (K only) IRT Factor Scores (K-3)
Emerging 5 61.2 5 -1.43 A 2 76.0 2 -0.91 B 21 75.1 22 -0.97 C 79 76.6 89 -0.88 D 647 95.7 750 -0.19
Crossing Midline Total Scores (K only) IRT Factor Scores (K-3)
Emerging 3 58.7 4 -1.22 A 34 75.7 52 -0.76 B 149 82.8 201 -0.54 C 568 96.7 766 -0.08
Emotion Expression Total Scores (K only) IRT Factor Scores (K-3)
Emerging 4 53.0 4 -1.84 A 90 77.1 119 -0.76 B 124 87.2 160 -0.44 C 110 87.2 159 -0.38 D 153 92.5 202 -0.23 E 178 100.7 295 0.17 F 49 103.1 154 0.57 G 46 115.4 170 0.80
Emotion Regulation Strategies Total Scores (K only) IRT Factor Scores (K-3)
Emerging 19 73.4 23 -0.92 A 112 78.8 138 -0.70 B 139 88.2 192 -0.33 C 209 92.4 325 -0.15 D 195 101.5 314 0.20 E 80 105.2 245 0.68
EAG K-3 Formative Assessment: Final Report September 2017 D-50
Emotional Literacy Total Scores (K only) IRT Factor Scores (K-3)
Emerging 7 66.0 7 -1.34 A 36 69.7 42 -1.07 B 65 78.9 79 -0.75 C 133 88.9 177 -0.42 D 279 92.7 364 -0.26 E 90 96.2 137 -0.08 F 45 103.5 94 0.35 G 49 104.7 111 0.38 H 40 114.5 133 0.80 I 10 120.1 76 1.07
Following Directions Total Scores (K only) IRT Factor Scores (K-3)
Emerging 5 71.2 5 -1.15 A 36 70.3 48 -0.97 B 35 80.6 57 -0.58 C 83 78.3 135 -0.51 D 125 86.8 178 -0.33 E 126 91.9 230 -0.03 F 344 102.8 632 0.35
Grip and Manipulation Total Scores (K only) IRT Factor Scores (K-3)
Emerging 0 0 A 39 75.9 50 -0.86 B 168 84.4 210 -0.57 C 234 93.5 299 -0.23 D 313 99.0 488 0.04
Gross Motor Total Scores (K only) IRT Factor Scores (K-3)
Emerging 3 70.7 3 -1.11 A 3 64.3 7 -0.95 B 1 64.0 1 -1.32 C 9 65.4 11 -1.19 D 34 73.9 51 -0.81 E 21 73.6 40 -0.61 F 110 84.2 154 -0.45 G 122 92.0 154 -0.31 H 94 94.7 160 -0.02 I 357 99.4 700 0.31
EAG K-3 Formative Assessment: Final Report September 2017 D-51
Hand Dominance Total Scores (K only) IRT Factor Scores (K-3)
Emerging 0 0 A 12 77.7 13 -0.78 B 176 82.4 199 -0.69 C 566 96.4 653 -0.17
Letter Naming Total Scores (K only) IRT Factor Scores (K-3)
Emerging 3 53.0 3 -1.73 A 8 56.1 8 -1.63 B 10 67.4 11 -1.25 C 6 72.0 7 -1.09 D 69 80.2 78 -0.78 E 207 86.8 249 -0.50 F 220 94.3 288 -0.24 G 231 104.1 397 0.21
Mathematical Patterns Total Scores (K only) IRT Factor Scores (K-3)
Emerging 28 64.1 31 -1.33 A 74 83.3 77 -0.66 B 26 83.2 38 -0.49 C 139 87.1 163 -0.49 D 198 93.0 274 -0.25 E 144 97.9 160 -0.18 F 79 102.1 141 0.17 G 43 103.5 108 0.31 H 23 118.6 109 0.72 I 0 0 17 0.76 J 0 0 65 1.08 K 0 0 38 1.27 L 0 0 12 1.37 M 0 0 1 1.38
Object Counting Total Scores (K only) IRT Factor Scores (K-3)
Emerging 3 44.3 3 -2.11 A 23 65.2 25 -1.25 B 27 75.6 31 -0.94 C 51 77.7 54 -0.85 D 34 83.1 37 -0.66 E 40 80.6 50 -0.74 F 215 91.7 247 -0.38 G 155 98.0 187 -0.14 H 66 99.2 136 0.09 I 48 104.7 94 0.16 J 92 107.0 281 0.57
EAG K-3 Formative Assessment: Final Report September 2017 D-52
Perseverance Total Scores (K only) IRT Factor Scores (K-3)
Emerging 9 54.3 10 -1.59 A 56 73.6 66 -0.93 B 103 81.8 128 -0.61 C 72 87.3 121 -0.41 D 181 90.1 261 -0.27 E 71 96.8 139 0.05 F 99 101.0 164 0.18 G 94 104.7 211 0.56 H 69 111.0 153 0.79
Print Awareness Total Scores (K only) IRT Factor Scores (K-3)
Emerging 1 78.0 1 -0.85 A 30 74.0 30 -0.99 B 84 75.8 84 -0.93 C 39 80.5 40 -0.78 D 93 85.2 104 -0.61 E 130 89.7 143 -0.46 F 74 96.4 106 -0.26 G 303 103.9 530 0.15
Problem Solving Total Scores (K only) IRT Factor Scores (K-3)
Emerging 10 54.9 11 -1.54 A 50 74.4 53 -0.94 B 109 79.4 128 -0.76 C 153 87.6 181 -0.49 D 205 95.4 320 -0.13 E 83 100.2 154 0.09 F 27 105.0 53 0.38 G 73 106.5 195 0.55 H 20 115.2 79 0.94 I 16 115.1 38 1.01 J 8 130.5 30 1.30
Reading Comprehension Total Scores (K only) IRT Factor Scores (K-3)
Emerging 18 57.2 20 -1.49 A 59 77.2 60 -0.88 B 121 83.3 130 -0.65 C 197 88.8 218 -0.49 D 87 94.2 145 -0.09 E 58 97.4 104 -0.04 F 66 101.1 145 0.24 G 95 110.0 218 0.52 H 53 111.1 204 0.77
EAG K-3 Formative Assessment: Final Report September 2017 D-53
Vocabulary Total Scores (K only) IRT Factor Scores (K-3)
Emerging 10 57.5 14 -1.35 A 101 76.6 117 -0.83 B 217 90.3 297 -0.34 C 257 94.7 321 -0.22 D 78 101.1 139 0.09 E 59 106.6 143 0.40 F 16 113.0 118 0.94 G 12 108.6 67 0.95 H 4 116.8 27 1.08
Writing Total Scores (K only) IRT Factor Scores (K-3)
Emerging 10 59.4 12 -1.36 A 21 74.5 21 -0.97 B 211 82.5 240 -0.68 C 384 95.6 479 -0.21 D 98 104.2 213 0.13 E 30 116.9 162 0.68 F 0 0 126 0.96 G 0 0 32 1.34
EAG K-3 Formative Assessment: Final Report September 2017 D-54
Exhibit D.2.4. Teacher survey: Kindergarten entry performance levels
K Teacher Survey 2017 Kindergarten Entry Expectations Welcome! The purpose of this survey is to collect teacher input on expectations for children’s skills and abilities at kindergarten entry. You are being asked to provide the SRI team with valuable information on your expectations for children when they begin kindergarten. The information will help K teachers know where to start the assessment at the beginning of the school year and better describe and report where children are at kindergarten entry.
We recognize that incoming kindergarten children show a range of skills so we are interested in what you consider are the typical, or average range of expected skills for children entering the kindergarten classroom and the lower end of the range for expected skills at kindergarten entry.
Directions:
On the following screens, you will be presented with the 18 construct progressions and asked to mark two points on each progression:
Typical: For the first point, select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: For the second point, mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
For example,
• If you think Level D is the skill level of a typical child at kindergarten entry, select the button next to Skill D in the column labeled "Typical".
• If you think Level B is the skill level of a child who is so far below the skill level of the typical child at kindergarten entry that she is likely to experience some difficulties at kindergarten entry in this area, then you would mark B in the column labeled "Far Below". In this example, a child at skill level A would also be likely to experience difficulties but we want you to mark the highest level at which children are likely to experience some difficulty at kindergarten entry.
• If you think that the “likely to struggle” level is lower than Level A, mark that choice for the column labeled "Far Below".
It is important that you answer each question. Your responses will help us generate reporting structures and guidance for teachers using this tool in the future.
If you need additional information (e.g., examples) on any of the progressions, feel free to consult the construct progression documents in your binder or on the Teacher Site. And if you have any questions, please contact [email protected] or [email protected].
EAG K-3 Formative Assessment: Final Report September 2017 D-55
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Domain: Language Development and Communication Construct: Book Orientation
Typical Far Below Skills: Book Orientation
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Holds or otherwise demonstrates awareness that a book is an object with
pages
□ □ B. Eventually holds the book upright (so pictures and text are right side up,
irrespective of front to back orientation)
□ □ C. Holds the book upright, opens it from the front cover, and turns pages (not
always one by one)
□ □ D. Holds the book upright, and starting at the front, turns the pages in order, one
page at a time
Construct: Print Awareness
Typical Far Below Skills: Print Awareness
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Attends only to pictures and ignores text.
□ □ B. Indicates the general area of text and/or where we read words (making a
distinction between pictures and text).
□ □ C. Indicates where to begin reading and that lines of text are read from top to
bottom (where reading begins, top to bottom directionality).
□ □ D. Indicates that lines of text are read from left to right (e.g., tracking text) (left to
right directionality).
□ □ E. Indicates that lines of text are read from left to right; and at the end of each
line, returns to the beginning of the next line of text (e.g., tracking text) (return sweep).
□ □ F. Distinguishes between a letter and a word on a page of text (excluding the
words a, A, and I).
□ □ G. Indicates one word on the page for each word read aloud (concept of word).
EAG K-3 Formative Assessment: Final Report September 2017 D-56
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Letter Naming
Typical Far Below Skills: Letter Naming
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Distinguishes letters from pictures, shapes, or numerals
□ □ B. Distinguishes own first name from other names or other words (e.g., on
folders, name charts, among other words or names)
□ □ C. Locates, talks about, or asks questions about letters in one’s environment
□ □ D. Identifies (names or selects) one or more letters in own first name
□ □ E. Identifies (names or selects) some upper case or lowercase letters (in
additional to letters found in own first name)
□ □ F. Identifies (names or selects) each of the 26 letters in some form (may be a
combination of uppercase and lowercase)
□ □ G. Identifies (names or selects) all 52 letters in uppercase and lowercase form.
Construct: Following Directions
Typical Far Below Skills: Following Directions
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Completes one-step directions when support is provided in the form of
visuals, gestures, or modeling.
□ □ B. Completes one-step directions (without support provided).
□ □ C. Partially completes two-step directions (without support provided) by
completing only one step or completing the steps out of order.
□ □ D. Completes two-step directions in the correct order (without support
provided).
□ □ E. Partially completes three-step directions (without support provided) by
completing two steps, or completing the steps out of order.
□ □ F. Completes three-step directions in the correct order (without support
provided).
EAG K-3 Formative Assessment: Final Report September 2017 D-57
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Writing
Typical Far Below Skills: Writing
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Dictates ideas, information, or stories to have them written down.
□ □ B. Uses drawings, scribbles or letter-like forms to express ideas, information or
stories.
□ □ C. Writes letters, words, or phrases to label drawings or express ideas,
information or stories.
□ □ D. Writes, demonstrating an understanding of purpose or audience or both.
□ □ E. Writes two or more related ideas, pieces of information, or events.
□ □ F. Writes using a genre-specific organizational structure.
□ □ G. Revises own genre-specific writing to provide clarity to the reader.
Construct: Reading Comprehension
Typical Far Below Skills: Reading Comprehension
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Listens to a story or informational text (with or without pictures) and
demonstrates an understanding of the text.
□ □ B. Views pictures or images and demonstrates an understanding of the story or
information presented.
□ □ C. Pretends to read a familiar book, retelling the story line or information
presented in the text.
□ □ D. Reads without noticing when the reading does not make sense.
□ □ E. Monitors reading, noticing that the reading does not make sense, without
attempting to repair meaning.
□ □ F. Monitors reading, noticing that the reading does not make sense, and
attempts unsuccessfully to repair meaning.
□ □ G. Monitors reading and attempts to repair meaning by asking for and using
comprehension strategies.
□ □ H. Monitors reading successfully using self-selected comprehension strategies
to maintain and improve meaning.
EAG K-3 Formative Assessment: Final Report September 2017 D-58
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Vocabulary
Typical Far Below Skills: Vocabulary
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Connects a word to its related concept.
□ □ B. Uses school-related words appropriately in one or more contexts.
□ □ C. Uses multiple words that are closely related in meaning to describe a single
school-related concept.
□ □ D. Determines the appropriate meanings of multiple-meaning words by using
context.
□ □ E. Uses at least two meanings of multiple-meaning words.
□ □ F. Determines the meaning of unfamiliar words by using known words and
knowledge of word structure (e.g., roots, affixes, compound words)
□ □ G. Recognizes both literal meanings and non-literal meanings (e.g., language
that is nuanced, more precise or descriptive, has shades of meaning) of words and phrases.
□ □ H. Uses non-literal meanings of words and phrases (e.g., language that is
nuanced, more precise or descriptive, has shades of meaning)
Domain: Cognitive Development Construct: Object Counting
Typical Far Below Skills: Object Counting
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Displays early counting behavior with 4-6 objects arranged in a line (i.e.,
says or indicates some number words while pointing to the objects but does not count all of the objects correctly).
□ □ B. Consistently counts 4-6 objects in a line correctly.
□ □ C. Consistently counts 4-6 objects in a scattered arrangement correctly.
□ □ D. Knows the last number word used while counting is the total quantity and
that the value of a collection of objects does not change unless objects are added or removed.
□ □ E. Consistently counts out 4-6 objects from a set of more than 10 objects.
□ □ F. Consistently counts out 8-10 objects from a set of more than 10 objects.
□ □ G. Correctly counts 18-20 objects in a scattered arrangement.
□ □ H. Uses beginning strategies (i.e., counting again from one or by repeating the
cardinal number in the original set and then counting on) to find the new total when one object is added to a set of 6-10 objects.
□ □ I. Procedures the correct number (without pause) when one object is added to
a set of 6-10 objects.
□ □ J. Produces the correct number automatically (without pause) when two
objects are added to a set of 6-10 objects.
EAG K-3 Formative Assessment: Final Report September 2017 D-59
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Mathematical Patterns
Typical Far Below Skills: Mathematical Patterns
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Duplicates sequential AB patterns with the same materials.
□ □ B. Duplicates sequential three-element patterns (e.g., ABB, ABC) with the
same materials.
□ □ C. Extends sequential AB patterns at least one unit.
□ □ D. Extends sequential three-element patterns (e.g., ABB, ABC) at least one
unit.
□ □ E. Duplicates sequential patterns (e.g., AB, ABC, AABB) using materials
different from those used in the model pattern (pattern abstraction).
□ □ F. Identifies repeating unit in sequential patterns.
□ □ G. Extends by at least one step or determines the missing step in spatial or
numerical growing patterns.
□ □ H. Communicates a recursive rule governing the next step in spatial or
numerical growing patterns.
□ □ I. Creates or enters data in a t-chart to document the relationship between the
ordinal position of a step in a growing pattern (i.e., first, second, third) and an important feature of the step.
□ □ J. Applies the relationship between the two variables in a t-chart to extend a
numerical growing pattern by at least one step.
□ □ K. Communicates a one-operation functional rule governing spatial or
numerical growing patterns and uses it to determine a far step. Note: a “far step” is more than 5 steps from the last represented step in the pattern.
□ □ L. Communicates a two-operation functional rule governing spatial or
numerical growing patterns and uses it to determine a far-step. Note: a “far step” is more than 5 steps from the last represented step in the patterns.
□ □ M. Creates an equation that symbolizes a functional rule governing a spatial or
numerical growing pattern.
EAG K-3 Formative Assessment: Final Report September 2017 D-60
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Problem Solving
Typical Far Below Skills: Problem Solving
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Acknowledges that a problem exists without attempting to solve it (this may
manifest as a child moving away from the problem).
□ □ B. Attempts to solve a problem by mimicking the motions and procedures of
others or seeking general support from others very early in the problem solving process (e.g., how do I do this? I don’t know what to do).
□ □ C. Attempts to solve a problem using random trial and error.
□ □ D. Solves or attempts to solve a familiar problem using procedures learned in
previous problem solving experiences.
□ □ E. Solves or attempts to solve a novel problem using procedures learned in
previous problem solving experiences without demonstrating knowledge of why the procedure is or is not successful
□ □ F. States a hypothesis about how to solve a novel problem, using both
concepts and procedures.
□ □ G. Solves or attempts to solve a novel problem by connecting concepts and
using familiar procedures.
□ □ H. Generates and explains an alternate problem solving approach (including
when an approach is not working).
□ □ I. Generates and explains multiple approaches for solving a problem.
□ □ J. Provides justification for why a chosen self- or peer-generated problem
solving approach might be the most efficient one for solving a problem.
EAG K-3 Formative Assessment: Final Report September 2017 D-61
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Domain: Approaches to Learning
Construct: Perseverance
Typical Far Below Skills: Perseverance
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Begins to persevere in familiar and/or interesting classroom activities but
quickly discontinues involvement in the activity after one or very few attempts.
□ □ B. Perseveres in familiar and/or interesting classroom activities, but
discontinues involvement after persisting through trials for a short period of the allotted time.
□ □ C. Perseveres in familiar and/or classroom activities and works through most
trials, but discontinues persisting before the end of the allotted time.
□ □ D. Perseveres in familiar and/or interesting classroom activities throughout the
activity, persisting through trials throughout the allotted time.
□ □ E. Perseveres in classroom activities that are novel and/or moderately difficult
for the child, flagging in effort at points of challenge in the activity.
□ □ F. Perseveres within the allotted time in classroom activities that are novel
and/or moderately difficult for the child sometimes persisting and maintaining effort at points of challenge in the activity.
□ □ G. Perseveres throughout the allotted time in classroom activities that are novel
and/or moderately difficult for the child, persisting and maintaining effort at points of challenge in the activity.
□ □ H. Perseveres throughout the allotted time in classroom activities that are very
difficult for the child, persisting and maintaining effort at points of challenge in the activity.
Domain: Social-Emotional Development
Construct: Emotion Expression
Typical Far Below Skills: Emotion Expression
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Acts, talks or reacts to an experience or stimulus without demonstrating
awareness of the reactions of others around them.
□ □ B. Shows awareness that his or her own actions, words, and reactions to an
experience or stimulus has an impact on others.
□ □ C. Makes choices to express emotions in ways that get their needs met.
□ □ D. Begins to express emotions in ways that support building relationships.
□ □ E. Generally expresses emotions in ways that support building relationships.
□ □ F. Reflects about emotions and the consequences of actions with support from
the teacher.
□ □ G. Independently reflects about emotions and the consequences of actions.
EAG K-3 Formative Assessment: Final Report September 2017 D-62
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Emotional Literacy
Typical Far Below Skills: Emotional Literacy
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Expresses verbally or behaviorally a range of emotions.
□ □ B. Shows an awareness of his or her verbal and/or behavioral expression of
emotions.
□ □ C. Identifies emotions in self.
□ □ D. Identifies emotions in others.
□ □ E. Indicates a response to an event can include one or more emotions,
including conflicting emotions, within themselves or another person.
□ □ F. Indicates the same event can cause different people to experience different
emotions.
□ □ G. Indicates knowledge of how the social context relates to his/her own and
others’ emotion experience.
□ □ H. Interprets the emotional experience of others using the social context (e.g.,
provides justification that includes social information and inferences about others’ intentions).
□ □ I. Predicts the emotional experience of other using the social context (e.g.,
provides justification that includes social information and inferences about others’ intentions).
Construct: Emotion Regulation Strategies
Typical Far Below Skills: Emotion Regulation Strategies
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. When offered strategies for regulating the expression of emotions, begins to
use the strategies offered with continual support.
□ □ B. When offered strategies for regulating the expression of emotions,
consistently uses the strategies offered with minimal support.
□ □ C. Sometimes uses learned strategies independently to regulate the
expression of emotions during routine activities.
□ □ D. Consistently uses learned strategies independently to regulate the
expression of emotions during routine activities.
□ □ E. Independently uses learned strategies for regulating emotions during
complex contextual transitions (e.g., exposed to multiple stimuli simultaneously), to accomplish a different or new type of task, because of interruptions, or because of changes in the daily routines.
EAG K-3 Formative Assessment: Final Report September 2017 D-63
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Domain: Health and Physical Development
Construct: Hand Dominance
Typical Far Below Skills: Hand Dominance
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Uses no established dominance for lead/dominant hand (switching still
continues).
□ □ B. Uses established dominant hand.
□ □ C. Manipulates with dominant hand with assistance from other hand.
Construct: Grip and Manipulation
Typical Far Below Skills: Grip and Manipulation
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Uses early fine motor skills (e.g., fisted grip, palmar grasp, or early scissor
grip) to hold and/or manipulate items, with whole arm movement.
□ □ B. Uses a more refined grip (e.g., using thumb and finger [pincer grip] or tripod
grip) to hold and/or manipulate objects with whole arm movement and increased stability from the shoulder.
□ □ C. Uses refined wrist and finger movement, beginning to transfer control of
movement from the shoulder to the elbow.
□ □ D. Uses hands with minimal elbow movement and primary control from wrist
and fingers.
Construct: Crossing Midline
Typical Far Below Skills: Crossing Midline
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Movement is isolated to one side of the midline (the invisible line running
from our head to our toes, dividing the body into left and right halves).
□ □ B. Inconsistently crosses midline.
□ □ C. Consistently crosses midline.
EAG K-3 Formative Assessment: Final Report September 2017 D-64
Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.
Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.
Construct: Gross Motor
Typical Far Below Skills: Gross Motor
□ □ “May experience significant difficulties in the classroom” is lower than Level A.
□ □ A. Walks on a flat foot or walks on the toes.
□ □ B. Walks by distributing weight from heel to toe WITHOUT arms and legs
moving in opposition.
□ □ C. Walks by distributing weight from heel to toe WITH arms and legs moving in
opposition.
□ □ D. Maintains balance and control when running.
□ □ E. Maneuvers around objects and people while running
□ □ F. Emerging galloping skills.
□ □ G. Gallops smoothly.
□ □ H. Hops forward smoothly on one foot.
□ □ I. Skips smoothly.
If you have additional comments about this survey in general or any of the progressions specifically, please provide feedback below. ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Thank you for completing this survey!
Your responses will be aggregated with others to inform how the K-3 Formative Assessment Consortium sets performance expectations for children at the beginning of kindergarten. If you have any questions, please contact [email protected] or [email protected].
EAG K-3 Formative Assessment: Final Report September 2017 D-65
Exhibit D.2.5. Kindergarten entry performance levels: Sample domain expert questionnaire
Instructions – [CONSTRUCT NAME] Please review the progression and construct charts in this packet, one progression at a time,
and respond to the questions. We ask that you provide a rationale for your responses that
includes your expert opinion as well as any references you used to formulate your response.
Learning/Construct Progression
To refresh your memory, we have included a copy of the construct progression in the email you
received with this packet. It describes the background and rationale for the progression as well
as the overarching understandings, skills in the progression, performance descriptors and
examples. We recommend you review the progression before you respond to the questions
below.
Charts
The first chart (Teacher Survey) illustrates where teachers would expect K children to fall on the
progression given their expectations in the first week of school. There are two categories/bars
on this chart: the blue bars represent the skill level of an average or typical incoming K child in
the first week of school, a child who is meeting classroom expectations and therefore may not
need additional support from the teacher outside of regular interactions. The orange bars
represent the skill level of a child who may need additional supports in order to meet classroom
expectations at the beginning of kindergarten.
The second chart (Child Assessment) illustrates actual child placements on the progression
from the Fall 2016 field test. These data were collected in the first 8 to 10 weeks of kindergarten
and represent where teachers actually placed children on the progression. Note that these
placements were based on observations that typically occurred after the first week of school.
Performance Level Descriptors
The research team has defined two performance levels that may help teachers use the
assessment and report aggregate information for groups of students. Because the purpose of
the K-3 Formative Assessment is to inform instruction, the performance levels have been
defined related to instructional expectations for the class. Please reference these descriptions to
inform your responses to the questions below.
Developing – child does not yet exhibit the knowledge and skills expected of children at
entry to kindergarten and may need additional support or assistance to fully participate in
kindergarten curriculum activities and instruction.
EAG K-3 Formative Assessment: Final Report September 2017 D-66
Demonstrating – child exhibits the knowledge and skills expected of children at entry to
kindergarten to fully participate in kindergarten curriculum activities and instruction.
Mathematical Patterns Teacher Survey Summary
When K teachers were asked “Indicate the skill level for a typical or average student in the first
week of kindergarten,” the majority of teachers (65%) responded skill level A as typical for K
entry. (blue bars)
When teachers were asked “indicate the skill level at which you would consider an incoming
kindergarten student far below typical”, 86% of responses for this construct were below skill
level A. (green bars)
Child Assessment Field Test Data
During the first 8 – 10 weeks of Kindergarten, teachers varied in their placement of children on
the progression with a majority of children (59%) landing between skills C and E.
4%
65%
7%13%
6% 4%
86%
13%0% 0% 0% 0%
0%
20%
40%
60%
80%
100%
BelowA
A B C D E F G H I J K L M
Mathematical Patterns - Teacher Survey
Typical Far Below
3%
9%
3%
16%
26%
17%
9%5%
3%0% 0% 0% 0% 0%
7%
0%
5%
10%
15%
20%
25%
30%
Emerging A B C D E F G H I J K L M Not assessed
Mathematical Patterns - Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-67
Your Activity Based on your knowledge of child development coupled with the data above, please respond to
the following questions for this construct. Provide a rationale with references where necessary
to inform your response. You may respond directly within this document.
1. In your opinion, which levels on the progression would you place into the “Demonstrating”
performance level?
Demonstrating – child exhibits the knowledge and skills expected of children at entry to
kindergarten to fully participate in kindergarten curriculum activities and instruction.
Please provide your rationale.
2. In your opinion, which levels on the progression would you place into the “Developing”
performance level?
Developing – child does not yet exhibit the knowledge and skills expected of children at
entry to kindergarten and may need additional support or assistance to fully participate in
kindergarten curriculum activities and instruction.
Please provide your rationale.
3. In your opinion are there levels on the progression where the performance levels overlap?
a. If so, which ones?
b. Why?
EAG K-3 Formative Assessment: Final Report September 2017 D-68
Exhibit D.2.6. Final K entry performance levels compared to teacher survey responses and K fall assessment data
3%
21%
50%
26%
51%
37%
10%1%1% 3%
11%
86%
0%
20%
40%
60%
80%
100%
Below A A B C D
Book Orientation
Teacher survey:Typical Teacher survey: Far Below Child Assessment
Developing Demonstrating
1%12%
64%
23%
46%53%
1%5%
20%
75%
0%
20%
40%
60%
80%
100%
Below A A B C
Crossing Midline
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-69
1%
38%
20%25%
10%4% 1%
64%
33%
3%12%
16% 15%20% 24%
7% 7%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G
Emotion Expression
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
51%
35%
14%
78%
22%
2%15% 18%
28% 26%
10%
0%
20%
40%
60%
80%
100%
Below A A B C D E
Emotion Regulation Strategies
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
26%16%
38%
16%
1% 3%
59%
33%
6% 1%1% 5% 9%18%
37%
12%6% 6% 5% 1%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H I
Emotional Literacy
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-70
17%
33% 32%
14%
1% 1%
49% 46%
4%1% 4% 5%11%
16% 18%
45%
0%
20%
40%
60%
80%
100%
Below A A B C D E F
Following Directions
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
28%
61%
10%1%
57%
43%
5%
22%30%
42%
0%
20%
40%
60%
80%
100%
Below A A B C D
Grip & Manipulation
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-71
4% 7% 10%22% 19% 20%
12%4% 1%
19%
48%
17%
4% 7%1% 3%1% 6% 3%
14% 16% 13%
46%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H I
Gross Motor
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
10%
70%
20%
43%
57%
2%
23%
75%
0%
20%
40%
60%
80%
100%
Below A A B C
Hand Dominance
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
7%
20%
3%
26%
40%
3% 1%
49%
36%
10%1% 3%1% 1% 1%
9%
28% 29% 31%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G
Letter Naming
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-72
4%
65%
7%13%
6% 4%
87%
13%4%
9%3%
18%28%
18%10% 6% 3%
0%
20%
40%
60%
80%
100%
BelowA
A B C D E F G H I J K L M
Mathematical Patterns
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
25%
41%
13%4% 6% 9%
1% 1%
68%
30%
1%3% 4% 6% 4% 5%
29%21%
10% 6%12%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H I J
Object Counting
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-73
1%
33% 35%
17%12%
1%
66%
31%
3%1%7%
13% 10%
24%
11% 14% 12% 9%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H
Perseverance
Typical Far Below Child Assessment
23%
40%
20%14%
1% 1%
67%
29%
4%4%10%
5%12%
16%11%
41%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G
Print Awareness
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
1%
35%42%
16%6%
68%
31%
1%1%6%
14%20%
28%
11%3%
10%3% 2% 1%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H I J
Problem Solving
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-74
4%
19%
46%
29%
1%
75%
25%
2%8%
16%26%
11% 8% 9% 12% 8%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H
Reading Comprehension
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
1%
45% 48%
4% 1%
82%
18%
2%13%
29% 34%
11% 8%2% 2% 1%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G H
Vocabulary
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
14%
78%
6% 1%
69%
28%
3%1% 3%
27%
51%
13%4%
0%
20%
40%
60%
80%
100%
Below A A B C D E F G
Writing
Teacher Survey:Typical Teacher survey:Far Below Child Assessment
EAG K-3 Formative Assessment: Final Report September 2017 D-75
Exhibit D.2.7. Percentage of kindergarten students developing and demonstrating expected skill levels in fall based on expert recommended performance levelsa
K Children
Assessed in Fall K Entry Performance Levels
Book Orientation 837 3 97
Crossing Midline 829 6 94
Emotion Expression 805 12 88
Emotion Regulation Strategies 787 2 98
Emotional Literacy 803 6 94
Following Directions 827 55 45
Grip & Manipulation 833 5 95
Gross Motor 832 8 92
Hand Dominance 835 2 98
Letter Naming 835 3 97
Mathematical Patterns 797 34 66
Object Counting 826 17 83
Perseverance 795 1 99
Print Awareness 843 4 96
Problem Solving 797 42 58
Reading Comprehension 784 10 90
Vocabulary 796 2 98
Writing 826 4 96 aPerformance levels for K entry were set as of the first week in school. Children were actually assessed any time during the first 12 weeks of school in the fall. Actual performance should in most cases exceed expected performance levels.
EAG K-3 Formative Assessment: Final Report September 2017 D-76
Exhibit D.2.8. Mean progression placements for children assessed in fall and winter
Children
Assessed Fall Winter Difference Standard Deviation Probability
Book Orientation (A-D) Grade K 763 4.8 5.0 0.17 0.53 .0001
Hand Dominance (A-C) Grade K 777 3.7 3.9 0.18 0.47 .0001
Crossing Midline (A-C) Grade K 773 3.7 3.8 0.17 0.55 .0001 Grade 1 182 3.7 3.8 0.13 0.52 .001
Grip & Manipulation (A-D)
Grade K 776 4.1 4.8 0.64 0.83 .0001 Grade 1 193 4.5 4.7 0.21 0.80 .001
Letter Naming (A-G) Grade K 782 6.7 7.6 0.87 1.04 .0001 Grade 1 192 7.6 7.8 0.23 0.51 .0001
Print Awareness (A-G) Grade K 780 6.3 7.7 1.37 1.71 .0001 Grade 1 181 7.9 8.0 0.06 0.25 .005
Object Counting (A-J) Grade K 773 7.4 9.4 1.95 1.95 .0001 Grade 1 182 9.7 10.3 0.64 1.22 .0001 Grade 2 115 10.6 10.9 0.23 0.59 .0001
Emotion Expression (A-G)
Grade K 744 4.7 5.8 1.08 1.58 .0001 Grade 1 193 5.4 6.5 1.10 1.56 .0001 Grade 2 115 6.4 6.7 0.29 1.51 .05 Grade 3 134 6.6 7.3 0.64 1.03 .0001
Emotion Regulation Strategies (A-E)
Grade K 735 3.9 4.5 0.61 1.22 .0001 Grade 1 194 4.5 5.1 0.60 1.22 .0001 Grade 2 115 4.7 5.0 0.24 1.10 .05 Grade 3 124 5.2 5.3 0.12 0.95 NS
Emotional Literacy (A-I) Grade K 747 5.2 6.5 1.30 1.98 .0001 Grade 1 162 6.2 7.6 1.38 1.58 .0001 Grade 2 115 7.8 8.8 0.87 1.59 .0001 Grade 3 124 8.1 9.2 1.08 1.37 .0001
EAG K-3 Formative Assessment: Final Report September 2017 D-77
Children
Assessed Fall Winter Difference Standard Deviation Probability
Following Directions (A-F)
Grade K 773 5.7 6.3 0.56 1.26 .0001 Grade 1 192 6.0 6.2 0.18 1.25 .05 Grade 2 115 6.3 6.5 0.16 0.74 .05 Grade 3 135 5.8 6.6 0.75 1.13 .0001
Gross Motor (A-I) Grade K 779 8.6 9.3 0.70 1.33 .0001 Grade 1 175 8.8 9.5 0.59 1.29 .0001 Grade 2 115 9.7 9.9 0.22 0.69 .001 Grade 3 123 9.7 9.9 0.20 0.83 .01
Mathematical Patterns (A-M)
Grade K 733 5.0 6.9 1.92 1.91 .0001 Grade 1 182 7.3 8.7 1.43 1.71 .0001 Grade 2 115 9.0 9.6 0.79 1.65 .0001 Grade 3 124 9.0 10.4 1.32 1.76 .0001
Perseverance (A-H) Grade K 742 5.5 6.5 1.00 1.97 .0001 Grade 1 192 5.9 7.2 1.30 1.59 .0001 Grade 2 115 7.4 7.7 0.24 1.42 NS Grade 3 135 6.8 7.7 0.88 1.71 .0001
Problem Solving (A-J) Grade K 729 5.0 6.4 1.32 2.14 .0001 Grade 1 180 6.1 7.7 1.57 1.87 .0001 Grade 2 115 7.9 8.9 1.18 1.58 .0001 Grade 3 127 7.4 8.9 1.49 1.52 .0001
Reading Comprehension (A-H)
Grade K 731 5.0 6.7 1.64 2.09 .0001 Grade 1 193 7.3 7.6 0.33 1.29 .001 Grade 2 115 7.4 8.1 0.71 1.31 .0001 Grade 3 134 7.6 8.3 0.63 1.19 .0001
Vocabulary (A-H) Grade K 743 3.8 5.2 1.46 2.01 .0001 Grade 1 183 4.9 5.9 1.02 1.70 .0001 Grade 2 115 6.4 7.5 1.11 1.28 .0001 Grade 3 135 6.2 6.9 0.66 1.36 .0001
Writing (A-G) Grade K 770 3.9 5.1 1.21 1.22 .0001 Grade 1 193 5.3 6.0 0.69 0.95 .0001 Grade 2 115 6.0 6.6 0.53 1.02 .0001 Grade 3 134 6.5 6.9 0.37 0.97 .0001
EAG K-3 Formative Assessment: Final Report September 2017 D-78
Exhibit D.2.9. Mean progression placements for children assessed in fall by grade level
Constructs Children
Assessed Numeric Average Difference Probability
Crossing Midline (A-C)
Grade K 829 3.7 Grade 1 184 3.7 +0.03 NS
Grip & Manipulation (A-D) Grade K 833 4.1 Grade 1 197 4.5 +0.37 .0001
Letter Naming (A-G) Grade K 835 6.7 Grade 1 196 7.6 +0.86 .0001
Print Awareness (A-G) Grade K 843 6.3 Grade 1 185 7.9 +1.65 .0001
Object Counting (A-J) Grade K 826 7.4 Grade 1 185 9.7 +2.28 .0001 Grade 2 126 10.6 +0.90 .0001
Emotion Expression (A-G) Grade K 805 4.7 Grade 1 195 5.4 +0.70 .0001 Grade 2 126 6.4 +0.99 .0001 Grade 3 138 6.6 +0.22 NS
Emotion Regulation Strategies (A-E)
Grade K 787 3.9 Grade 1 197 4.5 +0.55 .0001 Grade 2 126 4.7 +0.24 NS Grade 3 127 5.2 +0.49 .005
Emotional Literacy (A-I)
Grade K 803 5.2
Grade 1 164 6.2 +1.06 .0001 Grade 2 126 7.8 +1.60 .0001 Grade 3 126 8.1 +0.24 NS
Following Directions (A-F) Grade K 827 5.7 Grade 1 194 6 +0.29 .05 Grade 2 126 6.3 +0.28 NS Grade 3 138 5.8 -0.45 .01
EAG K-3 Formative Assessment: Final Report September 2017 D-79
Constructs Children
Assessed Numeric Average Difference Probability
Gross Motor (A-I)
Grade K 832 8.6 Grade 1 185 8.8 +0.20 NS Grade 2 126 9.7 +0.89 .0001 Grade 3 138 9.7 +0.03 NS
Mathematical Patterns (A-M) Grade K 797 5.0 Grade 1 184 7.3 +2.33 .0001 Grade 2 126 9.0 +1.61 .0001 Grade 3 127 9.0 +0.06 NS
Perseverance (A-H) Grade K 795 5.5 Grade 1 194 5.9 +0.41 .05 Grade 2 126 7.4 +1.54 .0001 Grade 3 138 6.8 -0.61 .01
Problem Solving (A-J) Grade K 797 5 Grade 1 181 6.1 +1.10 .0001 Grade 2 126 7.9 +1.75 .0001 Grade 3 138 7.4 -0.50 .05
Reading Comprehension (A-H) Grade K 784 5.0 Grade 1 196 7.3 +2.25 .0001 Grade 2 126 7.4 +0.13 NS Grade 3 138 7.6 +0.23 NS
Vocabulary (A-H) Grade K 796 3.8 Grade 1 184 4.9 +1.07 .0001 Grade 2 126 6.4 +1.54 .0001 Grade 3 137 6.2 -0.19 NS
Writing (A-G) Grade K 826 3.9 Grade 1 195 5.3 +1.48 .0001 Grade 2 126 6.0 +0.63 .0001 Grade 3 138 6.5 +0.49 .001
EAG K-3 Formative Assessment: Final Report September 2017 D-80
Exhibit D.2.10. Mean progression placements for children assessed in winter by grade level
Constructs Children
Assessed Numeric Average Difference Probability
Crossing Midline (A-C)
Grade K 782 3.8 Grade 1 193 3.8 0.00 NS
Grip & Manipulation (A-D) Grade K 784 4.8 Grade 1 193 4.7 -0.07 NS
Letter Naming (A-G) Grade K 783 7.6 Grade 1 192 7.8 +0.21 .001
Print Awareness (A-G) Grade K 782 7.7 Grade 1 181 8.0 +0.30 .0001
Object Counting (A-J) Grade K 782 9.4 Grade 1 193 10.3 +0.90 .0001 Grade 2 115 10.9 +0.54 .0001
Emotion Expression (A-G) Grade K 763 5.8 Grade 1 193 6.5 +0.69 .0001 Grade 2 115 6.7 +0.21 NS Grade 3 135 7.3 +0.53 .005
Emotion Regulation Strategies (A-E)
Grade K 773 4.5 Grade 1 194 5.1 +0.56 .0001 Grade 2 115 5.0 -0.09 NS Grade 3 135 5.3 +0.30 .05
Emotional Literacy (A-I) Grade K 768 6.5 Grade 1 162 7.6 +1.13 .0001 Grade 2 115 8.8 +1.18 .0001 Grade 3 135 9.2 +0.39 .05
Following Directions (A-F) Grade K 782 6.3 Grade 1 193 6.2 -0.16 NS Grade 2 115 6.5 +0.33 .05 Grade 3 135 6.6 +0.07 NS
EAG K-3 Formative Assessment: Final Report September 2017 D-81
Constructs Children
Assessed Numeric Average Difference Probability
Gross Motor (A-I)
Grade K 782 9.3 Grade 1 186 9.5 +0.21 .05 Grade 2 115 9.9 +0.41 .0001 Grade 3 123 9.9 +0.03 NS
Mathematical Patterns (A-M) Grade K 770 6.9 Grade 1 193 8.7 +1.78 .0001 Grade 2 115 9.6 +0.83 .005 Grade 3 124 10.4 +0.85 .05
Perseverance (A-H) Grade K 770 6.5 Grade 1 193 7.2 +0.71 .0001 Grade 2 115 7.7 +0.48 .001 Grade 3 135 7.7 +0.06 NS
Problem Solving (A-J) Grade K 766 6.4 Grade 1 193 7.7 +1.30 .0001 Grade 2 115 8.9 +1.20 .0001 Grade 3 127 8.9 -0.04 NS
Reading Comprehension (A-H) Grade K 781 6.7 Grade 1 193 7.6 +0.89 .0001 Grade 2 115 8.1 +0.49 .005 Grade 3 134 8.3 +0.18 NS
Vocabulary (A-H) Grade K 772 5.2 Grade 1 184 5.9 +0.69 .0001 Grade 2 115 7.5 +1.59 .0001 Grade 3 135 6.9 -0.62 .001
Writing (A-G) Grade K 780 5.1 Grade 1 193 6.0 +0.96 .0001 Grade 2 115 6.6 +0.61 .0001 Grade 3 134 6.9 +0.22 NS
EAG K-3 Formative Assessment: Final Report September 2017 D-82
Exhibit D.2.11. Percent of kindergarten children at higher, same, or lower levels on progression from fall to winter administrations
We looked at those children who were assessed on each construct in the fall and again in the winter. Ns ranged from 729 to 782 depending on the construct. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green) and the children who were placed lower on the progression in the winter (teal).
70
68
73
65
55
56
73
76
57
19
40
49
38
56
64
62
19
13
5
41
7
11
31
74
42
42
42
6
6
71
86
26
23
12
21
2
23
12
19
10
4
12
6
11
22
23
23
6
4
9
11
14
2
14
4
5
2
2
6
3
9
16
12
9
4
0 10 20 30 40 50 60 70 80 90 100
Writing (N=770)
Vocabulary (N=743)
Reading Comprehension (N=731)
Problem Solving (N=729)
Print Awareness (N=780)
Perseverance (N=742)
Object Counting (N=773)
Math Patterns (N=733)
Letter Naming (N=782)
Hand Dominance N=777)
Gross Motor (N=779)
Grip & Manipulation (N=776)
Following Directions (N=773)
Emotion Regulation (N=735)
Emotional Literacy (N=747)
Emotion Expression (N=744)
Crossing Midline (N=773)
Book Orientation (N=763)
Fall to Winter Progression Placements: Kindergarten
Higher Same, at the Max Same Lower
EAG K-3 Formative Assessment: Final Report September 2017 D-83
Exhibit D.2.12. Percent of grade 1 children at higher, same, or lower levels on progression from fall to winter administrations
We looked at those children who were assessed on each construct in the fall and again in the winter. Ns ranged from 162 to 193 depending on the construct. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green) and the children who were placed lower on the progression in the winter (teal). Note that Book Orientation and Hand Dominance were not assessed in Grade 1.
60
55
37
76
5
65
43
74
24
44
24
25
52
69
59
16
4
21
94
12
43
65
43
59
45
19
2
16
71
30
25
24
11
15
8
17
8
3
8
10
15
20
17
7
10
16
17
13
9
7
9
3
10
9
19
14
9
9
5
0 10 20 30 40 50 60 70 80 90 100
Writing (N=193)
Vocabulary (N=183)
Reading Comprehension (N=193)
Problem Solving (N=180)
Print Awareness (N=181)
Perseverance (N=192)
Object Counting (N=182)
Math Patterns (N=182)
Letter Naming (N=192)
Gross Motor (N=175)
Grip & Manipulation (N=193)
Following Directions (N=192)
Emotion Regulation (N=194)
Emotional Literacy (N=162)
Emotion Expression (N=193)
Crossing Midline (N=182)
Fall to Winter Progression Placements: Grade 1
Higher Same, at the Max Same Lower
EAG K-3 Formative Assessment: Final Report September 2017 D-84
Exhibit D.2.13. Percent of grade 2 children at higher, same, or lower levels on progression from fall to winter administrations
We looked at those children who were assessed on each construct in the fall and again in the winter. 115 2nd graders were assessed on the 12 constructs listed. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green), and the children who were placed lower on the progression in the winter (teal).
42
60
52
59
46
17
43
12
20
32
49
33
2
10
30
10
7
77
83
61
33
23
26
54
26
10
21
25
3
47
4
11
20
17
23
3
3
9
10
22
2
10
8
15
11
18
0 10 20 30 40 50 60 70 80 90 100
Writing (N=115)
Vocabulary (N=115)
Reading Comprehension (N=115)
Problem Solving (N=115)
Perseverance (N=115)
Object Counting (N=115)
Math Patterns (N=115)
Gross Motor (N=115)
Following Directions (N=115)
Emotion Regulation (N=115)
Emotional Literacy (N=115)
Emotion Expression (N=115)
Fall to Winter Progression Placements: Grade 2
Higher Same, at the Max Same Lower
EAG K-3 Formative Assessment: Final Report September 2017 D-85
Exhibit D.2.14. Percent of grade 3 children at higher, same, or lower levels on progression from fall to winter administrations
We looked at those children who were assessed on each construct in the fall and again in the winter. Third graders were assessed on the 11 constructs listed; Ns ranged from 123 to 135.This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green), and the children who were placed lower on the progression in the winter (teal).
43
50
41
81
63
65
12
43
27
61
52
8
33
13
8
84
44
42
21
25
34
40
19
10
9
23
3
8
15
14
15
15
10
7
8
15
3
4
16
4
7
0 10 20 30 40 50 60 70 80 90 100
Writing (N=134)
Vocabulary (N=134)
Reading Comprehension (N=134)
Problem Solving (N=127)
Perseverance (N=135)
Math Patterns (N=124)
Gross Motor (N=123)
Following Directions (N=135)
Emotion Regulation (N=124)
Emotional Literacy (N=124)
Emotion Expression (N=134)
Fall to Winter Progression Placements: Grade 3
Higher Same, at the Max Same Lower
EAG K-3 Formative Assessment: Final Report September 2017 D-86
Exhibit D.2.15 Progression placements for kindergarten children with and without IEPs
Construct and IEP Status Fall Winter N Mean SD p < N Mean SD p < Book Orientation (A-D)
IEP 67 4.6 0.82 .05
65 5.0 0.17 NS
No IEP 770 4.8 0.51 708 5.0 0.19
Crossing Midline (A-C)
IEP 68 3.5 0.68 .05
65 3.7 0.55 .05
No IEP 761 3.7 0.58 717 3.9 0.42
Emotion Expression (A-G)
IEP 65 3.9 1.62 .0001
63 4.7 1.92 .0001
No IEP 740 4.8 1.71 700 5.9 1.74 Emotion Regulation Strategies (A-E)
IEP 66 3.2 1.33 .0001
64 3.9 1.18 .0001
No IEP 721 4.0 1.26 709 4.6 1.27
Emotional Literacy (A-I)
IEP 65 4.5 1.61 .01
64 5.2 1.80 .0001
No IEP 738 5.2 1.76 704 6.6 2.10
Following Directions (A-F)
IEP 67 4.8 1.69 .0001
65 5.4 1.50 .0001
No IEP 760 5.8 1.45 717 6.4 1.06
Grip & Manipulation (A-D)
IEP 67 3.6 0.95 .0001
65 4.3 0.85 .0001
No IEP 766 4.1 0.91 719 4.8 0.53
Gross Motor (A-I)
IEP 67 7.8 1.79 .001
65 8.4 1.63 .0001
No IEP 765 8.6 1.71 717 9.3 1.28
Hand Dominance (A-C)
IEP 68 3.7 0.50 NS
65 3.8 0.44 NS
No IEP 767 3.7 0.47 718 3.9 0.26
Letter Naming (A-G)
IEP 67 6.0 1.63 .0001
66 7.0 1.05 .0001
No IEP 768 6.8 1.17 717 7.7 0.69 Mathematical Patterns (A-M)
IEP 65 4.6 1.95 NS
65 6.1 1.94 .0001
No IEP 732 5.1 1.83 705 7.0 1.83
Object Counting (A-J)
IEP 67 6.4 2.14 .0001
66 8.1 2.40 .0001
No IEP 759 7.5 2.23 716 9.5 1.75
Perseverance (A-H)
IEP 65 4.3 2.29 .0001 64 4.8 2.12 .0001
EAG K-3 Formative Assessment: Final Report September 2017 D-87
Construct and IEP Status Fall Winter N Mean SD p < N Mean SD p <
No IEP 730 5.6 2.04 707 6.6 2.02
Print Awareness (A-G) IEP 67 5.2 2.01
.0001 66 7.0 1.48
.001 No IEP 776 6.4 1.83 716 7.7 0.79
Problem Solving (A-J) IEP 65 4.0 1.67
.0001 65 4.9 2.33
.0001 No IEP 732 5.1 1.98 701 6.6 2.39
Reading Comprehension (A-H)
IEP 66 4.3 2.08 .01
65 5.6 2.21 .0001
No IEP 718 5.1 2.17 716 6.8 1.89
Vocabulary (A-H) IEP 65 3.4 1.40
.05 64 4.2 1.95
.0001 No IEP 731 3.8 1.36 708 5.3 1.96
Writing (A-G) IEP 67 3.3 0.90
.0001 65 4.1 1.25
.0001 No IEP 759 3.9 0.85 715 5.1 1.24
EAG K-3 Formative Assessment: Final Report September 2017 D-88
Exhibit D.2.16. Progression placements for kindergarten ELLs and non-ELLs
Construct and ELL Status Fall Winter N Mean SD p < N Mean SD p <
Book Orientation (A-D) ELL 72 4.9 0.26
.05 67 5.0 0.12
NS Non-ELL 765 4.8 0.56 706 5.0 0.19
Crossing Midline (A-C) ELL 72 3.8 0.45
NS 66 4.0 0.21
.05 Non-ELL 757 3.7 0.60 716 3.8 0.02
Emotion Expression (A-G) ELL 72 4.9 1.68
NS 67 6.2 1.25
NS Non-ELL 733 4.7 1.72 696 5.8 1.82
Emotion Regulation (A-E) ELL 72 4.3 1.09
.01 67 5.1 0.88
.001 Non-ELL 715 3.9 1.29 706 4.5 1.30
Emotional Literacy (A-I) ELL 72 5.7 1.88
.05 66 6.9 1.98
NS Non-ELL 731 5.1 1.74 702 6.4 2.12
Following Directions (A-F) ELL 72 5.9 1.35
NS 67 6.4 1.00
NS Non-ELL 755 5.7 1.51 715 6.3 1.15
Grip & Manipulation (A-D) ELL 72 3.6 0.95
NS 67 4.8 0.45
NS Non-ELL 761 4.3 0.87 717 4.8 0.58
Gross Motor (A-I) ELL 73 8.8 1.30
NS 66 9.4 1.05
NS Non-ELL 759 8.5 1.77 716 9.2 1.36
Hand Dominance (A-C) ELL 73 3.7 0.50
NS 67 4.0 0
.05 Non-ELL 762 3.7 0.47 716 3.9 0.29
Letter Naming (A-G) ELL 72 6.5 1.30
NS 67 7.7 0.67
NS Non-ELL 763 6.7 1.23 716 7.6 0.76
Math Patterns (A-M) ELL 71 4.9 1.93
NS 66 7.0 1.87
NS Non-ELL 726 5.0 1.83 704 6.9 1.66
Object Counting (A-J) ELL 72 7.0 2.26
NS 66 9.5 1.73
NS Non-ELL 754 7.5 1.98 716 9.4 1.86
Perseverance (A-H) ELL 72 5.0 1.97
NS 66 6.5 1.50
NS Non-ELL 723 5.5 2.10 705 6.5 2.14
EAG K-3 Formative Assessment: Final Report September 2017 D-89
Construct and ELL Status Fall Winter N Mean SD p < N Mean SD p <
Print Awareness (A-G) ELL 72 5.5 2.15
.01 66 7.7 0.70
NS Non-ELL 771 6.4 1.83 716 7.7 0.90
Problem Solving (A-J) ELL 72 4.9 2.43
NS 66 6.2 1.94
NS Non-ELL 725 5.0 1.93 700 6.4 2.47
Reading Comp (A-H) ELL 71 4.9 2.00
NS 66 6.6 1.79
NS Non-ELL 713 5.0 2.19 715 6.7 1.97
Vocabulary (A-H) ELL 72 3.7 1.09
NS 66 5.2 1.46
NS Non-ELL 724 3.8 1.40 706 5.2 2.02
Writing (A-G) ELL 71 4.0 1.06
NS 67 5.3 1.07
NS Non-ELL 755 3.8 0.85 713 5.0 1.29
EAG K-3 Formative Assessment: Final Report September 2017 D-90
Exhibit D.2.17. Gender differences* in construct performance by grade and test administration
Construct Kindergarten Grade 1 Grade 2 Grade 3 Fall Winter Fall Winter Fall Winter Fall Winter
Ns (850) (785) (198) (194) (126) (115) (138) (135) Book Orientation Girls NS na na na na na na
Crossing Midline NS NS NS NS na na na na
Emotion Expression Girls Girls NS NS NS NS Girls NS
Emotion Regulation Strategies Girls NS NS NS Girls Girls Girls Girls
Emotional Literacy Girls NS NS NS NS NS NS NS
Following Directions Girls NS NS NS NS NS Girls Girls
Grip & Manipulation Girls Girls NS NS na na na na
Gross Motor Girls Girls NS NS NS NS NS NS
Hand Dominance NS NS na na na na na na
Letter Naming Girls Girls NS NS na na na na
Mathematical Patterns NS NS NS NS NS NS NS NS
Object Counting NS NS Boys NS NS NS na na
Perseverance Girls Girls NS NS NS NS Girls NS
Print Awareness Girls NS NS NS na na na na
Problem Solving NS NS NS NS NS NS NS NS
Reading Comprehension NS Girls NS NS NS NS NS NS
Vocabulary Girls NS NS NS NS NS Girls NS
Writing Girls Girls NS NS NS NS Girls Girls
*Gender listed was placed higher on the progression. Mean differences were statistically significant atthe p < .05 level or lower. NS = non-significant or no differences. na = construct not assessed.
EAG K-3 Formative Assessment: Final Report September 2017 D-91
Exhibit D.2.18. Racial/ethnic differences* in construct performance by grade and test administration
Construct Kindergarten Grade 1 Grade 2 Grade 3 Fall Winter Fall Winter Fall Winter Fall Winter
Ns (739) (680) (164) (160) (125) (114) (105) (105)
Book Orientation NS NS na na na na na na
Crossing Midline NS NS Non-White NS na na na na
Emotion Expression NS NS NS NS NS Non-White NS NS
Emotion Regulation NS NS NS NS NS Non-White NS Non-
White
Emotional Literacy NS NS NS NS NS Non-White NS NS
Following Directions White NS NS NS Non-White
Non-White NS NS
Grip & Manipulation NS NS NS Non-White na na na n
Gross Motor NS NS Non-White NS NS NS NS NS
Hand Dominance White NS na na na na na na
Letter Naming NS NS White White na na na na
Math Patterns NS NS NS NS NS NS NS NS
Object Counting White NS NS NS White NS na na
Perseverance White NS NS NS NS NS NS NS
Print Awareness White NS NS NS na na na na
Problem Solving White White White NS NS NS NS Non-White
Reading Comp White NS White NS White NS NS NS
Vocabulary White NS NS NS NS NS NS NS
Writing White NS NS Non-white NS NS NS NS
*Race/ethnicity group listed (White or Non-white) was placed significantly higher on the progression.Mean differences were statistically significant at the p < .05 level or lower. NS = non-significant or no differences. na = construct not assessed
EAG K-3 Formative Assessment: Final Report September 2017 D-92
Exhibit D.3.1. Coding protocol for evidence analysis
Evidence Analysis Protocol 1. This protocol should be completed once agreement has been reached between the primary
and secondary coder for each construct.
2. Enter the construct abbreviation for the constructs to be coded in the designated coding
worksheet.
3. Copy and paste the child ID and state
from the sample spreadsheet provided
into the appropriate columns.
4. Locate all available evidence for the
assigned construct and child in the file
exported from the technology platform.
5. Code the data based on the rules listed
below.
6. For assigned students, record if any of
the pieces of evidence for a particular child and construct are a video or photo.
Note. If the piece of evidence is a photo or a video, go to the field test data on the K-3
technology platform to view that particular record.
7. Determine whether the Evidence for each student’s Status Summary is specific enough to
endorse a Status Summary (can where the child is on the progression be determined based
on just this piece of evidence?). This coding is conducted at three levels: (a) construct
relevance, (b) skill specificity, and (c) child specificity.
8. Coding Rules
When examining the evidence, the following rules apply to ensure consistency.
a. Each piece of evidence should be analyzed for the assigned construct only, regardless
of its value for use in determining Status Summary for other constructs.
b. Each piece of evidence should be taken at face value. Do not infer any conditions to the
teacher comments. This includes any inferences about the activity or what may have
occurred prior to or after the activity listed as evidence.
Constructs Language/
Communication Approaches to Learning BO Pe PA Motor LN HD RC CM Wr G&M Vo GM FD Cognitive
Social-Emotional OC EL PS EE MP
ERS
EAG K-3 Formative Assessment: Final Report September 2017 D-93
Coding descriptions and guiding questions for coding each piece of evidence
Levels Description Code Construct Relevant
When reviewing evidence for a child, are there one or more pieces of evidence that are relevant to the construct?
The evidence may not be specific to a child or skill, but may provide a description of activities or group skills the teacher observed that are related to the construct.
For example, “Asked class to walk over to cubbies, pick up crayons and notebooks and bring them back to their assigned seats” (Following Directions) or “children wrote stories about their vacation” (Writing). The evidence does not need to include child information to be construct relevant.
Number of pieces of evidence that are relevant to the construct
Skill-Specific When looking at all evidence for an assigned child, does the evidence describe key skills or behaviors found in performance descriptors from the construct progression?
The evidence may not be specific to one child, but may describe skills for a group. These skills are aligned to key words in the progression or other construct materials OR are examples of skills highlighted in the progression.
For example, “While solving addition problems after being taught a strategy to do so, students used ten frame and counters to solve. When faced with something difficult in the activity they looked at a sample problem.” (Problem Solving). Or “during read aloud, two children recognized their reading did not make sense” (Reading Comprehension).
Number of pieces of evidence that are skill-specific
Child-Specific When looking at all evidence for an assigned child, does the evidence describe any key skills or performance descriptors that can be attributed to that child? The teacher may use the child’s name, a pronoun or singular verbs in her/his evidence.
The evidence may not be specific to a skill on the progression, but clearly identifies that the teacher is observing a single child.
For example, “child opened the front cover” or “Tina reflected on her choice to take the blocks from Gina…” or “Malik offered another solution to keep the rice from spilling over”
If a teacher has copied the same evidence for every child in the class and it is not clear that the skill was related to the child being coded, mark the evidence is not child specific. For example, “Today I placed the students in a situation with their group to see how they would react to the problem” does not provide enough detail to be considered child-specific.
If a teacher has copied the same evidence for every child in the class and it is clear that she intended that all children met the skill level, mark the evidence as child-specific. For example, “The class was to draw a picture following my three- step instructions. All children in the class were able to follow the directions without additional support.”
Number of pieces of evidence that are child specific
EAG K-3 Formative Assessment: Final Report September 2017 D-94
Levels Description Code Master Coder Status Summary
If the Evidence, when taken together, is specific enough to assign a Status Summary, then use the Construct Progression to endorse a Status Summary rating. If the evidence, taken together, is not specific enough to draw a conclusion about the Status Summary (e.g., there are not any pieces of evidence that are child-specific or skill-specific), code as 99. If unsure of the code to be given, though the evidence appears to have met some of the criteria above, code as 0.
Coding tips:
• Remember to code each child’s Status Summary based on all ofthe available evidence for that child.
• When looking at multiple pieces of information, it may be useful tolook at the evidence chronologically. This may give a morecomplete picture of how the teacher is seeing the child’s behaviorand skills.
• If a teacher has recorded classroom-wide activities or a summaryof how she/he observed for all her evidence, code this as 99. Theinformation is not specific enough about the child to determine thechild’s placement on the progression. Do not attempt to extrapolateinformation from the preliminary learning status for a child or groupof children.
Status Summary
99 = not specific enough
0 = unsure
9. If a Master Coder Status Summary was assigned, check what Status Summary the teacher
originally endorsed and record it in the coding spreadsheet.
10. Given the teacher evidence in total, does it seem reasonable that he/she may have
assigned this Status Summary?
Code responses:
1 – Yes, believe the teacher was reasonable in his/her assignment of a Status Summary made based on the evidence entered. This could be the case even when the reviewer’s Status Summary is not an exact match to the teacher’s Status Summary.
0 – No, believe the teacher assigned a Status Summary inconsistent with the evidence presented.
EAG K-3 Formative Assessment: Final Report September 2017 D-95
Exhibit D.3.2. Internal reliability coefficients for evidence analysis
Evidence analysis reliability coefficients As part of the evidence analysis and to ensure consistency among reviewers, a 10 percent
sample for each construct was coded by two reviewers prior to coding the full sample. Cohen
kappa estimates, shown below, ranged from .85 to 1.00, indicating almost perfect to perfect
agreement among reviewers. Reviewers discussed any disagreements to reach consensus and
clarify coding rules to be used with the remaining 90 percent of the sample.
Internal Reviewer Agreement Coefficients (Cohen κ)
Construct Cohen Kappa (Cohen κ) Book Orientation 1.00
Crossing Midline 0.94
Emotion Expression 1.00
Emotional Literacy 0.96
Emotional Regulation Strategies 0.96
Following Directions 1.00
Grip & Manipulation 0.89
Gross Motor 1.00
Hand Dominance 0.89
Letter Naming 1.00
Mathematical Patterns 0.89
Object Counting 0.94
Print Awareness 0.93
Perseverance 1.00
Problem Solving 0.85
Reading Comprehension 1.00
Vocabulary 1.00
Writing 1.00
EAG K-3 Formative Assessment: Final Report September 2017 D-96
Exhibit D.4.1. A profile of teachers participating in the field test (n = 145)
Characteristic N % Mean (SD)
Kindergarten 85 59%
1st grade 22 15%
2nd grade 16 11%
3rd grade 12 8%
Mixed or Multi-grade 9 6%
Other 1 1%
Has paid paraprofessionals or volunteers in the classroom 78 54%
Has computer in the classroom 137 95%
Has access to mobile device or tablet at school 126 87%
Has good internet access in classroom 111 77%
Has spotty internet access in classroom 34 24%
Received PD on formative assessment 73 50%
Already using formative assessment before this project 110 76%
Participated in K-3 Formative Assessment pilot 38 26%
Gender (Female) 142 98%
Education Level: Master’s degree or higher 60 41%
Years of teaching experience 14.5 (10.3)
American Indian/Alaskan native 0 0%
Asian 5 3.4%
Black/African American 0 0%
Native Hawaiian/Other Pacific Islander 1 0.7%
White 138 95.2%
Unknown/Missing 1 0.7%
Hispanic/Latino Origin 7 4.8%
EAG K-3 Formative Assessment: Final Report September 2017 D-97
Revised August 11, 2016
Exhibit D.5.1. Training guidance for administrators
Recommendations for Administrators Supporting Teachers
Help teachers create a positive classroom cultureo Promote a culture that encourages conversation, dialogue and listening.o Investigate “quiet” classrooms. Who is doing the talking/listening?o Model good questioning, probing and listening.
Support teachers’ efforts for using developmentally appropriate practiceso Ask teachers to think about the types of learning formats they use (e.g., whole group, small group,
centers/stations). Is there a balance? Do the learning formats they use provide opportunities forstudents to interact with each other and the materials in the classroom?
o Offer an opportunity for teachers to reflect on their current instructional practices and consider how theymight enhance what they’re doing to increase active participation in engaging learning experiences.
o Allow opportunities for teachers to try new practices.o Arrange for teachers to visit colleagues who successfully use various learning formats and instructional
strategies.
Support intentional planning and decision making by providing opportunities for teachers to work andlearn together
o Put it on the agenda (PLC Meetings, Staff Meetings) to talk through the evidences and the process.o Participate in on-site or online Professional Learning Communities (PLCs) that discuss observation-based
data collection and data trends.o Provide planning time for pilot teachers to meet with others (e.g., grade specific, cross-grade, support
staff).o Establish protocols for collaboration and identify expected outcomes.
Check-in with teacherso Establish a communication protocol - who, how, when.o Determine a focus and make time to visit classrooms, talk with teachers, and ask questions.o Use your own observation data and faculty input to determine staff development opportunities.o Capture positive happenings and celebrate the learning experiences.
Support technology effortso Provide strong wireless connection and access to a computer in the classroom.o Arrange access to a tablet/smartphone for collecting evidence.o Encourage teachers to use the App on their phone/tablet to collect evidences.o Supply contact information for tech support (local and Teaching Strategies).
Consider adjusting time, policies and/or personnel to provide additional supporto Consider schedule adjustments.o Reallocate time on agendas for current meetings to allow specific time for work and collaboration.o Consider removing other expectations.o Offer additional support in the classroom.o Think about how other staff could support this process (e.g., instructional coaches, ELL, EC).
EAG K-3 Formative Assessment: Final Report September 2017 D-98
Exhibit D.7.1. Construct performance for each validity claim
AtL
Validity Claims
Boo
k O
rient
atio
n
Prin
t Aw
aren
ess
Lette
r Nam
ing
Follo
win
g D
irect
ions
Writ
ing
Rea
ding
C
ompr
ehen
sion
Voc
abul
ary
Obj
ect C
ount
ing
Mat
h P
atte
rns
Pro
blem
Sol
ving
Han
d D
omin
ance
Grip
& M
anip
ulat
ion
Cro
ssin
g M
idlin
e
Gro
ss M
otor
Em
otio
n E
xpre
ssio
n
Em
otio
nal L
itera
cy
Em
otio
n R
egul
atio
n S
trate
gies
Per
seve
ranc
e
Claim 1.1 Proper sequence
% teachers reporting skills in right order 99 99 99 98 97 99 95 95 95 95 99 99 99 92 94 93 94 99
Frequency distribution (K only) - Smooth or Uneven S U S S S U S U U U S S S U U S S U
Skill IRT factor score means in order (K-3) - Yes or Possible problem (small Ns/skills too close) Y Y Y Y Y Y Y P P Y Y Y Y P Y Y Y Y
Claim 1.3 K entry performance levels
% teachers agreeing with expert on demonstrating level (at cutpoint or higher) 76 76 70 1 85 76 98 17 10 6 90 72 87 56 60 74 100 98
% children placed at or above demonstrating level after K entry (expected pattern) 97 96 96 45 96 90 98 83 66 58 98 95 94 92 88 94 98 99
Claim 1.2 Sensitive to growth
% teachers reporting construct captured growth well 98 96 93 96 92 90 84 92 86 88 99 97 96 91 84 85 84 91
Higher mean placements in winter than fall (Consistent, Inconsistent pattern) C C C C C C C C C C C C C C C C I I
Higher mean placements at higher grade level (K-1, 1-2, 2-3) (Consistent, Inconsistent pattern) na C C I C C I C C I na I I C C C I I
Claim 1.7 Useful for K-3 Instruction
% teachers reporting construct helpful to instruction (very, somewhat) 66 75 73 75 81 76 70 77 68 79 60 68 63 51 64 64 65 73
Claim 1.5 Teacher confidence
% teachers confident in accurately placing child on progression 99 97 95 98 95 93 83 96 88 89 97 92 93 94 74 75 71 89
Claim 1.4 Fidelity of implementation: Reliability of status summary
% children with evidence specific enough to code 78 74 78 80 84 56 69 78 79 55 84 84 74 66 66 45 40 33
% children status summary consistent with teacher evidence 82 89 88 86 95 87 71 66 75 79 90 90 100 96 81 64 86 85
% children teacher's status summary matches master coding 79 85 88 79 86 78 67 59 66 72 84 74 96 83 76 40 55 60
Claim 1.6 Incorporate assessment into classroom routines
% teachers reporting they could assess in daily classroom actvities 92 90 81 99 96 93 78 81 54 83 99 98 84 46 78 77 76 97
% teachers reporting construct easy or very easy to assess 98 97 99 86 93 84 66 95 82 60 98 94 89 95 24 31 22 54
ASSE
SSM
ENT
CONT
ENT
ASSE
SSM
ENT
PRO
CESS
Domains Language & Communication Cognitive Physical/Motor Social-Emotional
85-100
70-84 Good
1-69 Poor/Unsure
Very Good
Construct Performance
EAG K-3 Formative Assessment: Final Report September 2017 D-99
Appendix E National Experts Supporting Project Activities Exhibit E.1. National domain experts who supported different project activities
EAG K-3 Formative Assessment: Final Report September 2017 E-1
Exhibit E.1. National domain experts who supported different project activities
Expert Title & Affiliation Domain (Construct) Project Activities
Dr. Stephen Sanders Professor, Physical Education University of South Florida
Physical/Motor Development X X
Dr. Jill Sells Developmental Consultant Seattle, WA
Physical/Motor Development X
Dr. Laura J. Colker Child Development Consultant Washington, DC Cognitive Development X
Dr. Darryl Greenfield Professor of Psychology & Pediatrics University of Miami
Cognitive Development (Science) X
Dr. Douglas Clements
Professor of Early Childhood Learning University of Denver
Cognitive Development (Mathematics) X
Dr. David Dickinson Professor, Dept. of Teaching & Learning, Peabody College Vanderbilt University
Language Development & Communication
X
Dr. Susan Neuman Professor of Early Childhood and Literacy Education New York University
Language Development & Communication
X X
Dr. C. Cybele Raver Vice Provost for Research and Faculty Affairs & Early Childhood Researcher New York University
Social-Emotional Development; Approaches to Learning
X X
EAG K-3 Formative Assessment: Final Report September 2017 E-2
Expert Title & Affiliation Domain (Construct) Project Activities
Dr. Clancy Blair Professor of Cognitive Psychology New York University
Approaches to Learning; Social-Emotional Development (Problem Solving, Following Directions)
X X
X
Dr. Marilou Hyson Early Childhood Development & Educational Consultant Stockbridge, MA
Approaches to Learning Social-Emotional Development
X X
Dr. Mary Louise Hemmeter
Professor, Department of Special Education Vanderbilt University
Social-Emotional Development X X
Dr. Arthur Baroody
Professor Emeritus, Department of Curriculum & Instruction University of Illinois at Urbana-Champaign
Cognitive Development (Object Counting) X X
Dr. Howard Goldstein
Professor, Communication Sciences & Disorders University of South Florida
Language Development & Communication (Vocabulary)
X
Dr. Dorothy Strickland
Professor of Reading Rutgers, The State University of New Jersey
Language Development & Communication (Reading Comprehension, Writing)
X
Dr. Bethany Rittle-Johnson
Associate Professor, Psychology and Human Development Vanderbilt University
Cognitive Development (Math Patterns) X X
Dr. Lauren Little
Assistant Professor The University of Kansas Medical Center Occupational Therapy Education Department
Physical/Motor Development X X
EAG K-3 Formative Assessment: Final Report September 2017 E-3
Expert Title & Affiliation Domain (Construct) Project Activities
Dr. Karen McFadden
Assistant Professor, Dept. of Early Childhood Education Brooklyn College, The City University of New York
Social-Emotional Development (Perseverance)
X X
Dr. Brian Kissel Associate Professor, Reading Strand University of North Carolina Charlotte College of Education
Language Development (Writing, Book Orientation, Print Awareness, Letter Naming)
X X
Dr. Jean Vintinner Clinical Assistant Professor University of North Carolina Charlotte College of Education
Language Development (Reading Comprehension, Vocabulary, Problem Solving)
X X
Dr. Emily Frye
Assistant Professor Dept. of Psychological & Brain Sciences College of Arts & Sciences Indiana University
Cognitive Development (Math Patterns) X
EAG K-3 Formative Assessment: Final Report September 2017 E-4