Running head: IMPLEMENTATION OF SAXON PHONICS
Implementation of Saxon Phonics and Spelling K Evaluation Project Design
Michelle C. Wallace
Virginia Commonwealth University
Author Note
Michelle C. Wallace, Graduate School, School of Education, Virginia Commonwealth
University.
This is an assignment for EDLP 711. The author collaborated with a peer to complete
Field Experience 3: Self-selected Evaluation Project.
IMPLEMENTATION OF SAXON PHONICS 2
Implementation of Saxon Phonics and Spelling K Evaluation Project Design
Elementary schools have the important role of laying the groundwork for creating
successful students. They must teach students many competencies to include social, behavioral,
and instructional skills. One of the most important of these skills is teaching children how to
read. This skill is necessary to promote further success in school as well as in life. Schools are
responsible for teaching students fundamental skills in Pre-Kindergarten and Kindergarten which
will carry them throughout life. One such skill is reading. If students do not learn to read, then
they will never be successful in other subjects which will ultimately impact their success in life.
Saxon Phonics and Spelling – K (2005) has been implemented at ABC Elementary
School since 2009-2010. ABC Elementary School is located in a rural area in Virginia and is
one of three elementary schools within the school division. ABC Elementary School was subject
to a grade reconfiguration for the 2010-2011 school year. ABC Elementary School serves
students in Pre-Kindergarten, Kindergarten, Early Childhood Special Education, and Head Start.
The student population is 49% White, 48% Black, and 3% Hispanic/other, with a Free and
Reduced Lunch rate of 69% (C. Gravely, personal communication, October 5, 2012). An
evaluation of Saxon Phonics and Spelling – K (Saxon Phonics) is needed to evaluate the
implementation of the program at ABC Elementary School. Focus group meetings were held to
gather information regarding the implementation of Saxon Phonics. A responsive evaluation
will be conducted to collect data and prepare a report to the client, ABC Elementary School.
This paper will examine the purpose for the responsive evaluation using a theoretical
lens; reveal the focus for the evaluation questions; and review the evaluation design selection,
evaluation components, and rationale behind the selection. Finally, a logic model will illustrate
the program actions, what the program does, and how the investments will be linked to results.
IMPLEMENTATION OF SAXON PHONICS 3
Purpose
The purpose of the evaluation is to collect information regarding the implementation of
Saxon Phonics at ABC Elementary School. After meeting with staff for focus group meetings at
ABC Elementary School, it became evident that parts of the Saxon Phonics program are being
implemented with fidelity, where other areas still need to be developed. Concerns about the
implementation of the Saxon Phonics program are partly due to emerging technologies, a lag
time in initial training and current needs, and a lack of supplemental fluency materials (Focus
Groups, personal communication, October 5, 2012). Within the values branch, the evaluator’s
role is to be the communicator that promotes dialogue that is meaningful (Mertens & Wilson,
2012). During the evaluation, the evaluators will engage in meaningful conversations with the
stakeholders for effective communication. The evaluators will remain aware of power
differences that exist within the organization and how power may influence the evaluation.
Using leadership roles through the leadership process, the evaluators will bring to light possible
conflicts and provide space for possible discussions about the implementation of Saxon Phonics
(Mertens & Wilson, 2012).
Evaluation Questions
The focus for the evaluation of Saxon Phonics will be the actual implementation versus
the perceived implementation of the Saxon Phonics program in Kindergarten classes at ABC
Elementary School. The evaluand, Saxon Phonics, is a classroom based program; however,
instructional support personnel provide supplemental instruction to reinforce phonemic
awareness and other aspects of the program. The Saxon Phonics program includes (a) the
controlled vocabulary and practice, (b) the annotated bibliography, (c) coding, (d) spelling, (e)
assessments, (f) pacing, (g) handwriting, and (h) reading fluency (Simmons, 2006). The
IMPLEMENTATION OF SAXON PHONICS 4
principal at ABC Elementary is interested in determining how effective the implementation of
the Saxon Phonics program is based on the fact that some components of the program are not
being used. The following questions will be answered as a result of the responsive evaluation.
1. Does the lack of the classroom implementation of all the components of Saxon
Phonics impact the effectiveness of the program?
2. How has the lack of on-going professional development for the Saxon Phonics
program affected the fidelity of the program?
3. How can specific technology enhance the effectiveness of the Saxon Phonics
program?
Design Selection, Components, and Rationale
Evaluation Design Selection
Dividing the world of evaluation into four separate paradigms is one way of organizing
the major influences that have affected the evolution of this transdiscipline (Mertens & Wilson,
2012, p. 133). Fundamentals of the four paradigms will lead the evaluator to different evaluation
approaches. The four paradigms for evaluations are postpositivist, pragmatic, constructivist, and
transformative. The postpositivist paradigm is based on quantitative data. This paradigm was
not chosen by the evaluator because the quantitative data on the evaluand is limited. Even
though the evaluation will use data resources, the purpose of the evaluation is not based on
statistical outcomes. The pragmatic paradigm is based on data deemed important by
stakeholders. This paradigm was not selected by the evaluators because the data used by the
stakeholders will not affect the outcome of the evaluation. The constructivist paradigm looks at
the values and perspectives through different methods. The evaluator chose the constructivist
paradigm and values branch to evaluate the Saxon Phonics program because the evaluators will
IMPLEMENTATION OF SAXON PHONICS 5
use qualitative data to look at varying perceptions and values of the different stakeholders.
Finally, the transformative paradigm focuses on viewpoints of marginalized groups. This
paradigm will not be used by the evaluator since the evaluation does not have a need to look
specifically at the viewpoints of marginalized groups to evaluate the implementation of the
Saxon Phonics program at ABC Elementary.
The constructivist paradigm promotes the responsive evaluation. A responsive
evaluation includes the values which stakeholders find important to the evaluation. These values
drive the development of investigative questions to be answered by the evaluation. A responsive
evaluation allows the evaluators to modify the evaluation based on the needs of the stakeholders.
This allows the evaluators to be responsive throughout the evaluation. The responsive evaluation
will allow stakeholders to explore their perceptions and beliefs about the implementation of the
Saxon Phonics program as they will be active participants in the evaluation process.
Background information gathered through focus group discussions regarding the
implementation of Saxon Phonics led the evaluators to pose evaluation questions using the
responsive evaluation design. The evaluators will utilize qualitative data from observations,
interviews, and documents. Quantitative data will be limited to student data demonstrating
progress of phonological attainment. The responsive evaluation was selected because it utilizes
observations, narrative data, product displays (e.g. student work), and input provided by
stakeholders (Mertens & Wilson, 2012, p. 251). According to Stake (1991), evaluation questions
should reflect the values of what is considered important based on what has been observed by the
evaluator prior to the evaluation of the program (Mertens & Wilson, 2012, p. 151).
The evaluators of this program are internal evaluators. The evaluators are employed by
ABC Elementary School Division and therefore meet the criteria of an internal evaluator. An
IMPLEMENTATION OF SAXON PHONICS 6
external evaluator is not needed at this point in the evaluation process. This evaluation is
formative in nature because the evaluation is based on the current implementation of the Saxon
Phonics program. It is not a summative evaluation because the evaluation will end prior to the
completion of an entire implementation phase.
Components of the Evaluation
The responsive evaluation will provide primarily qualitative data which will help the
evaluators find value in the Saxon Phonics program implementation. Quantitative data will be
limited to student achievement data from the beginning of the year to mid-year. Qualitative data
analysis helps to identify themes in the data (Mertens & Wilson, 2012). Quantitative data
analysis uses statistics to reduce a large data set into more meaningful terms (Mertens & Wilson,
2012). The proposed evaluation will utilize a limited mixed methods approach incorporating
both quantitative and qualitative data, with more reliance on qualitative data. For the purpose of
this evaluation, a mixed methods approach will allow the evaluators to consider quantitative
results during the program implementation and determine through qualitative measures how
much the Saxon Phonics program impacts student achievement.
The evaluation will employ different methods to collect data including observations,
interviews, documents, and student data. Each of the questions posed for the evaluation will be
answered through the data collection process. Question one refers to the classroom
implementation of the Saxon Phonics program, or the lack thereof. The evaluators will use
observations, interviews, and student data to determine how the varying implementation of the
program between classes impacts the effectiveness of the program, if at all. Question two refers
to the professional development for the Saxon Phonics program. The evaluators will use
documents, observations, and interviews to determine how the lack of professional development
IMPLEMENTATION OF SAXON PHONICS 7
since the beginning of the program implementation has had an effect on the fidelity of the
program. Question three refers to how technology could enhance the Saxon Phonics program.
The evaluators will use interviews, observations, and student data to determine the influence that
technology has on the effectiveness of the program. Student data will be limited to measuring
achievement of phonic and literacy attainment. Once the data has been collected, the evaluators
will triangulate the data to determine the degree of implementation of the Saxon Phonics
program. By crossing the data sources, both qualitative and quantitative, the evaluators will get a
clearer image of the impact of the program being evaluated. This process will validate the data
and make it more meaningful to the investigation. The methods for collecting data through
observations, interviews, documents, and student data will be explained.
Observations.
Observations will be conducted in Kindergarten classrooms, Title I Reading classes, and
English Language Learner pull-out sessions. Additional instructional support personnel will be
included in the observations during sessions where students with disabilities are mainstreamed
into the classroom, as well as during sessions when the Speech/Language Pathologist is
integrating the Sounds in Motion program. Sounds in Motion is a program specifically designed
as a kinesthetic learning model to teach the sounds of the alphabet (M. Wallace, personal
communication, November 6, 2012). It is not a component of the Saxon Phonics program but
may have an impact on its implementation. During the observations, a checklist rubric will be
used that includes a description of each of the components of the Saxon Phonics program. The
rubric will help the evaluators determine the level of implementation of each component.
According to Stake (1991), observations are a key factor in responsive evaluations and can
IMPLEMENTATION OF SAXON PHONICS 8
contribute to the overall understanding of the program and can also assist in resolving any issues
identified (Mertens & Wilson, 2012, p. 151).
Interviews.
Interviews will be conducted with stakeholders. The stakeholder groups will represent
(a) teachers, (b) division instructional staff, (c) the building administration, and (d) instructional
support personnel. Interviews will allow for stakeholders to speak freely in regard to their
implementation of the Saxon Phonics program in their classrooms. Interviews will also provide
an opportunity for division and building administration to express their views of the program, as
well as their expectations for program outcomes. The interviews may also lead to suggestions
for strengthening the program or changing the way it is being implemented in classrooms.
Specific questions will be developed by the evaluators to coincide with the components of the
Saxon Phonics program.
Documents.
Program documents, as well as documents kept by the teachers, will be reviewed. The
evaluators will look at teachers’ lesson plan books for documentation of the components used
from the Saxon Phonics program. Classroom materials and student work samples will be
reviewed to determine the level of implementation of the program. A document checklist will be
developed and utilized to collect data on which program components are being implemented
across settings. Saxon Phonics has some optional components. These optional components will
be included in the document review to determine if any teachers are using those components to
enhance their classroom implementation.
Student Data.
IMPLEMENTATION OF SAXON PHONICS 9
Student data will be collected as a part of the program evaluation and will serve as
quantitative data to track the student progress. Six-week assessments on upper and lower case
alphabet recognition and letter sounds will be reviewed. Student achievement will be compared
using baseline data from the beginning of the school year and on mid-year assessments. The
principal informed the evaluators that the baseline data on student knowledge of the upper and
lower case alphabet knowledge was collected (C. Gravely, personal communication, November
6, 2012). Data from the Phonological Awareness and Literacy Screening (PALS) assessment
will be collected from the beginning of the school year and mid-year for students identified as
risk for learning phonological concepts. All Kindergarten students took the PALS assessment at
the beginning of the year, but only students who were identified as at-risk by the pre-assessment
will be tested using the PALS mid-year assessment (C. Gravely, personal communication,
November 6, 2012). The alphabet knowledge and PALS data will be reviewed by classroom to
determine the impact of the implementation of Saxon Phonics by classroom and to identify
whether or not there is an effect on student achievement. Due to the time constraints of this
evaluation, mid-year data will be used to measure student achievement. If the evaluation could
extend to the end of the school year, then end of year data would also be included.
Rationale
Various other evaluation methods could have been selected to use to determine the
effectiveness of the implementation of Saxon Phonics at ABC Elementary School. Using the
responsive model for the evaluation will fuel conversations about the actual implementation of
the Saxon Phonics program in the Kindergarten classrooms and supplemented by the
instructional support personnel. As issues arise during the evaluation, the evaluators can be
responsive to those issues at that time. “Conducting an evaluation that is responsive to the needs
IMPLEMENTATION OF SAXON PHONICS 10
of the program stakeholders. . .will ensure that what stakeholders have learned through their
active participation is meaningful and is more likely to be used for improving and enhancing
programs” (Mertens & Wilson, 2012, p. 151).
Logic Model
The evaluation of Saxon Phonics will rely on certain inputs, outputs, and resources which
will provide anticipated outcomes based on a regimented timeline for completion of the
evaluation and reporting. The inputs will include proposed data sources from the Saxon
curriculum, observations, interviews, program documents, and student data. For the data
analysis, observations and interviews will provide a wealth of information to help determine the
level of fidelity in the program implementation. Student data will be instrumental in helping the
evaluators determine the effectiveness of the Saxon Phonics program when compared to the
actual implementation of the components. Although it is not the intent of the responsive
evaluation format, data may allow the evaluation to slide into the transformative paradigm.
Depending on the student demographic data provided by the building administrator, the
evaluators may also be able to determine if specific Gap groups as identified by the federal
Annual Measureable Objectives (AMOs) are determined to be a greater need for phonics
interventions. Gap groups are described by three categories: Gap group 1 represents students
with disabilities, students who are English language learners, students who are economically
disadvantaged; Gap group 2 represents African-American students, not from Hispanic origin;
and Gap group 3 represents Hispanic students of one or more races (Virginia Department of
Education, 2012). The activities to conduct the evaluation and collect data will follow a specific
timeline and will require specific resources.
IMPLEMENTATION OF SAXON PHONICS 11
The timeline for the evaluation will fall between the months of December 2012 and
March 2013. During December, the evaluators will present the evaluation proposal to the client.
Data collection tools will be developed during this time and approved by the client. Beginning
in January, evaluators will begin the review of student data (baseline and mid-year). Also in
January, interviews and classroom observations will be conducted of the Kindergarten
classrooms. Evaluators will also interview division and building administration staff. During
February, a mid-term formative report will be provided by the evaluators to the client.
Classroom observations and interviews will be conducted with the instructional support
personnel. In March, the evaluators will present a summative report of findings to the client.
Specific dates will be determined upon the client accepting the evaluation proposal. The client
will need to agree to provide required resources for the evaluation. Required resources include:
a room for evaluators to work and conduct interviews; classroom coverage for teachers to
participate in interviews; access to student data agreed upon by the client and the evaluators; and
time for classroom observations to occur. It is suggested that $250 be budgeted by the client for
incidentals such as snacks for the interviewees and ink for printing purposes. Since the
evaluators are internal evaluators, there will be no cost for their time. The evaluators will
provide tangible resources such as a recorder for interviews, checklists to be used during
observations and document reviews, and other data collection tools.
It is anticipated that the evaluation will provide information based on data triangulation
which will help ABC Elementary School determine short and long term implications. In the
short term, the evaluation will determine the implementation of the Saxon Phonics program, the
level of support offered to teachers, and the effectiveness of the program. Long term outcomes
may implicate future needs for a more successful implementation of the program, determine the
IMPLEMENTATION OF SAXON PHONICS 12
most effective components of the program, and possibly indicate which Gap groups benefit or
suffer the most from the implementation of the program. A logic model is used to illustrate the
sequence of the Saxon Phonics program actions, what the program does, and how the
investments in it will be linked to its results. As demonstrated in Figure 1, the evaluation process
is impacted by certain assumptions and external factors which may influence the data collection
process for the evaluation of the implementation of Saxon Phonics.
Logic Model
Inputs Outputs Outcomes -- ImpactActivities Participation Short Long
-Saxon Curriculum
-Observations
-Interviews
-Program Documents
-Student Data *PALS *6 Week assessments *Pre and post assessments
-Observation of classrooms
-Individual interviews
-Pre- and Post-assessments for students
-Review of program documents
-Presentation of mid-term and final reports
-Students
-Teachers
-Building and district administration
-Two internal evaluators
-Determining the implementation of Saxon Phonics in classrooms
-Determine level of support for Saxon Phonics offered by instructional support personnel
-Determine the effectiveness of the implementation
-Determine needs for future implementation
-Determine which components are most effective for student achievement
-Determining the needs of Gap groups as defined by AMOs
Assumptions-The client will provide time for the teachers to participate in the interviews.-The client will provide access to student data needed for the evaluation.-The responsive evaluation will be sufficient to answer the questions regarding the implementation of the program.
External Factors-Time will be given for the interviews to take place.-Resources will be available for classroom coverage during interviews.-A schedule will allow for classroom observations of all Kindergarten classrooms and interactions with the instructional support personnel.
Figure 1. The logic model for the evaluation of Saxon Phonics demonstrates the input and output components of the evaluation of the program. Also considered are the assumptions and external factors which influence the outcomes of the evaluation.
A meta evaluation will be conducted throughout the entire process of the evaluation to
determine the quality of the evaluation process. Mertens and Wilson (2012) suggest that the
IMPLEMENTATION OF SAXON PHONICS 13
meta evaluation involve the stakeholders to discuss the evaluation design and process. Also,
methodologies should be discussed about the data collection strategies and how information will
be disseminated to the client and other stakeholders (Mertens & Wilson, 2012). During the
evaluation planning process, the evaluators considered program evaluation standards and have
determined that the evaluation meets criteria for utility, feasibility, propriety, and accuracy.
Table 1 depicts the evaluation standards which have been met thus far in the evaluation planning
phase or are planned to be met during the evaluation action phase.
Table 1
Summary of The Program Evaluation Standards Met for the Evaluation Proposal for the Implementation of Saxon Phonics at ABC Elementary School
Standard Descriptor
Utility 1 Evaluator Credibility
Utility 2 Attention to Stakeholders
Utility 3 Negotiated Purposes
Utility 7 Timely and Appropriate Communication and Reporting
Feasibility 1 Practical Procedures
Feasibility 3 Resource Use
Feasibility 4 Project Management
Propriety 1 Responsive and Inclusive Orientation
Propriety 2 Formal Agreements
Propriety 3 Human Rights and Respect
Propriety 4 Clarity and Balance
Accuracy 1 Trustworthy Conclusions and Decisions
Accuracy 2 Valid Information
Accuracy 3 Reliable Information
Accuracy 4 Explicit Evaluand and Context Descriptions
Accuracy 5 Sound Qualitative and Quantitative Methods
Accuracy 8 Valid Communication and Reporting
IMPLEMENTATION OF SAXON PHONICS 14
Resources
Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A
comprehensive guide. New York, NY: The Guilford Press.
Saxon Publishers. (2005). Scientific research base for Saxon phonics and spelling K-3:
Foundational research and program efficacy studies. Retrieved from
http://saxonpublishers.hmhco.com/HA/correlations/pdf/s/saxon_phonics_research.pdf
Simmons, L. (2006). Saxon phonics and spelling K: Instructional overview. Austin: Saxon
Publishers, Inc.
Virginia Department of Education (2012). ESEA flexibility. Retrieved from
http://www.doe.virginia.gov/federal_programs/esea/flexibility/index.shtml