View
214
Download
1
Category
Preview:
Citation preview
Assignment 4.1
Formative Evaluation of an Internet Searching Tutorial
Submitted By:
Sun Hong Hwang
Paul Kelly
Meredith Rogers
Q410 – Formative and Summative EvaluationMarch 15, 2003
Front-End Analysis Problem
The Park Hill School District (Kansas City, Missouri) identified a gap between teacher performance utilizing the Internet (actual) and an optimal expectation that teachers be able to effectively and efficiently find information to support instructional processes.
Proposed Solution Based upon the facts obtained via a front-end analysis (performance analysis,
environmental analysis, learner analysis, and needs assessment), it was determined that K-12 teachers in the Park Hill School District needed training in effectively searching the Internet to close the gap between the “optimals” and “actuals” found in during the front-end analysis.
Formative Evaluation of an Internet Searching Tutorial
Introduction
Instructional Materials/Product Materials
“Take the Garbage out of Internet Searching” was developed specifically as a training tool to address the performance problem outlined on the previous slide.
Learner Objectives Learners will. . .
. . .explore a variety of search engines and online directories and note differences of each.
. . . effectively utilize a search engine and a directory for online searches of information.
. . . improve the reliability of simple searches with advanced operators. . . . explore natural language searches. . . . effectively choose the proper search engine for a given online search
task.
Formative Evaluation of an Internet Searching Tutorial
Introduction
Instructional Materials/Product Status of Materials
“Take the Garbage out of Internet Searching” is currently being developed, awaiting the results of this formative evaluation prior to distribution.
Media “Take the Garbage out of Internet Searching” is currently available for preview
in an online version and a CD-ROM format. The product/materials were developed using Macromedia Authorware 6.0, which can be exported in a web format or as an executable runtime program from CD-ROM.
The CD-ROM version was utilized for the purposes of this report. The URL to the online version is:
http://content.parkhill.k12.mo.us/users/pvkelly/Web/internetsearching-beta2.htm
Formative Evaluation of an Internet Searching Tutorial
Introduction
Goals of Evaluation The goal of this formative evaluation is to improve the effectiveness of
the instructional materials prior to usage by a large population of K-12 teachers. The goals of this formative evaluation process include . . .
1. . . . identifying the deficiencies in learning effectiveness of the instructional materials;
2. . . . locating ease of use problems with the instructional materials; and
3. . . . evaluating the efficiency of the instruction within the materials.
Evaluators An Expert Reviewer Two One-on-One Reviewers
Formative Evaluation of an Internet Searching Tutorial
Formative Evaluation Framework
Reviewer Mr. Robert Borst, Middle School Mathematics Teacher at Gentry Middle
School, Columbia, Missouri Expertise
Teacher of courses for the SISLT at the within the track of technology in schools Web page designer & creator of multi-media presentations for the purpose of
conducting teacher in-services for the Columbia Public School District.
Evaluation Goals Evaluate the strengths and weaknesses of the program’s efficiency of
operation Evaluate the usefulness of the included content.
Evaluation Tool A single three-page questionnaire was to Mr. Borst to complete on his own
as he worked through the various screens of the program. (Appendix 01)
Formative Evaluation of an Internet Searching Tutorial
Expert Review
Recommendations of Expert Review The information is too wordy. There is too much to read through with
some of the topics outlined in the program. He said that the amount of information is far more in-depth that most K-12 teachers will ever need to know or will ever use.
The information is “presented” for the most part rather than given in an interactive mode for the user to demonstrate his or her skills.
The transitions between screens are inconsistent and very aggravating at times. A single type of quick transition needs to be considered,
rather than a different type of screen change with every new topic.
Formative Evaluation of an Internet Searching Tutorial
Expert Review
Reviewer A Amy Carso, Middle School Language Arts Teacher at Gentry
Middle School, Columbia Public School District Expertise
A novice when it comes to understanding how the computer works and all its capabilities.
She describes herself as being “pretty comfortable [with technology now]”
Reviewer B Keri McConnell, 5th Grade Teacher, Park Hill School District,
Kansas City, Missouri Expertise
She serves as a trainer for many technology-based staff development initiatives.
She is well respected by her peers for her use of technology in the classroom, as well as her ability and willingness to show others.
Formative Evaluation of an Internet Searching Tutorial
One-on-One Reviews
Evaluation Goals The reviewers evaluated . . .
. . . the directions and commands of this instruction program for clarity and operability from a learner’s perspective.
. . . the aesthetics design of the program for engaging the learner. . . . the level of interaction for the learner. . . .the content with regards to meeting the needs that K-12 teachers
generally desire in order to improve the efficiency of their Internet searching abilities.
Evaluation Tool A single three-page questionnaire was reviewed each user during their
sessions and given to them to complete on their own as they worked through the various screens of the program (Appendix 02).
Formative Evaluation of an Internet Searching Tutorial
One-on-One Reviews
Recommendations of One-on-One Reviews Font size is too small making the user having to strain to read
some of the material. There needs to be more interaction for the learner as they are
reading through the information. The reviewers suggest the inclusion of sample exercises, pre-test questions or actual connections to the Internet to practice some of their skills.
There was no real challenge to this program for the user. Both reviewers summarized the program, as just basically being a reading exercise, which many educators know will not keep a
learner engaged in any learning process for very long.
Formative Evaluation of an Internet Searching Tutorial
One-on-One Review
Recommendations of Formative Evaluation Based on the data collected by the expert reviewer and the two one-on-one
reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently.
Font Needs to be larger or more bold; change style to a more squared shape.
Directionality Either enlarge the arrows on the quiz screens or remove them and just
make the words the link; be consistent with what are links within the program, add a skip; create ways to restart the program more easily; add a skip button on opening screens to bypass; fix or remove the backwards button on the first two screens.
Aesthetics Use only one kind of transition between screens; use a lighter background
(e.g. white or ecru) and/or a deeper font color (e.g. navy or black).
Formative Evaluation of an Internet Searching Tutorial
Recommended Revisions
Recommendations of Formative Evaluation (continued) Based on the data collected by the expert reviewer and the two one-on-one
reviewers, the following items should be considered for revision in order to improve the program with the hope of addressing the objectives of this program more efficiently.
Content Content assume the user has background knowledge on the topic;
simplify key terms or provide examples to explain the terms directly following the term in the write-up; put in correct answers to incorrectly answered questions to give clearer feedback.
Operation A claim statement needs to be made explaining that Authorware software
is needed in order to run the program and that it can be easily downloaded to run the program form the web version, or that it is included on the CD-ROM already.
Interaction Build in more interactive features such as practice examples where the
user could be practicing on the Internet with the instructional materials open; cut down the content and focus on the key elements used in everyday searches; add more quiz questions, provide a pre-test on some skill; add a help menu.
Formative Evaluation of an Internet Searching Tutorial
Recommended Revisions
Formative Evaluation of an Internet Searching Tutorial
Self-Evaluation
Strengths of our Evaluation The organization of the evaluation tools The use of the scripts during both types of reviews Communication within our evaluation team and the
analysis of our results.
Areas of Improvement The development of clear and specific goals and criteria for
our reviewers to ensure they meet our needs for evaluation The clarity in the objectives of the instructional materials so
that the tools developed for the evaluation are better aligned.
Formative Evaluation of an Internet Searching Tutorial
References
Fitzgerald, G., & Koury, K. (2001-2002). The KidTools Support System. U.S. Department of Education. Project #K033271. http://www.coe.missouri.edu/~vrcbd
Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. London, England: Kogan Page Limited.
Recommended