Articles

An Evaluation of the Grade Improvement Materials, “Where There’s a Will There’s an A”

LARRY WEBER and THOMAS M. SHERMAN

Education is an enormous enterprise in the United States. Funds devoted to it exceed all other public expenditures, including defense, and amount to billions of dollars annually. Because of the tremendous size of public and private education, a huge industry has emerged to produce necessary academic materials and supplies to support it. Initially these products were developed for distribution to and use by the schools themselves. However, recently an increasingly large number of educational products have been produced for direct marketing to the public. These materials include grade improvement, reading, vocabulary development, and mathematical products that have been widely advertised in mass market media such as radio, cable TV, and tabloid newspapers.

Because educational products have traditionally been purchased by school personnel, it might be assumed that professional judgment was exercised in evaluating and selecting them. As such, there was some security that the quality and value of products could be assured. The same assurance may not be available with the newer direct market products. Rarely, if ever, are claims about their quality supported by research evidence. Generally their value is based on individual testimonials, which frequently are solicited and likely influenced by the product’s manufacturer. Thus, it is possible these products not only do not perform as advertised but also could be worthless, or even harmful. It is unfortunate that, unlike the scrutiny and accountability to which we subject materials and programs used in the public schools, no requirement is made of commercial producers of educational products that are sold directly to the public.

This study is an initial effort directed at evaluating one such program. The purpose of this research was to evaluate the grade improvement materials, “Where There’s a Will There’s an A” (Olney, 1988, 1989). They are allegedly designed to provide skills and techniques that are useful in helping students achieve better in school. Since no evidence, other than testimonial statements of students, teachers, and parents, etc., of the effectiveness of the materials is available (according to conversations with the author) and in consideration of the seller’s claim that over one million copies of the materials have

Larry Weber and Thomas M. Sherman l Division of Curriculum and Instruction, College of Education, Virginia Polytechnic

Institute and State University, Blacksburg, VA 24061-0313.

Evaluation Practice, Vol. 14, No. 2, 1993, pp. 127.131. Copyright @ 1993 by JAI Press, Inc.

ISSN: 0886-1633 All rights of reproduction in any form reserved.

127

128 EVALUATION PRACTICE, 14(2), 1993

been sold, it appeared important to investigate the accuracy of assertions made that students’ academic performance is improved quickly and easily (e.g., “moving from a ‘C average’ to an ‘A average’ in a single academic term” and “making all A’s and B’s instead of C’s and D’s”).

METHOD

Subjects and Setting

This study used comparison groups comprised of higher education students enrolled in four undergraduate educational psychology classes at a large middle Atlantic region land grant university. All were upperclass students preparing to become elementary and secondary teachers. They numbered 123, of which 98 volunteered to participate in the study. Students were assigned randomly to experimental and control groups.

Procedures

Professor Olney’s instructional materials come in three versions directed at students at the elementary, secondary, and higher education levels. (The research reported in this paper used only college-level students.) They include print, audio, and video formats. The materials focus on twenty propositions (suggestions) which purport to enhance a student’s ability to make satisfactory progress in college and get better grades. They include obviously worthwhile recommendations like faithfully attending class and taking good notes, as well as gimmicks like dropping courses to “cut losses” and pestering professors about grades.

The experimental group was shown the complete video program during the first week of the fall semester and was provided access to it throughout the term. They also were given a copy of a companion instructional booklet to keep; the booklet is part of the

package that all customers receive when they purchase the video version for college students. The control group did not view the video or receive the booklet.

An examination (immediate post-test) based on the grade enhancement suggestions contained in the video tape and in the companion booklet was constructed and administered to students in the experimental group immediately after they viewed the videotape. (The same test was given earlier to students in the control group, after which time they were dismissed from the experiment.) The purpose of the examination was to measure the students’ knowledge of the material contained in the video tape and to ascertain if the scores of the two groups were different.’

At the end of the fall semester (approximately four months later) the same examination (end of semester post-test) was administered to both groups. The purposes of the end of semester tests were to measure retention in the experimental group and to determine if the groups’ scores differed.

In addition to the above data, which provided evidence about the effectiveness of the program material for teaching its content, other data were collected which yielded information about its value for enhancing achievement in college. Since much of the testimonial information presented in advertisements addressed grade improvements, comparisons between the experimental and control groups were made on the following dependent criterion variables:

‘Where There’s a Will There’s an A” 129

1. Course grade in Educational Psychology 2. Class rank in Educational Psychology 3. Fall semester grade point average (GPA) for all courses 4. Spring semester GPA for all courses.

RESULTS

Useable data were available on 73 students, 36 in the experimental group and 37 in the control group. Findings revealed that the experimental group scored significantly higher than the control group on the immediate post-test examination administered on the night of the treatment, and on the end-of-semester post test. The average percent correct scores earned by both groups are shown below. The differences between the two groups were significant for both testing times.

Experimental Group

X Score

Control Group

X Score t

Value Probability

Immediate Post-Test End-of-Semester Post-Test

These data indicate information presented in

94% 67% 20.5 .OOOl 88% 71% 8.8 .OOOl

that the experimental group learned the grade enhancement the program and retained it throughout the semester. Thus, it

appears that the video tape and booklet were effective for conveying the information advocated in them, concerning grade improvement strategies for college level students. They also indicate that the knowledge between the two groups was significantly different, with the experimental group having superior knowledge of material presented.

However, when tests on the criterion variables were conducted, no significant differences between experimental and control groups were found. These data are shown below.

Experimental Group

X Score

Control Group

X Score t

Value Probability

Educational Psychology Course Grade Mean Class Rank Fall Semester GPA Spring Semester GPA

3.54 3.55 -.04 .97 9.26 9.50 -.18 .86 3.11 3.10 .07 .94 3.11 3.23 -.83 .41

Based on the above data, it appears that the grade improvement materials entitled “Where There’s a Will, There’s an A” were not effective as a means to improve academic achievement for this group of college students.

130 EVALUATION PRACTICE, 14(2), 1993

DISCUSSION

Since the early studies of Whipple and Curtis (1917) who characterized academic achievement as the relatively simple implementation of overt skills, there has been a tendency to portray academic learning as more and more complex. Robinson (1970) proposed a sophisticated heuristic for study, which implied that learning involved strategy as well as skills. More recently applications of cognitive science and behavioral

research have produced additional themes in the study of academic learning (Brown, 1978; Entwistle, 1979; Sherman, 1984). From the literature it seems reasonable to conclude that, like most other human capabilities, academic success will be influenced by a variety of factors, e.g., skills, aptitude, strategies, and metacognitive abilities. In spite of the evidence, materials and programs which purport to provide quick and easy

solutions to all types of educational deficiencies continue to be proposed. The study improvement literature suggests that procedures designed to enhance student learning which use short term and simple approaches have limited potential (Sherman, 1985).

In contrast, effective instructional programs for study improvement offer support to students over longer periods of time and use a variety of strategies (Ford, 198 1; Weinstein

& Mayer, 1986). The approach and content of the materials evaluated in this study appear inconsistent

with this relatively well-developed body of research. For example, one celebrity spokesperson states, in a television commercial about the materials, that “you’d be amazed what one three-hour seminar can do.” If there is any lesson to be learned from years of research and practice about strategies for improving academic achievement, it is that a three-hour seminar will not produce dramatic gains. The results of this study support that

conclusion. While some of the suggestions made in the “Where There’s a Will There’s an A” are

based on sound psychological principles of learning and instruction, other materials are “gimmicky” superficial devices, which portend to be effective. For example, it is suggested that students register early before the “good” classes fill up; drop hard courses before the

drop deadline; and take tough courses at other universities and transfer the credits. Students are also advised to write using an erasable ink pen while studying, on the basis that it’s a means to assure that learning is complete; and how to get jobs, scholarships

and loans without interest. Olney (1989) states that the purpose of the materials is to help students get better grades with less effort. The substantive advice offered includes going to every class, taking good notes, turning in only well-written papers that have been thoroughly revised and edited, turning work in on time, doing all extra credit assignments, regular self testing, devoting time and energy to studying, and reading texts before class. It is unclear how engaging in these commonly advised practices will be effortless. Moreover, the materials appear to offer nothing beyond common wisdom. For most, knowing is not the problem; it is in the doing that difficulty arises.

There are several limitations in this study. It was conducted on a small number of selected students, and the authors caution about generalizing to other subjects whose make- up varies widely from those who participated in this experiment. Moreover, the fact that the grade point averages of both groups was high on all comparisons that were made may have interfered with our ability to demonstrate significant differences because of ceiling effect; and the failure of some students to volunteer to participate (less than 25% of the

“Where There’s a Will There’s an A” 131

students in all classes) cast additional doubt on the representativeness of our subjects. Regardless, we believe our study was so sufficiently rigorous in its design and implementation and that the findings are so straightforward and consistent throughout all comparisons, that our conclusions about the quality of the achievement enhancement materials “Where There’s A Will, There’s an A,” are justifiable.

Finally, we would like to say that panaceas to cure the learning problems of students are not new to education. The public schools have been inundated with them for decades.Recently more ventures into education by private commercial groups have occurred. Regrettably, there is no requirement that they demonstrate the efficacy of their educational programs and materials. This study is an initial evaluation of one of these direct-market commercial programs. It is hoped that it might provide an impetus for the investigation/ evaluation of similar programs, such as reading improvement programs, privately operated learning centers, and workshops designed to improve standardized test scores on national examinations.

NOTE

1. The KR-20 reliability coefficient for the combined groups of students (experimental and

control) was .86.

REFERENCES

Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Gloser (Ed.), Advances in instructionalpsychology. Hillsdale, N.J.: Erlbaum.

Entwistle, N. J. (1979). Stages, levels, styles or strategies: Dilemmas in the description of thinking. Educational Review, 31, 123-3 1.

National Commission on Excellence in Education. (1983). A nation at risk. Washington, D.C.: United States Department of Education.

Ford, N. (1981). Recent approaches to the study and teaching of “effective learning” in higher education. Review of Educational Research, 51, 341-377.

Olney, C. W. (1988). Where there’s a will there’s an . . . A. Paoli, PA: Chesterbrook Educational Publishers.

Olney, C. W. (1989). Where there’s a will there> an . . . A. Paoli, PA: Chesterbrook Educational Publishers.

Robinson, F. P. (1970). Effective study (4th ed.). New York: Harper & Row. Sherman, T. M. (1984). Proven strategies for successful learning. Columbus, OH: Merrill. Sherman, T. M. (1985). Teaching academic learning skills in college learning improvement program.

Paper presented at the annual meeting of the American Educational Research Association, Chicago, Ill.

Weinstein, C. F., & Mayer, R. F. (1986). The teaching of learning strategies. In M. C. Wittrock (Ed.), Handbook of research on teuching (3rd ed.). New York: Macmillan.

Whipple, G. M., & Curtis, J. N. (1917). Preliminary investigations of skimming in reading. Journal of Educational Psychology, 8, 33-49.