Evaluation and Program Planning, Vol. 16, pp. 211-274, 1993 Printed in the USA. All rights reserved.

0149-7189/93 $6.00 + .OO 1993 Pergamon Press Ltd.

TOWARD A RECIPROCAL RESEARCH, DEVELOPMENT, AND DISSEMINATION SYSTEM

The Struggle for a New Paradigm in the Reauthorization of the Office of Research and improvement

LAURENCE PETERS

Counsel House Subcommittee on Select Education and Civit Rights, Washington, DC

ABSTRACT

This article argues the case for in-depth evaluations of many federal education programs, par- ticu~ar~y those that provide site-specific informai~on capable of informing program improve- ment. The reauthorization of the Office of Educaiiona~ Research and improvement (OERI) provides a rare opportunity for the Department of Education to play a leadership role in this effort. It concludes with a discussion of how key features of H.R.856, a bill to reauthorize OERI, will lead to the development of a reciprocal research, development, and dissemination system.

Try to imagine an electronic technology store where products get changed on a periodic basis but no one knows what is selling and whether the new merchandise represents any improvement over the old. How long would such a store remain in business? The federal gov- ernment survives by perpetuating the illusion that some- one somewhere in Washington, D.C. is paying attention to the thousands of projects that it funds. For example, many education programs claiming to promote school improvement are funded on the basis of claims by grant writers - rarely are the peer reviewers who recommend that they be funded (the executive branch who are charged with evaluation, or Congress who appropriates the money) able to validate their claims or report back to the district as to how their model programs might be improved or used in different locations. So what then happens to the thousands of grantees at the local level that fail to live up to expectations or to those that do succeed? What legacy at the district level remains of the programs that come and go? The truth is that often we simply do not know.

Most evaluations produced by the grantees themselves are flawed not only because of obvious conflict of in- terest considerations but because at best they do little more than try to establish crude pre- and post-test ef-

fectiveness without a sensitivity to the variety of possi- ble variables influencing outcomes, effectiveness, or attention to replication in other contexts. The General Accounting Office (GAO) for example recently dis- missed the evaluations of the Drug Free Schools and Communities Act by stating that ‘<program evaluations have provided little useful information on what actually reduces student drug or alcohol use” (U.S. GAO, 1990, p. 5). Thus, of the $1.3 billion the Congress has appro- priated since the passage of the legislation in 1986, we know only a minimal amount as to the future directions of what works to prevent drug abuse. This state of af- fairs is more the rule than the exception. Of the over 200 programs that the Department manages, few receive anything but cursory scrutiny particularly in terms of what works. The Congressional Research Service was re- cently asked by the Subcommittee on Select Education and Civil Rights to survey current federal education pro- grams and identify which ones contained research de- velopment, dissemination, and evaluation activities. Of 67 activities analyzed, only 33 had an explicit evaluation component, while 22 of the 67 activities did not specify recipients of any deliverable products. (Congressional Research Service, 1991). These findings are consistent with the overall tendency of federal legislation in this

Requests for reprints should be sent to Laurence Peters, Select Education Subcommittee, U.S. House of Representatives, House Annex #I, Rm. 518, Washington, DC 20515.

271

272 LAURENCEPETERS

area to emphasize an already pronounced national bias toward faddism rather than to begin to take those steps necessary to build a systematic research, development, and dissemination system.

Notwithstanding Carroll’s insight (see Carroll, 1993), there is often a significant and complex gap between knowledge production and utilization; the system is not designed to yield even the most basic information about the nature of that gap for the hundreds of sites that could directly benefit from simply obtaining sufficient knowl- edge to improve their own programs. This situation may be changing. As Elois Scott points out (see Scott, 1993) with respect to the $6 billion Chapter 1 program, the De- partment of Education is currently conducting case stud- ies to examine promising practices. This trend represents

an encouraging break from previous emphasis on large scale collections of aggregate data and outcome perfor- mance statistics. Clearly, if there were stronger institu- tional linkages between the Office of Compensatory Education and the Office of Educational Research and Improvement (OERI), some critical steps toward long- term program improvement could be taken. OERI’s un- fortunate marginalization from any serious inclusion in the discussion with respect to Chapter 1 or any other ma- jor federal education improvement program means that federal education policymaking is often rudderless and ineffective. For federal education policymaking to have a serious impact it needs to begin to develop a fully re- ciprocal educational research, development, and dissem- ination system.’

In a new political climate in which budget deficits con- trol which of the many possible investments in human capital will be pursued with any vigor, it is now even more imperative that educational research and evalua- tion shake its poor relation image and begin to engage directly in helping to shape long-term educational poli- cymaking. For this change to occur, however, educa- tional researchers need to be able to defend themselves against the charge that the discipline lacks even the po- tential to provide credible information. A recent article in Educational Researcher entitled “The Awful Repu- tation of Education Research” (Kaestle, 1993, pp. 23- 31), encapsulates a widespread perception that the enterprise is fundamentally flawed. However, what is re- vealing from KaestIe’s interviews with those once in the thick of educational policymaking in Washington, D.C. (individuais of the caliber of John Brademas and Ernest Boyer) is that they complain not about the essential value of educational research but of its failure to become rel- evant to the task of informing policy decisions. Brademas, for example, is quoted as stating that legis- lators working on the Elementary and Secondary Edu- cation Act “were hungry for credible counsel articulated in understandable ways,” for “access to intelligence, which goes beyond information” (p. 23). Boyer notes that during his tenure as Commissioner of Education, he “never felt that research was seen as a centerpiece of

policy at any point along the way,” a climate he notes “that has gotten worse in the 1980s.“The disconnect be- tween policy and practice has a price for education de- cision makers at all levels, including practitioners. Instead of acknowledging the basis of Boyer and Brademas perspective on educational research, both re- searchers and policymakers tend to talk across each other and avoid the dialogue that would close the gap between information and intelligence, theory and its application.’

As a consequence, both sides continue to perpetuate illusions; some researchers continue to believe that low budget research and development efforts remain a via- ble option, whereas many policymakers believe that one- time demonstration projects without credible evaluations continue to be wise investments. It is this noncommu- nication that continues to impair our federal leadership ability, particularly when it comes to reaching high-risk families and children. As Elizabeth Schor has argued in Within Our Reach: Breaking the Cycle of Disadvantage, 1988, belief in magic bullets as far as this population is concerned is perennial. Schor’s argument that we have amassed a considerable amount of experience in better serving the at-risk but do not act productively on that knowledge because of a semimystical belief that, in Dr. Schor’s words, we can develop alternative “‘solutions on the cheap’ without sacrifice, miracles that change out-

comes without cost to taxpayers.” (p. 264) Some measure of realism may be in the process of per-

colating through a political system that has kept re- search, evaluation, and policymaking from producing an effective synergy. Last year Representative Major R. Owens (D-NY), Chairman of the Subcommittee on Se- lect Education and Civil Rights, was successful in hav- ing his bill, H.R. 4014, to reauthorize the Office of Educatioi~al Research and Improvement (OERI) unan- imously approved by the House of Representatives. Al- though the Senate failed to act on the measure in the last days of Congress, Representative Owens has reintro- duced a similar bill, H.R. 856, to be acted on in the 103rd Congress.

The proposed legislation aims to place OERI at the center rather than in the margins of the nation’s school improvement efforts. The bill would establish a new Na- tional Educational Research and Priorities Board and five new Institutes; the Education of At-Risk Students, Innovation in Educational Governance and Manage- ment, Early Childhood, Student Achievement and Post- secondary Education, Libraries and Lifelong Learning, In addition, H.R. 856 proposes a National Education Dissemination system to oversee an “America 2000 Communities Special Assistance Program” (hereafter referred to as “America 2000 CSAP”). The program’s purpose will be to provide guidance in the form of re- search-based technical assistance for the 50 poorest con- gressional districts to enable them to “achieve the national education goals and other objectives for educa-

A Reciprocal Research, Development, and Dissemination System 273

tional improvement through the continuous, intensified application and utilization of the results of educational research.” The program authorizes the federal govern- ment at least on a pilot basis to develop a system of tracking federal grants coming into high poverty districts to assist them as they move to identify promising and effective approaches to meet the national education goals. In addition, the new program will address the spe- cial problems surrounding urban school improvement and collect evaluation information on what works and for whom.

The initiative reflects the findings contained in the RAND study Ed~c~t~o~~l Progress: Cities ~o~ii~~e to Improve their Schools, (Hill, Wise, & Shapiro, 1988), which analyzed the work of six major urban centers in their efforts to improve their schools. The “America 2000 CSAP” proposal takes the report’s central insight that a comprehensive community-wide effort is neces- sary to improve these failing systems and translates its “bottom up” reform message for those highly impover- ished communities that lack the infrastructure necessary to mount a significant educational improvement plan. The District Education Agent will play the core role of catalysing community and business support around the six national educationa goals. Rather than solely focus- ing on curriculum frameworks and related state man- dates based on the goals, the program will work with the community to foster a new sense of long-term change.3

The five proposed Institutes will serve to quarterback the effort to provide a solid research base for the Dis- trict Education program. Administered by a director, advised by a board composed of researchers and prac- titioners intimately familiar with the implications of re- search at the classroom and school building level, the research institutes will work to ensure coherence and continuity within the research agenda. The Institutes will identify those promising innovations, whether they are within the private or federal sector, that show the most potential and will require intensive site-specific analysis.

Such an approach will not be cheap. Important stud- ies of this kind range in the millions of dollars. The RAND Corporation study, Project Alert, might be il- lustrative of the sort of approach that is needed. RAND undertook a series of longitudinal studies of Project Alert, a drug education curriculum for seventh and eighth graders. The evaluation took 7 years and cost $9 million. However, not all this research needs to be de- signed on such a massive scale. We need to learn the les- sons as to why a huge “planned variation experiment” of “Followthrough,” an extension of Head Start into the early grades, failed so completely to illuminate. The eval- uation, over a lo-year period, compared the outcomes of pupils enrolled in the follow-through and non-fol- low-through childhood education programs, at a cost of between $30 and $50 million. The research was narrowly directed to measuring which approach yielded the best results according to standardized test data. Instead of

asking the question, why certain approaches were more successful than others, the huge investment in time and resources was manipulated to develop the controversial conclusion that approaches that emphasized the “basics” were more effective than others. (House, Glass, McLean, & Walker, 1978):

We are beginning to develop approaches that, in the words of Elisabeth Schor, begin to “capture the essen- tial extra dimensions that characterize successful pro- grams” (1988, p. 268). FIPSE’s “action research” approach (see Carroll, 1993) provides perhaps one lead- ing model, and there are numerous other approaches, some pioneered in large part by public-private entities, such as Public Private Ventures (PPV) (in the area of job training) and the Manpower Development Research Corporation (MDRC) (in the area of welfare reform). The Institutes will have an added advantage of working collaboratively with locally based “Learning Grant In- stitutions” and their district education agents located in high poverty “America 2000 Communities.” These en- tities will be able to suggest sites for evaluating pro- grams, benefit from the latest research and evaluation studies, and generally connect in a comprehensive fash- ion the federal research and development investment with emerging Iocal issues. The creation of this kind of reciprocal symbiotic system is not entirely unprece- dented. The Department of Agriculture developed such an interactive dissemination system in the era that pre- ceded the Civil War. Its genesis derived from the basic insight that a great deal of research-based information, if implemented, could significantly expand the produc- tivity of American agriculture. Its system of Land Grant Universities, county extension agents, and experimen- tal testing stations were all designed to transfer knowl- edge in ways corresponding to the natural ways people accept and adapt to change-not through mandates or simple one-way distributions of information but through regular discourse and exchange. A District Education Extension Agent would be skilled in energizing volun- teers, working with community leaders and the business community and generally serving as a catalyst for change within their communities.

The Office of Educational Research and Improvement (OERI) is uniquely capable of playing a key role to guide us into a new era in which information is used strategi- cally instead of bureaucratically. The stated mission of the office is to “achieve the goal of quality education” through the “continued pursuit of knowledge about ed- ucation through research, improvement activities, data collection and information dissemination.” It is not suf- ficient for OERI to be simply instrumental in putting in place curriculum standards, school delivery standards, and outcome measures either for schools or students, however valuable that may be. OERI must be involved in the long-term and messy business of school change for the expanded group of individuals that now include community and business leaders as well as teachers, par-

274 LAURENCE PETERS

ents, and principals, who now must be involved in the daily job of school reform.

All of this will not be easily achieved. Powerful inertia serves to maintain educational research and evaluation within a politicized orbit. The old paradigm in which program fragmentation mirrors a larger more pernicious disorganization thriving on a weak management center and a confused research mission benefiting from an op- portunistic notion of research in which rapidly chang-

ing priorities serve to justify almost any new politically driven fashion. We need to understand that to fully change to a new research and development paradigm, one that places the expanded constituency of knowledge seekers and users at its center, will take a new consis- tency of purpose and seriousness of effort. H.R. 8.56 sig- nals the beginning of the discussion that must now take place.

ENDNOTES

I. In the context of the well-known fragmentation of Department

of Education programs, the statement by James B. Thomas, Jr., In-

spector General of the U.S. Department of Education, provides an

interesting commentary on the challenge posed by the lack of program

linkages: “Fragmentation in the way programs are created and admin-

istered is a hindrance to carrying out the Department’s mission and

work toward attaining the Goals. As special areas such as early child-

hood education capture the attention of the Federal government, the

approach has been to add existing laws or to write new laws with the

end result being the creation of new programs. Often, this means that

programs targeting similar types of areas are administered by differ-

ent principle offices in the Department. This approach to creating pro-

grams somewhat impedes what is really needed-a strategic approach

to solving the complex problems like those emphasized by the Goals.”

(Subcommittee on Labor, Health and Human Services and Education

Committee on Appropriations on February 16, 1993, p. 6).

2. These comments by policymakers on the essential disconnect be-

tween policy and practice are not confined to the United States. The

same phenomena is reflected in the U.K., where research, in the words

of Ernest Boyer was “not out in front of action.” It is hard, for exam-

ple, to find a single piece of research evidence that supported the sweep-

ing changes contained in the 1990 Education Reform Act that centered

on school choice. The words of one British Permanent Secretary 01

the Department of Education and Science before a House of Com-

mons Select Committee perhaps indicates why in framing the sweep-

ing 1990 Education reform bill, researchers were ignored. “I have to

\ay. of course that the great thing about educational research is that

a part of it is rubbish and another part (I will not be specific about

the proportions) leads nowhere and is really indifferent; it is 1 am afraid,

exceptional to find a piece of research that really hits the nail on the

head and tells you pretty clearly what is wrong and what is happening

or what should be done” (Simon & Taylor, 1981, p. 164).

3. The authors of the RAND study (Hill, Wise, & Shapiro, 1989)

express the potential of community involvement as follows, “the seri-

ous sharing of concern and responsibility demands accountability to

community expectations. People want serious measures of progress,

not sterile scorekeeping. Civic and business leaders expect to knot+

about test scores, but they also want to know how money is being used,

how teachers are being selected, prepared, monitored and evaluated,

how resources are distributed to schools, and what learning opportu-

nities students really have.” (p. 38)

4. It was perhaps this kind of evaluation that Schor had in mind

when she wrote: “. the reasonable demand for evidence that somc-

thing good is happening as a result of the investment of funds often

exerts unreasonable pressures to convert both program input and out-

comes into whatever can be readily measured. This rush to quantify,

which engages funders, policymakers, academics, policy analysts, and

program administrators alike has had damaging effects on the devel-

opment of sound interventions aimed at long term outcomes. Programs

are driven into building successes by ducking hard cases. Agencies shy

away from high-risk youngsters, who provide scant payoff for effort

expended when it comes to bottom-line totals. Energy is diverted into

evaluation research that asks trivial questions and sacrifices signifi-

cance to precision.” (Schor. 1988. p, 268)

REFERENCES

BECKER, C. (1978). The national evaluation of follow-through be-

havior-theory-based programs come out on top. Education and Ur-

ban Society, fO(4). 43 l-458.

SCHOR, Dr. (1988). Within our reach: Breaking the cycle of disud-

vantage. New York: Anchor Press.

SIMON, B., & TAYLOR, W. (1981). Educalion in [he f9RO.s. In B.

CONGRESSIONAL RESEARCH SERVICE. (1991). Memorandum Simon & W. Taylor (Eds.), Lundon: Batsford.

concerning education research, development, dissemination and eval-

uation activities. Washington, DC: Author. U.S. GENERAL ACCOUNTING OFFICE. (1990). L)rug education:

School-bused programs seen as useful but impact unknokzn. Wash-

HOUSE, E.R., GLASS, G.V., MCLEAN, L.D., & WALKER, D.F.

(1978). No simple answer: Critique of the follow through evaluation.

Harvard Educafional Review, 48(2), 128-l 59.

ington, DC: Author.

U.S. GENERAL ACCOUNTING OFFICE. (1992). Edmation i.smr.s.

Transition Series. Washington, DC: Author.

KAESTLE, C.F. (1993). The awful reputation of education research. U.S. CONGRESS HOUSE COMMITTEE ON EDUCATION AND Educational Researcher, 22(l), 23-3 1. LABOR. SUBCOMMITTEE ON SELECT EDUCATION. (1991).

Education 2005: The role of research in an overwhelming campaign RAND. (1989). Educalionalprogress: Cities mobilize to improve their for education in America (Serial No. 102-M). Washington, DC: U.S. schools. Hill, Paul, Wise E., Shapiro 1.. Government Printing Office.