. . . . . . . . ..~.........~..~...~..~...~~~~~..~~~~.....~...~....~~~~......~...~............................~..University of Wisconsin-Madison * National Center for Improving Science Educafion

Workshop Report

Research on SWhat Have We We Need to Know?

Synthesis of t

Volume 2: Br

William H. Clune,Dianne C. Bowcoc& Ricardo Mesquit

L. Webb,

Funded bNation

National Institute for Science Education (NISE) Publications

The NISE issues papers to facilitate the exchange of ideas among the research anddevelopment community in science, mathematics, engineering, and technology (SMET)education and leading reformers of SMET education as found in schools, universities,and professional organizations across the country. The NISE Occasional Papers pro-vide comment and analysis on current issues in SMET education including SMET inno-vations and practices. The papers in the NISE Research Monograph series report find-ings of original research. The NISE Conference and Workshop Reports result fromconferences, forums, and workshops sponsored by the NISE. In addition to these threepublication series, the NISE publishes Briefs on a variety of SMET issues.

The preparation of this paper was was supported by a cooperative agreement between the NationalScience Foundation and the University of Wisconsin-Madison (Cooperative Agreement No. RED-9452971). At UW-Madison, the National Institute for Science Education is housed in the WisconsinCenter for Education Research and is a collaborative effort of the College of Agricultural and LifeSciences, the School of Education, the College of Engineering, and the College of Letters and Science.The collaborative effort is also joined by the National Center for Improving Science Education,Washington, DC. Any opinions, findings, or conclusions are those of the author and do not necessarilyreflect the view of the supporting agencies.

Workshop Report No. 4

Research on Systemic Reform:What Have We Learned? What Do We Need to Know?

Synthesis of the Second Annual NISE ForumVolume 2: Proceedings

William H. Clune, Susan B. Millar, Senta A. Raizen, Norman L. Webb,Dianne C. Bowcock, Edward D. B&ton, Ramona L. Gunter, & Ricardo Mesquita

National Institute for Science EducationUniversity of Wisconsin-Madison

December 1997

i

The writers wish to acknowledge Andrew C. Porter for his valuable comments on our manuscripts, SarahK. A. Pfatteicher for organizing the development of this document during spring 1997, Deborah Stewart

for her skilled copy editing and document management, and Paula A. White and Becky S. Torrisi forattending to the many other tasks needed to complete the document.

Contents

Contents of Volume 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . ..*..............................*........................................................... vii

Forum Agenda. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..*................*.........*.*...*......................... ix

Status and Trends of Systemic Reform: Research on Systemic Reform

Pascal D. Forgione, Jr................................................................................................. 1Luther S. Williams........................................................................................................ 3

What Have We Learned? What Do We Need to Know?

William H. Clune .......................................................................................................... 5Susan Fuhrman ............................................................................................................ 8Jane Butler Kahle ....................................................................................... . ............... 1 1Uri Treisman .............................................................................................................. 1 6Daryl Chubin ............................................................................................................. 2 1

The Role of Curriculum in Systemic Reform

Rodger W. Bybee ........................................................................................................ 23Margaret B. Cozzens .................................................................................................. 28Richard Greenberg .................................................................................................... 39Sylvia T. Johnson ....................................................................................................... 4 1Thomas A. Romberg ................................................................................................... 46

The Role of Teaching and Learning in SystemicReform

Christopher Dede ....................................................................................................... 50LeRoy Lee ................................................................................................................... 52Susan Loucks-Horsley................................................................................................ 54Shirley Malcom ......................................................................................................... .60Lauren B. Resnick ..................................................................................................... .62

The Role of Evaluation in Systemic Reform

Tom B. Corcoran ....................................................................................................... 64Paul B. LeMahieu ...................................................................................................... 69Norman L. Webb ........................................................................................................ 7 1Iris R. Weiss ............................................................................................................... 73Andrew A. Zucker & Patrick M. Shields.. ................................................................. .75

. . .1 1 1

Forum Highlights and Looking AheadCora B. Marrett ......................................................................................................... 80Michael W . Kirst ........................................................................................................ 83Larry E. Suter ............................................................................................................. 87

Appendixes

A: Current Agencies Involved in Curriculum Improvement ................................................. 8 9B: Comprehensive Mathematics Curricula & Funded Projects with Contact Information.. .90C : Comprehensive Science Curricula.. ................................................................................. .92D: Criteria for Judging SSI Strategies.................................................................................... 9 3

iv

Contents of Volume 1

Contents of Volume 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Forum Agenda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Status and Trends of Systemic Reform: Research on Systemic ReformPascal D. For&one, Jr. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Luther S. Williams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

An Introduction to the Papers and Think Piece Themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6William H. Clune and Norman L. Webb

Themes Articulated in Forum Participant Think Pieces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Susan B. Millar, Dianne C. Bowcock, Ramona L. Gunter, and Ricardo Mesquita

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..‘...............................................13

1 . The Definition of Systemic Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2. Implementation of Systemic Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3. Curriculum in the Context of Systemic Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4 . The Critical Value of Evaluation of Systemic Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Forum Highlights and Looking AheadCora B. Marrett ................................................................................................................. 47Michael W. Kirst ................................................................................................................ 50Larry E. Suter ...................................................................................................................... 54

AppendixesA: Demographic Information about Forum Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

B: Contact Information for Forum Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

V

Preface

The goal of the Second Annual NISE Forumwas to generate and share knowledge of systemicreform, including the role of curriculum, the role ofteaching and learning, and the role of evaluation.To reach this goal, panelists from around the nationshared their expertise in these various areas withparticipants who themselves brought variedexpertise to structured conversations and network-ing sessions.

These two volumes are the result of a collabo-ration among four NISE teams and, more broadly,of a collaboration among the NISE researchers, theForum panelists, NSF staff, and the Forum partici-pants. The Interacting with Professional Audiences(IPA) team, Policy Analysis of Systemic Reform(PASR) team, Strategies for Evaluating SystemicReform (SESR) team, and Formative Evaluation(FE) team worked together to design, implement,and evaluate the Forum. The four teams thencollaborated in the writing of Volume 1. The FEteam (Susan Millar, Dianne Bowcock, RamonaGunter, and Ricardo Mesquita) analyzed the 483think pieces written during the Forum and pro-duced the “Themes Articulated in Forum Partici-pant Think Pieces.” Senta Raizen and Ted Britton(IPA), William Clune (PASR), and Norman Webb(SESR) supported the FE team’s analysis byframing key themes based on their readings of thethink pieces. William Chme and Norman Webbdrew heavily on the FE team’s analysis of the thinkpieces as well as the panelists’ papers in writingtheir introduction.

Given the way this document was developed,readers should view the two volumes as elementsof a conversation through which a better under-standing of systemic reform is emerging.

Volume 1: Analysis

l Introduction to the Papers and Think PieceThemes. In this introductory section,

William Clune and Norman Webb integratethe other two sections by developing someof the more compelling points raised by theparticipants’ think pieces and the panelists’contributions. William Clune is Voss-Bascom Professor of Law at UW-Madisonwith research interests in education lawand policy, systemic reform, and adequacyin school finance. Norman Webb is asenior research scientist for WCER withresearch interests in mathematics educa-tion, evaluation, and assessment.

l Themes Articulated in Forum ParticipantThink Pieces. During three breakoutsessions, Forum participants were asked towrite ‘wlink pieces” in reaction to thepanelists’ remarks. This synthesis of thethemes in the think pieces, including manyquotations from the writers, provides thereader a window on issues, questions, andcontrasting viewpoints about systemicreform that the Forum’s varied participantsarticulated.

Volume 2: Proceedings

l Papers Presented. The ideas, knowledge,and experience of 24 panelists involved inimplementing, researching, and evaluatingsystemic reform appear in this section.Most of these papers were prepared aheadof time and distributed to participants.Others are transcriptions of the panelists’remarks. (Print and audio versions ofseveral of these are available on theTEECH Web site, ht@://teech. terc. edu/modes/papers/systemic4apers.cfi)

Both volumes include papers from the openingplenary session and from the closing plenarysession.

vii

Second Annual NISE Forum: February 24-25,1997

AGENDA 1Research on Systemic Reform:

What Have We Learned?What Do We Need to Know?

Monday February 24

8:30 - lo:30 Opening PlenaryAndy Porter, chair

Status and Trends of Systemic ReformLuther Williams, Marshall Smith

What Have We Learned? What Do We Need to Know?Bill Clune, Susan Fuhrman, Jane Butler Kahle, Uri Treisman;Daryl Chubin, discussant

lo:30 - lo:45 Break

lo:45 - 12:00 Breakouts

What Have We Learned? What Do We Need to Know?

12:lS - 1:30

1:30 - 3:30

3:30 - 3:45

3:45 - 5:oo

Lunch

Afternoon PlenarySenta Raizen, chair

The Role of Curriculum in Systemic ReformRodger Bybee, Midge Cozzens, Richard Greenberg,Sylvia Johnson, Tom Romberg

The Role of Teaching and Learning in Systemic ReformChris Dede, LeRoy Lee, Susan Loucks-Horsley, Shirley Malcom,Lauren Resnick

Break

Breakouts

Curriculum, Teaching and Learning in Systemic Reform

ix

Tuesday, February 25

8:30 - 9:30 Opening PlenarySenta Raizen, chair

The Role of Evaluation in Systemic ReformBernice Anderson, Tom Corcoran, Paul LeMahieu, Norman Webb,Iris Weiss

9:30 - 9:45 Break

9:45 - lo:45 Breakouts

Evaluation in Systemic Reform; Discussions with Panelists

Monday’s Panelists Rodger Bybee, Jane Butler Kahle, Bill Clune, Chris Dede,Richard Greenberg, Sylvia Johnson, Lauren Resnick,Susan Loucks-Horsley, LeRoy Lee, Uri Treisman

Tuesday’s Panelists Bernice Anderson, Iris Weiss, Tom Corcoran, Paul LeMahieu,Norm Webb

11:OO - 12:30 Closing PlenaryAndy Porter, chair

Forum Highlights and Looking AheadMike Kirst, Cora Marrett;Larry Suter, discussant

X

Status and Trends in Systemic Reform:Research on Systemic Reform

Pascal Forgione, JI:Commissioner of Education Statistics,Department of Education

I bring you greetings from Acting DeputySecretary Marshall “Mike” Smith who reallywanted to be here this morning. Mike asked meto reflect briefly on the impact of the Third Inter-national Mathematics and Science Study(TIMSS) and what it has done for the characterand quality of the national conversation aboutsystemic reform. While on the one hand, sys-temic reform is very complex and challenging, itis also very easy to trivialize this construct. It’seasy for people to define what they are doing assystemic without actual substance or verification.

Fortunately, TIMSS is not about quick fixes.In fact, the Grade 8 data-all that’s been releasedthus far-suggest that there are no simple solu-tions. TIMSS has provided data to explode threemyths. I hope in June when the Grade 4 data areout that we confirm these findings. First, eighthgraders in the United States have more hours ofinstruction in science and mathematics than Ger-man or Japanese students. Despite our shorterschool year, if you take the length of the periodsand the frequency per week at eighth grade com-pared to Japan and Germany, U.S. students inGrade 8 have more hours of instruction.

Second, on our questionnaires, Americanteachers reported assigning more homework thantheir Japanese or German counterparts. Andwhen we looked at the after-school work, all ofthe work put together, American students do asmuch homework as their German and Japanesecounterparts.

Finally, if you take three hours of TV watch-ing per night as heavy TV watching, as manyJapanese children are heavy TV watchers as chil-dren in the United States. So these three simplemyths of time, homework, and TV are not what

, it’s about.There are not going to be simple answers.

We’ll see whether the next two data sets-Grade

4 in June 1997 and Grade 12 in spring 1998-confirm these initial findings.

The TIMSS design overtly includes the keyelements emphasized in a notion of systemic re-form. The study had five major components thatutilized a variety of methods to examine thebreadth and the depth of the essential elements.Basically, one only has to look at the table ofcontents of our report. This report tries to giveAmerica a systemic view about achievement,curriculum, teaching, the lives of our teachers,and the lives of our students. In fact, the Grade 8summary report called Pursuing Excellence waswritten explicitly to be read on an airplane in onehour by a businessman. Now remember, this is astatistical report. It may not appear to be a statis-tical report when you first look at it, becausethere are no standard errors; we don’t talk aboutstandard deviations. We tried to write in clearEnglish about what these findings mean. I canassure you that every sentence went through astrict adjudication review process. Everything inthe report is supported by data.

Let me highlight for two areas the kinds ofquestions that TIMSS is raising for this Nation.The first is the area of curriculum where Profes-sor Bill Schmidt has developed very innovativemethodology to look at textbooks and curriculumof the fifty countries. The kind of questions weasked in our report under curriculum included:

. Who sets the curriculum?

. Is the curriculum as focused as in other coun-tries?

. Is it as advanced as in other countries?

. What do we mean by advanced?n What is the content that really is advanced at

the eighth-grade level and how much time isspent in class?

I

These are the kinds of powerful questionsthat we need to engage the Nation on and builddeeper understanding through the analysis of theTIMSS database. It will take over a year to rollout the three populations of TIMSS data, fromNovember 1996 (Grade 8), through June 1997(Grade 4), to spring 1998 (Grade 12).

The second area that I want to emphasize isthe innovative work of Professor Jim Stigler whoconducted a three-country classroom video studyof mathematics teaching. This is only Grade 8mathematics and only in Germany, Japan, and theUnited States. It’s unfortunate that we were notable to include Grade 8 science, but this studywas quite an innovation five years ago. Thisvideo study answers the kinds of questions thatwe could never answer with questionnaires oreven case studies:

.

.

.

.

.

.

How do teachers in the three countries teach?How do the instructional practices differ?How do they organize and present their les-sons?What are the cultural scripts of the lessons?What is the role of the teacher and the stu-dent during seat work and then class work?Are the students passive or active learners?What proportion of class time is spent ininstructionally relevant activities? in socialor housekeeping activities?Are teachers aware of the reform?Do they think they’re doing it and do we seeit?

This video study included a half nationalsample in the three countries. We have 81 lessonsof a U.S. teacher. We selected the schools; weselected the teachers; and we selected the classperiod. It’s a time warp capturing Americanteaching at Grade 8 in mathematics, and we’venever had that before. We also have made avail-able a CD-ROM and a videotape with six les-sons, two for each country, from the study. Forexample, you are able to look at a Japanese alge-bra lesson and compare it to an American one. It

stimulates a terrific conversation about qualityteaching.

TIMSS also taught us to be careful aboutdrawing conclusions, even with statistical stan-dards such as reliability. There’s a need to con-tinue to verify and to confirm the data. Let megive you an example. American teachers, 95% ofthem, said they knew the NCTM standards, and75% of them said they were implementing them.When we examined the videotapes we found afocus on conceptual thinking in only 20% of thelessons. Again, what people called high levelmathematics often ended up being the processesand not the deep mathematics. Thus, we werecautious in reporting the mathematics question-naire data because the mathematics video datadid not confirm the claim. In science, we hadvery similar data. Grade 8 science teachers saidthey also were implementing the standards. Weheld off releasing the findings because we werenot able to corroborate the data. So we need to becareful in our survey reporting.

It’s also ironic that in TIMSS we probablyhave more information about American curricula,textbooks, teaching, and instruction than in anyother national database. This database is the kindthat we need to build, with rich videos and cur-riculum components. It is a limited database inthat it’s cross sectional and not longitudinal, andit also doesn’t often go to the student level. ButTIMSS whets our appetite.

I’m pleased to say that Mike Smith, NealLane, director of the National Science Founda-tion, and Ernie Gibbons, the President’s advisoron science, are working with the Domestic PolicyCouncil to assure cross-agency follow throughand coordination. Wouldn’t it be wonderful if thefederal government could be systemic about ourpartnerships to support the challenging reformsin science teaching and learning? I think we’rebeginning to have the collegial and cooperativeconversations.

Thank you for this opportunity to share withyou a brief overview of the TIMSS findings. Ilook forward to being your partner as we moveforward in this exciting work.

2

Luther S. WliamsAssistant Director for Education and Human Resources,National Science Foundation

I’m going to make a few comments bearingon the topic of status and trends of systemic re-form. They are the end results of observations,experiences, and thinking about systemic reformin the context of NSF’s programs. I will try toframe them vis-&is your conference. You askedtwo very important questions about research onsystemic reform: What have we learned? Whatdo we need to know? The answer to the secondquestion is, “Substantial”; it is probably limitless.About the first question, if by research you meana fundamental exercise informed by some theoryor hypothesis that leads to substantial advance-ment of the knowledge base, the answer is, “Verylittle.” I would argue that the size of the knowl-edge base is one of the challenges in anantiquality context to systemic reform. Few stud-ies, in my judgement, configure to equal re-search.

promote a comprehensive restructuring in a sys-tem that is highly resistant to Unitarian ap-proaches. That recognition, in my view, from aresearch perspective, is one of the most impor-tant factors that should bear on any research de-sign. There is essentially no f=ed set that definesthe problem that one is going to take up. None-theless, under the rubric of NSF’s and others’programming, substantial progress has beenmade in a variety of states and in large urbancommunities. My sense is that progress has ob-tained insights that have done a very good job ofcontrolling for a series of antisystemic orantiquality forces. I want to spend a few minutessharing my list of those insights.

There’s another observation with respect tosystemic reform: Today, almost every conceiv-able machination bearing on K-12 education,and now increasingly higher education, can ac-commodate itself under the rubric of systemicreform. That’s okay at some level, and that is in-tegral to the American culture. But it does makevery challenging the maintenance of focus onsystemic reform, particularly of a K-12 system. Imake that observation because systemic reform isin the context of a system. In the case of NSF’sprograms in the K-12 domain focusing only onscience and mathematics, a very powerful lessonthat’s been learned over the past several years ishow exceedingly complex that system is. Thatfinding of complexity, I would submit, is rarelyacknowledged, whether one is talking about astate or a large urban community. Complexity ismade worse by the fact that systems are highlyfragmented. In effect one is attempting to pro-mote reform in a system that is elusive and insome instances presents itself in multiple con-structions in real time. The system is also ex-ceedingly conflicted--even contradictory-ow-ing to counter-instance agendas, roles, partici-pants, and players. Therefore one is attempting to

One of the most important challenges is toeffect the conversion by process from a tmidi-mensional to a multifaceted approach. All of theelements of the system are counter to a multifac-eted approach; the system seeks to devolve to aunidimensional construction.

The second issue that has to be overcome isconcerned with a very challenging nonlinear,very dynamic, ever-changing problem construc-tion. It is made more difficult if emphasis isgiven to mathematics and science learning as op-posed to educational processes. That is a veryimportant distinction.

Third, systems that have made substantialprogress have been very explicit and have almostelevated to a mandatory level the required infra-structure that is needed for mathematics and sci-ence learning. Pat Forgione alluded to part of thisin talking about one dimension of one of the vari-ables that equal this very, very complicated con-struction. Let me extend that further. All of theresearch he discussed has to do with eighth-grademathematics, but his question is, “At the end ofthe day, what is the value of the take home?’ Oneknows nothing about eighth-grade science; oneknows nothing about the K-7 enterprise that ac-tually informed mathematics learning. Unlessone thinks eighth-grade mathematics is the endpoint, it is comparatively inconsequential relativeto the rest of the sequence. I’m trying to empha-

3

size the vast domain about which one does nothave comparable findings. We need to make thetransition to emphasis on science learning in thecontext of the total system.

The fourth factor that has been very impor-tant where progress has been made is having in-dividuals who operate from the context of atheory of reform, at least for their urban schooldistrict or their state. The reform was not haphaz-ardly deposited in a series of machinations in asystem without some reference to why, in fact,one is engaging in an explicit set of activities:What is the chronology in which one engages inthose activities? What are the goals? What arethe road maps? What are the strategies associatedwith those? What kind of formative evaluation isconducted in order to make early adjustments,and what outputs are valued? Is it the orderlyprogression year by year through the system, oris it the definitive outputs associated with sciencelearning?

Progress has also been made in systemswhere there has been substantial attention to ef-forts to manage the role of others, including thefederal government, which is not unitary in itscontributions to standard mathematics and sci-ence education. How does one reconcile all ofthe conflicting inputs that bear on systems fromWashington through funding and through othermachinations? Similarly, state apparatus is an en-tity that requires some accommodation-witnessthe difference among the goals for learning in thevariety of national or state assessment systemsthat are present across the states. What do theycommunicate differentially about the reasonableoutcomes?

Another factor is higher education. Scienceand mathematics reform is critically dependanton the contributions from the higher educationcommunity. But there is one matter on which Iwould argue some thoughtful consideration, ifnot apprehension, needs to obtain. The sugges-tion that higher education as a domain is an ex-emplar of successful conclusion of reform or re-structuring requires one to engage in a supremefiction. There are enormous contributions to behad from the higher education community. (It ismostly scientific and technical expertise.) It isnot a sector that schools should turn to for under-

standing school systems. In contrast, some schooldistricts have done a very good job of workingout arrangements with the business sector. Thatsector actually does have experience in this arenaof systemic reform or restructuring. The businesssector has taken on very challenging multifacetedproblems and made progress. So there’s a lot tobe learned from that sector. I would argue thatschool systems that have made great progress be-yond having very productive relationships withthe business sector have forged very excellentrelationships with a variety of community-basedorganizations that have been able to bring to bearan advocacy that is not found in other sectors.They’ve also catalyzed parents and employedthem in the support of their efforts.

Successful sites also recognize that they’reworking in a very complicated, multifaceted,ever-changing, nonlinear system. Early on, thesesites included “strategies for innovation replica-tion” to develop some sense of how to scale up.Stringency is brought to documenting, communi-cating, and securing gains, so that they in effectbecome demonstrable nodes toward reform. Onecan continue to build on them and try to drive thesystem toward greater educational, financial, andintellectual economies while becoming more so-phisticated in promoting reform. Many states re-quire doing that in a systemwide fashion, con-verting a variety of resources whether in highereducation, state departments of education, orschool systems. In other words, all of these ele-ments about which I’ve spoken have been contig-ured in a system that is supporting an agendathat, if not unitary, has been reduced beyond itsnatural limits of replication.

There is a challenging research agenda trans-forming what we should learn. Some of the sites(maybe all of them in contrast, because they’vehad very different experiences) represent excel-lent source materials for mounting the research,not only in terms of documenting reform inprogress, but also in trying to identify in a multi-faceted domain the fmite number of overarchingand control elements that are crucial. We reallydon’t know that number. An exceedingly impor-tant and timely contribution from research wouldexplicate within an undefined set of variables theminimum set that’s really crucial to driving re-

4

form, leaving aside the variation that one finds would suggest you do that with deliberatenessfrom system to system. and near exclusion of the ever present and well

Again, I acknowledge the very important understood, which is of little value. The impor-work you do and very much urge you to take up tant research to be done is that which 95% of usthese matters under your research agenda. But I do not already purport to understand.

What Have We Learned? What Do We Need to Know?

William H. ClunePolicy Analysis of Systemic Reform, NISE Team LeaderUniversity of Wisconsin-Madison

The research of the Policy Team of the NISEhas concentrated in three areas: implementationof systemic reform, implementation of the na-tional standards, and equity. I would like to re-port findings and key research questions in eacharea and then turn to what we are learning aboutthe links between systemic reform and studentoutcomes and the implications for policy and re-search.

selective curriculum replacement? By curricuhmreplacement I mean the implemented curriculum,as actually taught in schools.

Findings on the National Standards and LocalImplementation

The Knapp Synthesis of Research on Imple-mentation of Systemic Reform

large “grain size,” unguided local discretiontension between problem solving and formalanalysisconfusion over “deep understanding” and“habits of mind”

. widespread planning

. incremental change in the classroom

. absence of broad and deep systemic change

. key knowledge gap: strategies leading towholesale v. selective curriculum replace-ment

On the first topic of implementation, we

political controversy over content and conse-quenceskey knowledge gap: alternative packages ofcurricula, materials, assessments, develop-ment, technology thatfit the standards and alternative educationalgoals and philosophiesdraw on knowledge in NSF Systemic Initia-tives, TIMSS

commissioned a paper by Michael Knapp that heentitled “Between Systemic Reforms and theMathematics and Science Classroom.” This pa-per synthesized research not only on NSF’s Sys-temic Initiatives but other systemic reforms suchas in Michigan and California. The paper foundthat systemic reform had stimulated widespreadplanning and discussion, in other words had be-come a major presence on the policy landscape,but had produced only incremental change in theclassroom, that is, an absence of broad and deepor systemic change. The key research question is,Which strategies lead to wholesale as opposed to

The fmdings on local implementation, orimplementability, of the national standards werecomplementary. The standards are very generaland comprehensive, which allows localities andtextbook writers to claim compliance withoutmaking much real change. There is a tension be-tween constructivists who emphasize problemsolving and traditionalists who prefer formalanalysis. There is real confusion about how todefine deep understanding of the subject mattersand so-called habits of mind, as well as politicalcontroversy over content and consequences, such

as for the traditional group of college-bound stu-dents. A key knowledge gap is the availability ofpackages of curricula, assessments, teacher de-velopment materials, and technology (such asinteractive learning) that fit the standards, as wellas alternative educational goals and philosophies,including both college prep and vocational goals.In other words, it looks like some of the disputescannot be resolved by consensus but instead willrequire the design of highly developed curricularoptions. Relevant knowledge of this kind existswithin NSF and many Systemic Initiatives buthas not been pulled together adequately and con-nected with systemic reform.

The Case of the Virginia Standards and theImportance of Nonpoliticized Review

. quality should be recognized

. without blocking legitimate options

. without ignoring implementation in symbolicpolitics

It is important that we do serious quality re-views of the entire chain of developing andimplementing standards, as illustrated by thepolitics now surrounding the Virginia standards.If these standards are indeed of high quality andare appropriate for certain educational goals andphilosophies, that conclusion should be firmlyestablished and widely publicized by indepen-dent researchers. On the other hand, if alternativemodels are needed, those, too, should receive ap-propriate support. And there is an important issuebeyond disputes over competing goals and phi-losophies, because politicization can present dan-gers for any kind of standards-based reforrn If aparticular set of standards is not appropriate forall educational goals and philosophies, a political“victory” for one group may be costly to manystudents; and symbolic politics over standardsmay actually prevent us from even looking at anyreal development of the implemented curriculumand teacher capacity.

We should not have expected that the searchfor “universal high standards” or “hard stuff for

all kids” would be easy, but neither should weleave this central question to politics. Perhapsinternational comparisons, such as TIMSS, canshed some light on options for implementing uni-versally high standards.

Findings on Equity

. ethnic/gender subgroups differ in outcomesn mixed evidence on equalization through

course takingl controversy over equity and academic

coursesn controversy over how to measure the gap and

expand access. controversy over indicators as gatekeepersn key research question: high standards,

greater access

We had two papers on educational outcomesin mathematics and science by race and gender,written by Bill Tate and Albert0 Rodriguez.These updated and refined similar work done inthe past. They found many substantial differ-ences across subgroups of race, ethnicity, andgender and mixed evidence for the propositionthat similar course taking will produce similarresults. Equivalent courses narrow the achieve-ment gap but don’t eliminate it and are not avail-able to many students. Clearly, the effort to mea-sure and report on gaps in equity yields valuableinformation.

But we also encountered serious debatesabout equity indicators. One is whether weshould stick with college preparatory courses, orwhether less academic courses are also neededfor more equity. This debate appears to be occur-ring in the field through a conflict of contendingforces, rather than through rational design andoptions. A related debate is whether the studentassessments used to measure achievement gapsare the correct measure of equity and especiallythe correct criteria for selective admissions.Thus, the key research question is how to simul-taneously raise standards and broaden educa-tional access.

6

The Need for Theory Linking Process andOutcomes (what we need to know from re-search on systemic reform)

. outcomes are changed by process

. value of theory shown in example of coursetaking and achievement

Across all three areas, we need a bettertheory linking the process of systemic reform, orpolicies, with outcomes. Theories that link pro-cess and outcomes in education, or any area ofsocial science, are valuable and rare. It took along time, for example, to establish the connec-tion between course taking and achievement tothe point that we now consider course taking anintermediate outcome, as much as a process vari-able.

Key Process Components of Successful Sys-temic Reform

A change strategy controlling

= local politics, resources, and administration, content and pedagogy. school improvement= public acceptance= longitudinal data on curriculum and achieve-

ment

If we look across sites attempting systemicreform, the more successful sites have a changestrategy involving control over (and through) thepower structure, curriculum and teacher develop-ment, school improvement, public acceptance,and data on incremental changes in curriculumand achievement.

Critical Flaws in Real Change Strategies

= political, ideological, racial conflict= political vision without pedagogical delivery. school improvement without scaling up. external accountability without internal com-

mitmentn public rejection of professional agendas. lack of resources, limits of volunteerism

. absence of planned sequence of classroomchange

On the other hand, unsuccessful sites haveone or more of a familiar list of flaws: lack ofpolitical integration and will, political visionwithout a pedagogical delivery structure, schoolimprovement or teacher enhancement projectswith no realistic strategy for scaling up, externalaccountability or technical assistance without in-ternal buy-in, unmanageable public controversy,perhaps over a previously low-profile profes-sional agenda, lack of resources and the limits ofvolunteerism, and the absence of a planned strat-egy for incremental change in schools and class-rooms.

Implications for Research and Systemic Initia-tives

. clarifying the profiles of successful process,e.g., policy plus delivery, networks plus scal-ing up

. setting initial conditions and incentives

. matching resources to vision: no “moonshotson a shoestring”

. cost-effectiveness of partially successful sys-temic reforms?

. research question: enough real success forthe cost?

. skepticism plus sensitivity to small variations

If the conditions for successful systemic re-form are really much more stringent than previ-ously suspected, what are the implications forany policy seeking to expand the scope of suchinitiatives? First, the profile of success wouldneed to be clarified, so that sites could decidewhether the necessary elements were in place.Realistically, there probably would be severalprofiles, such as policy plus delivery and net-works plus scaling up.

These profiles would have to be translatedinto initial conditions for participation, but if wewant anyone to accept the conditions, the incen-tives for participation should match the intensityof the commitment. If we expect that a full-scalesystemic reform will be built from scratch, seri-

7

ous attention should be given to the resourcesthat could support this kind of effort, instead ofexpecting a “moonshot on a shoestring.”

Perhaps the most interesting bottom-linepolicy question is the cost-effectiveness of par-tially successful systemic reforms. Suppose thatnothing like full-scale systemic change will beproduced, in the sense of wholesale replacementof the curriculum and substantial gains inachievement for all children. But also supposesignificant progress in both process and out-comes, namely, that a system has been built thatis capable of producing continuous improvementin curriculum and instruction with assurances ofsocially valuable change in curriculum andachievement. Then suppose that the initial invest-ment is not very large, as is usually the case inSystemic Initiatives.

The situation represents failure of systemicreform in the large sense, but success of asystemic-type reform as judged bycost-effectiveness. The key research questionwould be when there are enough pieces of a com-plete reform to produce some significant results

at a relatively modest cost, in other words, whena reform crosses some as yet undeterminedthreshold of cost-effectiveness.

Nothing is more common in policy than am-bitious rebuilding projects being scaled down tomodest home improvements (except perhaps forcomplete failure), and we should not lose sight ofthe broader objectives, because they do appearfeasible in some places. On the other hand, Iknow of nothing in policy that should prevent usfrom accepting cost-effective policies, especiallyin this vital area of social policy. Some commen-tators suggest that systemic change is alwayslong-term, and progress is recognizable from theconstant vector of change over time and acrossinstitutional locations, rather than from anyrapid, wholesale transformation. At base, it isthe old question of the half-full or half-emptyglass, with the key questions being just howhalf-full and what is the cost of the drink. An-swering those questions will require a kind ofresearch that is appropriately skeptical of opti-mistic claims yet sensitive to the importance ofsmall, but important, variations in actual perfor-mance.

Susan FuhrmanDirector, Consortium for Policy Research in EducationUniversity of Pennsyivania

My remarks focus primarily on the processof standards-based systemic reform. I trust thatother speakers will address the accumulating evi-dence on the effects of these reforms. For ex-ample, we know that many teachers are at leastaware of new policy directives and are favorablydisposed toward them. In some schools and dis-tricts, important changes in teacher practice, suchas increased use of real books and stories insteadof basal texts and more hands-on activities in sci-ence, have been noted. CPRE researchers sawpositive effects on 4,800 students in transitionhigh school mathematics courses linked to ambi-tious NCTM-like standards. Students inCalifornia’s Math A courses and New York’sStretch Regents courses were much more likelythan general-track students to complete a mini-mal college-preparatory sequence by the end of

high school. And students in transitional math-ematics classes posted better achievement testscores than students in general mathematics (al-though they still lagged behind students in col-lege-preparatory mathematics classes). We arealso finding that professional development di-rectly linked to the course content has contrib-uted to changes in practice and improved studentachievement.

My choice to focus on the process of enact-ing and implementing standards-based reformsreflects my background in political science, but itis also appropriate to the current state of reformdevelopment. The reforms are not yet in place inmost states and districts; they are still under de-velopment. Their story is still being written. Fur-ther, many of the most important challenges fac-ing reformers, as well as lessons about the

progress of these reforms, concern the difficultiesassociated with the reform process and politics.

I will focus on seven lessons about theprogress of standards-based reform. I draw pri-marily on CPRE research in nine states (CA, CT,FL, GA, KY, MN, NJ, SC, TX) and 25 districtsbetween 1990 and 1995 (Massell, Kirst, &Hoppe, 1997).

1. Standards-based reforms continue to makeprogress, despite changes in leadership and po-litical turbulence. Although there was oppositionto the reforms in each of our states, particularlyin the 1994-95 period, they were not dismantled.Despite the vocal, and often virulent, objectionsof religious conservatives who asserted that stan-dards interfered with the prerogatives of familiesand of antigovernmental forces who saw stan-dards as infringements on the authority of localschools, standards development continued. Inwell-established American tradition, new re-forms, such as charter schools, were developed torespond to those who advocated devolution, andthey were simply added to the books while stan-dards reforms continued to be developed andimplemented. Political rhetoric focused on thenewer reforms, while under the surface, andmore quietly than when they were originallychampioned, standards documents, new assess-ments. and related policies were promulgated andthe slow process of classroom implementationbegan. Standards policies were modified in re-sponse to opposition, as noted below, but the ideaof standards-based reform continued to be robust.

2. Much of the continuing momentum behindstandards-based reforms can be traced to the ac-tivity of nongovernmental forces. Professionalassociations, networks, and collaborations, somenational and some state-specific, were importantsources of support and expertise. For example,standards developers at both state and districtlevels drew on national documents and examplesfrom other states. Sometimes nongovernmentalorganizations provided revenues. This is the casein eight districts undertaking standards-based re-forms with the help of the Pew Charitable Trusts.Certainly, the associations and partnerships en-hanced the legitimacy of standards-based re-forms. Most national or regional gatherings ofeducation policymakers became occasions to

highlight the record of states and districts consid-ered “in the lead” in these reforms and to drawimplications from their experiences. These meet-ings, and the diversity of groups-from unions tobusiness leaders-supporting standards-basedreforms, were very reinforcing.

3. The federal government was an importantsource of support. Goals 2000 funds were usefulin many states, particularly for subsidizing pro-fessional development. The standards frameworkof the Improving America’s Schools Act ap-peared to lend greater legitimacy to state efforts,but it was just taking shape when we were last inthe field. We have plans to examine its influenceover time. In the context of this Forum, the criti-cal contributions of the National ScienceFoundation’s Statewide Systemic Initiatives mustbe noted. The SSIs in seven of our nine states(CA, CT, FL, GA, KY, NJ and TX) were respon-sible for developing the mathematics and sciencestandards. And the SSI can sometimes be cred-ited with giving standards reforms an essentialboost. For example, Georgia’s subject-matter re-visions in most areas had been stalled for years,but thanks to the SSI, work in mathematics andscience progressed.

4. That support for standards reforms camefrom many sources was essential for its politicalsurvival, but the very diversity of supportersmade achieving coherent direction for educationmore difficult. Standards were supposed to rep-resent agreement on what students should knowand be able to do, but the many sources of stan-dards-national associations, local and state de-velopment committees, new specifications devel-oped by test publishers-frequently varied in in-structional vision. Policymakers and educatorsare drawing on multiple sources in developingtheir own versions of standards, and an importanttopic for future research is the coherence of theresulting products. For example, contrary to thefears of conservative critics, local educators werenot circumscribed by state standards. In mostcases, they used state frameworks as only onesource of their own standards and found statestandards too vague to be really useful. They de-veloped their own frameworks, using many ex-amples and models.

9

5. An additional reason to worry about thecoherence of emerging standards documents isthe trend toward adding “‘the basics” back intostate standards. Part of the political and publicopposition to standards centered on the new-fangled notions of learning and pedagogy incor-porated in early standards documents. The newapproaches seemed to threaten traditional skillsand common-sense notions of what school wasabout-reading, writing, arithmetic, spelling, etc.To respond, many states sought more “balance”in their frameworks, for example, by incorporat-ing phonics into language arts standards that pre-viously had a whole language orientation. Simi-larly, emerging state assessments were modifiedto add multiple-choice items back in and to in-clude norm-referenced as well as criterion-refer-enced components. Will “balance” mean morethoughtful integrations of approaches or simpleaggregations, with traditional practices on onepage and new ideas on another? Future researchshould attend to this question.

6. Early on in the standards reform move-ment, it became clear that the work of standards-based reform is extremely demanding. The ideaof making change in virtually every aspect ofpolicy-curriculum, assessment, teacher prepara-tion, professional development-not just at oncebut in a coordinated fashion so they all linked tostandards, represented a huge challenge to a po-litical system that was expert in incremental, dis-jointed change. Over time, the overwhelming de-mands of standards-based reform have becomeeven clearer, as has the need to sequence reformcomponents to make them manageable and fea-sible. Teachers are furious when new assess-ments come on line before the standards, towhich they are supposedly tied, are developedand before any curricular materials are available.They are even angrier when such new assess-ments are incorporated in accountability systemsthat carry consequences. Yet such disconnects insequencing are not uncommon at the state anddistrict level. In the states we studied, some thattook more incremental, more step-by-step ap-proaches to reform have had less upheaval thansome of the previous reform leaders--one reasonmay be that the incremental states were able to

be more deliberate about each step and plan morecarefully about how to bring things on line.

7. The most important factor influencing theprogress of reform development and ultimatelywhether standards reforms can support meaning-ful changes in teaching and learning is the capac-ity of the system. Capacity is essential at everylevel. States must manage complex processes ofstandards development that balance public andprofessional input, choose or create assessmentsthat balance needs for adequate information forparents and teachers and the desire to modelgood instruction through challenging items; re-vise licensing systems; support schools and dis-tricts in curriculum development and profes-sional development-and this is just some ofwhat is required. Districts must do much of thesame, focusing on daily support to schools goingabout the difficult process of improving instruc-tion. Teachers and administrators must developnew approaches to teaching and learning, whilefrequently managing new site-based governancepolicies and developing ways of relating to oneanother and to parents. Parents are expected tosupport new, more challenging expectations fortheir children, understand and respond to new,very complex accountability systems, and seethat schools are supported with adequate re-sources. Everyone needs help.

There are some signs that the need forgreater capacity is recognized. For example,Connecticut, Florida, and Minnesota were amongthe states actively revising their teacher certifica-tion processes to support more challenging vi-sions of instruction. Many states have orches-trated or supported teacher networks that groupteachers by grade level or subject in order to pro-vide continuing support; some states and districtsare facilitating school access to reform designsand technical assistance.

But capacity-building efforts to date aredwarfed by the need and undercut by omissionsand contradictions. For example, in our states,state and local central agencies continue to bedownsized. At the state level, a number of agencybudgets were cut by about 25% during this pe-riod, coming on top of earlier cuts during the1980s. Little has been done to see to the quality

1 0

of professional development available to teach-ers. The prevailing mode is to push money downto the school or district (that is, where the state ismaking a new investment in professional devel-opment at all) assuming that demand for profes-sional development will be generated by newstandards and assessments and that the demandwill in turn generate supply. This turns out to betrue; suppliers do show up. But much of what isavailable is of very low quality, and few placeshave tried to develop criteria to guide selectionof professional development providers or to edu-cate consumers about good choices. And, finally,a crying need is for curriculum and materials, re-flecting the standards, that can be used for dailyinstruction. Increasingly, teachers themselves arequestioning the romantic notion that day-to-daycurricula should be developed by collaborativeschool or department-level groups. They haveneither the energy nor the desire to create fromwhole cloth when others might have invented

Jane Butler Kahle*Condit Professor of Science EducationMiami University

Although much has been written about thenature and policies of systemic reform bypolicymakers, those papers provide only a partialvision of systemic reform-one from the outsidein. Less has been written from the field, i.e., fromthose who are actively trying to promulgate re-form either in the classroom or at the state level.Even less has been written about changes inteacher practices and in student learning-yet,without those changes the reforms eventuallywill wither and fade away. The encompassingnature of systemic reform provides critical rolesfor national and state leaders, for professional

* The preparation of this paper was sponsored in partby the National Science Foundation, Grant # OSR-92500 (J. B. Kahle and K. G. Wilson, Co-PrincipalInvestigators). The opinions expressed are those of theauthor and do not necessarily reflect the position ofN S F .

very usable and good materials. We need bettermethods of sharing what is available and moreattention paid to development.

Systemic reform has proven to be both hardyand difficult. Challenges multiply, but the basiclogic has enormous appeal and staying power. Itis important to act on the lessons we are learningabout the challenges, for example, by supportingmore extensive efforts to develop and dissemi-nate excellent curricula. It is also important tokeep tracking the results. Evidence of positiveeffects will be critical to the reform’s stayingpower over time. We must show that the hardwork is worth it.

ReferenceMassell, D., Kirst, M. W,, & Hoppe, M. (1997). Per-

sistence and change: School reform in ninestates. Unpublished manuscript, University ofPennsylvania, Philadelphia, Consortium forPolicy Research in Education.

groups, and for individual teachers. Experiencewith several systemic initiatives in one state,Ohio, forms the basis for the following discus-sion of the challenges to reform and the changesneeded for success.’ My discussion focuses onthe challenges faced by one state’s efforts, onevidence of changes in teaching practice and instudent learning, and on the meanings that maybe drawn from that evidence. Specifically, Ohio’sreform was characterized by the following pa-rameters. It

9 developed a regional infrastructure to sup-port and sustain reform.

n focused on teacher professional developmentthat emphasized content, provided in-depthexperiences in inquiry and problem solving,and extended over one year.

. targeted middle schools for equity and eco-nomic reasons.

1 1

. included substantive involvement of univer-sity scientists and mathematicians.

. avoided K-12 curriculum development.

Four years into the reform, changes in par-ents’ perceptions of science, in principals’ sup-port for inquiry instruction, in teachers’ practiceof inquiry, and in students’ learning of sciencewere assessed. These findings articulated boththe challenges and changes of the systemicinitiative’s efforts.

What Are the Challenges?

The challenges describe five aspects of sys-temic reform that must be addressed and aligned.Because each one poses risks to a part of the edu-cation community, varying levels of success havebeen reached. However, the lessons learned inattempting to meet the challenges provide in-sights and directions for the future.

Challenge One: Sustained professional de-velopment of a validated model can produce aculture shift in participants, but it is costly andtime intensive. Given a state cohort of over 7,500teachers per grade level, there are neither the hu-man nor the financial resources to reach morethan a small fraction of the target audiencewithin a five-year period.2

Lesson Learned Not only are human and fi-nancial resources limiting, but the pool of teach-ers who can, or will, undertake sustained profes-sional development is limited. Teachers who maybe characterized as needing professional devel-opment the most---ones in poor schools, oneswith general licenses, ones with few courses inscience or mathematics, ones teaching out of cer-tification areas, ones who are disenchanted ordisenfranchised-do not readily volunteer for arigorous summer of mathematics or science.Rather, they must be reached in their communi-ties and schools, the academic program must beat the level at which they teach science and/ormathematics, and the materials used must be di-rectly applicable in their classrooms and withtheir students.

To meet this challenge, research validatedcurricula were identified. Next, teachers, who

had had at least one year of professional develop-ment, offered local, 40-hour workshops for theirpeers. Districts supported the teacher-instructorsand often required all science or mathematicsteachers at the targeted grade to attend. Ostensi-bly, the workshops were to help teachers learn touse standards-based curricula; in reality, muchmathematics and science was learned.

Challenge Two: Any reform has a limited andunique function. Although it must offer resourcesthat are not available during its lifetime, thoseresources eventually must be assimilated into theongoing educational system.

Lesson Learned At the beginning of the re-form, both the Systemic Initiative and the OhioDepartment of Education divided the state intoeight professional development regions. Two setsof centers were established that tested two differ-ent paradigms for professional development. TheSystemic Initiative insisted upon regional col-laboration before identifying and supporting itsregional centers, which, then, delivered thesix-week institutes. They were taught by regionalacademic leaders (both outstanding teachers andPh.D. scientists and mathematicians) who wereavailable during the school year to assist teachersin their classes, to work on curriculum teams, andto provide local workshops. The SystemicInitiative’s model was successful beyondanyone’s expectations. The Department’s lesscostly model involved short-term, usuallythree-day, workshops with limited classroomfollow-up. The teaching staff rotated, and manywere imported for a few days work. Furthermore,the selection process for the center was competi-tive, resulting in antagonism between the unitsfunded (districts, colleges, county offices) andthose passed over.

It was obvious that the two regional unitsneeded to merge, and merger has occurred. Themergers have been slow and fraught with diffi-culties; for example, how to retain the sustainedprofessional development model within themerged regional center. However, systemic re-form is about taking chances, about building con-sensus, and about moving ahead with the results.In a large complex state, regionalization isneeded, and it is better to have one unit than

12

none, and two units are not sustainable. Further-more, the merged center is fully assimilated intothe state’s educational system.

Challenge Three: Systemic Initiatives haveunderestimated the difficulty of getting mathema-ticians and scientists-at all levels-to work to-gether, the difficulty of shifting university facultyfrom teaching by lecture to inquiry, as well as thedifficulty in communicating between campusesand across disciplines. In Ohio, two separategroups of mathematicians developed mathemat-ics by inquiry courses, because that process wasmore efficient in terms of time than the collabo-ration necessary to develop one course. Later,when middle school mathematics and scienceteachers were both in institutes on the same cam-puses (often in adjacent classrooms), both groupsactively resisted working on integrated units orcomparing strategies across disciplines.

Lesson Learned It takes time and effort toencourage collaboration. Yet, without it, a reformis only pockets of change, not systemic. There-fore, the Systemic Initiative instigated collabora-tions with Ohio’s three Urban Systemic Initia-tives and the Appalachian Rural Systemic Initia-tive. Further, it identified collaborative relation-ships with Ohio’s Mathematics and Science Coa-lition, the Parent Teachers Association, and manyregional and local businesses and foundations.

Challenge Four: Although Ohio’s SystemicInitiative focused on individual (or groups of)teachers, a school is a more viable unit ofchange. Teachers need a support system for thereforms they are initiating in their classrooms. Aschool focus also produces the visibility to attractexternal market-driven resources that may con-tinue the reform after the funding period.Changes in a school’s science program or sciencedepartment-with documented improvement inresults-is a phenomenon that may be quicklycommunicated to parents and policymakers.

Lesson Learned As part of the assessment,described below, brief site visits were made to 12schools, primarily in poor urban or rural areas.Both the observational and questionnaire datagathered suggested that there were greaterchanges in learning environments, in teaching

practices, and in student outcomes in schools thathad a critical mass of reform teachers (up to one-third of the mathematics/science faculty), com-pared to schools with only a few teachers, or anisolated teacher with the sustained professionaldevelopment. A supportive group of teachers isespecially important because of the high mobilityof principals.3

Systemic or standards-based reform requiresa critical and self-sustaining mass of teachers in aschool. The solution was twofold: first, districtswere requested to require or provide incentivesso that all appropriate teachers would be in-volved in local professional development; and,second, the intensity and depth of the profes-sional experiences were moderated with cautionand some trepidation. (See Challenge One.)

Challenge Five: All initiatives that are sys-temic in nature have important research/develop-ment and dissemination/support roles.Well-researched and validated professional de-velopment packages, such as Physics by Inquiry(McDermott, Shafer, & Rosenquist, 1996) havesustainability independent of the instructors.Such packages can be assimilated very quicklyinto existing delivery systems, such as a state’sregional centers. Where such packages do notexist (or lack research validation), the SystemicInitiative must take the professional developmentpackages through carefully controlled field testsand refinement activities to document their valueand sustainability through research studies.

Lesson Learned There is neither time normoney to do it all. Ohio has learned two lessons.First, find and use the expertise of others, and,second, assess progress and outcomes in order torefine and improve your efforts. As mentionedearlier, in the fourth year of the reform, the Sys-temic Initiative undertook a major study to de-scribe the landscape of science and mathematicseducation in the state. The intent of the assess-ment was to tell the reform story in terms ofchanges in learning environments, in teachingpractices, and in student learning. Because of thefocus on equity, the schools selected to gatherstudent achievement data were in poor urban orpoor rural districts.

1 3

What Are the Outcomes?

Four years into the reform, a comprehensiveassessment of learning environments; teachingpractice; principal, teacher, and student attitudesas well as student learning was undertaken. Thestudy consisted of two levels and involved thecollection of both quantitative and qualitativedata. Level one consisted of a random sample of126 schools, drawn from all the schools in thestate that had at least one teacher who had com-pleted the initiative’s professional training. Atlevel one, principals and all teachers who taughteither science or mathematics (grades 5 through9) completed questionnaires concerning class-room instruction, administrative support, andparent influence as well as issues of schoolchange. Level two consisted of brief site visits to12 (from the original random sample of 126) se-lected schools. In those schools, students andparents also completed questionnaires, studentscompleted achievement tests, and principals,teachers, and randomly selected students and par-ents were interviewed. At each site visit school, ateacher who had had the sustained professionaldevelopment (reform teacher) was matched witha teacher who had not had that experience(nonreform teacher). In addition, a randomly se-lected class of the reform teacher was matchedwith a comparable class of the matchednonreform teacher. Changes were identified bycomparing the responses of these matched groupsof teachers and students.

Using 1990 and 1992 public release itemsfrom the National Assessment of EducationalProgress (NAEP), science and mathematicsachievement measures were developed by teamsof faculty, teachers, and regional leaders. Testitems focused on process, not product, becausethe reform’s goals were to increase conceptualunderstanding as well as skills needed to inter-pret and use scientific and mathematical informa-tion. In Miller’s (1996) discussion of barriers tosystemic reform, he notes that there is often adisconnect between the practical paradigm of re-form (focused on process) and the technical para-digm of education (focused on product). This dis-connect was avoided by developing new achieve-ment measures. When possible the questionnaires

for principals, teachers, students, and parentscontained the same questions, phrased appropri-ately. That strategy allowed comparison of re-sponses across groups. For example, did bothteachers and students respond that manipulativeswere used at least once a week? The results indi-cate that students responded similarly to theirteachers concerning instruction in reform andnonreform classes. That is, students in reformclasses significantly more often talked with eachother about the subject, had to support theirclaims, and were encouraged to ask questions.Interestingly, significantly more students andteachers in reform classes reported that theirprincipal had learned to accept classroom noise.

When the ways in which students learn wereexamined, interesting and significant differenceswere found between reform and nonreformclasses. Students in classes taught by reformteachers significantly more often wrote abouthow they solve problems, solved problems insmall groups, and used hands-on manipulatives.Those strategies are recommended both by theNational Science Education Standards (NSE$National Research Council, 1996) and by the re-search literature concerning strategies to improvethe participation, attitudes, and achievement lev-els of girls and minority students. One of thesix-week content courses incorporated the com-puter as a learning tool; another heavily usedgraphing calculators. Further, both the NCTMstandards and NSES argue for the incorporationof technology into science and mathematics les-sons (National Council of Teachers of Mathemat-ics, 1989; NRC, 1996). However, neither practicewith, nor information about, the efficacy of tech-nology in promoting learning affected the use ofcalculators or computers. The lack of appropriateequipment and software remains a major chal-lenge to implementing the reform agenda.

Briefly, there were significant differences inscience achievement in favor of reform classes,as measured on the Discovery Inquiry Tests. It isimportant to note that it was a low-stakes test(grades were not affected) and that it focused ona student’s ability to interpret information and onconceptual, not factual, understanding. Becauseof the reform’s focus on equity, the results alsowere examined for any gender, race, and/or

14

group (reform versus nonreform) differences.Those analyses revealed interesting patterns ofachievement; for example, minority students (inthis case, African American) in reform classesscored significantly higher than their peers innonreform classes. In fact, African American sev-enth and eighth graders in reform classes scoredas well as white students in nonreform classes.Because the data were collected from reform andnonreform students taking the same type of class(e.g., general science, introductory life science)with a “matched” teacher in the same school,economic differences were not a major factor.

Further, in science classes taught by teachersinvolved in the systemic reform, both AfricanAmerican and white females scored higher thanthe males in their racial group. In reform classes,white females, shown by other research to be thegroup most socialized away from science(Campbell, 1991; Campbell & Connoly, 1987;Kahle & Damnjanovic, 1996), scored higher thanwhite males or African American females ormales. When science test scores were subdividedinto physical science and life science items, fe-males in classes taught by the reform teachersscored higher than males on the physical scienceitems. These fmdings contrast with those of otherstudies. Using a large national data set (NELS:88), gender differences have been found inachievement in physical science, but not in biol-ogy (Burkham, Lee, & Smerdon, 1995). Theseresults may be the first time that a genderachievement gap in physical science, favoringgirls, has been reported. Clearly, the type ofteaching observed and recorded in responses tostudent and teacher questionnaires-more use ofmanipulatives, more time to talk about science,more opportunities to write about science, in-creased use of cooperative learning groups-hasaffected achievement, particularly for studentswho have been underrepresented in science.

A Further Challenge

Once the data were collected and analyzed,the challenge was to distribute the findingswidely in an accessible way. Over 10,000 copiesof a small, easy-to-read publication, the PocketPanorama (Kahle & Rogg, 1996), have been dis-

tributed across the state and nation. Becausemost state departments of education do not haveeither the time or the expertise to performlarge-scale research and dissemination activities,documentation, validation, and dissemination ofchange and of best practice provide unique andimportant roles for Systemic Initiatives. Such ac-tivities are critical for the public’s understandingand acceptance of systemic reform. Indeed, re-search and dissemination may be the key rolesfor externally funded reform initiatives within astate. The final lesson learned is that research orassessment without dissemination benefits onlythose who are already involved in the reform.Dissemination of findings in practical andeasy-to-use ways informs others of theinitiative’s success and invites them to becomeactive participants in it. Further, assessment,coupled with wide-spread dissemination, pro-vides the basis for successful reform strategies tobecome sustained through the existing educa-tional system.

All parts of a reform must be addressed andwork together if the results are to be systemic.The challenges in one state led to alterations inits reform strategies; those alterations, in turn,led to wider participation and acceptance of thereform. Although the changes described in theassessment cannot be directly attributed to theprofessional development and support strategiesthat were part of the reform, the findings suggestthat improved learning is associated with im-proved practice that is initiated through sustainedprofessional development.

Notes1. Ohio was one of the first ten states to receive Na-

tional Science Foundation funding for a StatewideSystemic Initiative. Further, Ohio’s three cities thatwere eligible for Urban Systemic Initiative funds,Cincinnati, Cleveland, and Columbus, have beenawarded grants. Ohio also has five counties in theAppalachian Rural Systemic Initiative.

2. NSF Systemic Initiative awards are for up to fiveyears, although contracts are renewed annually.Three of the first cohort of states were terminatedduring the five-year cycle.

3. In Ohio, over 50% of principals are in that positionin a particular school for four years or less.

15

ReferencesBurkham, D. T., Lee, V. E., & Smerdon, B. A. (1995).

Gender and science learning early in high school:Subject mat ter and laboratory experiences . Unpub-lished paper, University of Michigan at Ann Arbor.

Campbell, J. R. (199 1). The roots of gender inequityin technical areas. Journal of Research in ScienceTeaching, 28,25 l-264.

Campbell, J. R., & Comroly, C. (1987). Decipheringthe effects of socialization. Journal of EducationalEquity and Leadership, 7(3), 208-222 .

Kahle, J. B., & Rogg, S. R. (1996). Apocketpan-orama of the landscape study, 1995. Oxford, OH:Miami University.

Kahle, J. B., & Damnjanovic, A. (1996, April). Esjrectsof inquiry teaching on the achievement levels of

urban middle school science s tudents by race andsex. Paper presented at the annual meeting of theAmerican Association for the Advancement of Sci-ence, Baltimore, MD.

McDermott, L. C., Shafer, P. S., & Rosenquist, M. L.(1996). Physics by inquiry. New York: Wiley.

Miller, K. W. (1996). Paradigmatic school philoso-phies as barriers to school reform. Science Educa-tor 5(l), 1-6.

National Council of Teachers of Mathematics. (1989).Curriculum and evaluat ion s tandards for schoolmathematics. Reston, VA: Author.

National Research Council. (1996). NationaZ scienceeducation standards. Washington, DC: NationalAcademy Press.

Uri TreismanProfessor of MathematicsDirector; Charles A. Dana Center for Mathematics and Science Education and the Texas SSIUniversity of Texas at Austin

My purpose in these remarks is to commentnot only on what we, the Dana Center/SSI, havelearned, but also on what we are trying to learnabout systemic reform as it is practiced in Texasand other large states. I have organized my re-marks around the first four drivers of systemicreform as enshrined in the NSF’s assessment andmonitoring system for its systemic initiatives. Inmy view, these drivers do in fact capture the es-sential dimensions of the Dana Center/SSI’swork and provide quite a useful framework fordescribing the complex of interlocking projectsand initiatives that we have found necessary formoving the system forward in ways that respectour fundamental commitment to equity.

I. Policy

The extent to which state or local educationalpolicies affect practice is determined by the na-ture and sharpness of the teeth associated withthem in the relevant accountability system. Inthis sense, managing the policy side of systemicreform can be thought of as a kind of educationalorthodontics-requiring lots of steady pressure atjust the right place constantly applied. In the

work in question, generating and maintaining therequired steady pressure requires the develop-ment and maintenance of relationships with indi-viduals and groups whose interest in mathematicsand science is quite minor. But more on this later,in my discussion of public engagement.

It is my observation that in many states thepolicy focus of the mathematics and science edu-cation communities has been almost exclusivelyon shaping particular policy documents, i.e., onsetting the content of state curriculum frame-works. Too little attention has been paid to themechanisms through which these documentshave their influence-an essential issue in sys-temic reform.

In Texas, for example, the processes for suc-cessfully developing state curriculum f&me-works and for getting them adopted by a conser-vative State Board of Education were remarkablysimilar for mathematics and science. Unusualand sometimes unnatural coalitions had to bebuilt; a broad perspective had to be inculcated inindividuals who are both skilled in and fond offighting over arcane and often politically mean-ingless issues. Incentives had to be created forvarious leaders to support positions that differed

16

in minor ways from those of the professional or-ganizations that give them authority and specialinfluence. Most challenging, of course, was pro-ductively engaging the opponents of standards-based reform, who often argued vociferouslyabout the fine details of the proposed curriculumframeworks but did so in ways that were actuallyintended to shake confidence in public education.

Yet, these now-adopted mathematics and sci-ence frameworks have very different effects onpractice at the school and district levels. By law,each constitutes the legal base of any state ex-aminations in its subject area. But, herein lies thecrux. While mathematics is tested at many gradelevels, science is tested only at grade eight.Moreover, state ratings of school performance,which are widely published and attended to by-among others-the real estate industry, are influ-enced heavily by mathematics scores but not atall by science scores. The effect is profound.School administrators spend heavily to help theirstudents learn mathematics; they spend no morethan is absolutely necessary on science unless itis a special interest of theirs, their spouse’s, or ofan all-too-rare group of their teachers who areboth science people and effective negotiators.

This unpleasant reality has consequences forthe kinds of networks one builds, the state-levelprofessional development one offers, and so on.In mathematics, one can count on felt need todrive action. In science, one dependson argu-ments about the general good, the future, and soon-arguments that, in the absence of a clearthreat to the nation’s security, are very, very hardto make.

Whereas, in mathematics, the Texas SSI cansupport (and find support for) the implementa-tion on a large scale of high quality curriculasuch as those developed with NSF support, inscience, the primary challenge is to organize po-litical support for the inclusion of science in theaccountability system. In practice, one confrontsa Catch-22 situation. The legislature will onlymove to include science in the accountabilitysystem if it believes that the school system canmeet the challenges it will generate. Few legisla-tors want their constituents’ schools to face yetanother challenge they cannot overcome. Thesesame legislators will only support increases in

funding for science if they feel an overwhelmingdemand from their constituents. This demand isnot there because it is not generated by the ac-countability system. And so on.

In such an environment, the natural but verylong-term strategy is capacity building-creatingand supporting the kinds of statewide profes-sional development networks, collaboratives,etc., that will generate reasonable confidence thathigher standards can indeed be met. Until then,science standards will serve as a banner and agoal. Mathematics standards will be drivers oflocal action.

Another and even more important questioncenters on equity and the allocation of scarce re-sources in the educational system. It is widelyassumed in our community-indeed it is themantra of systemic reform-that standards-basededucational policy is a friend of, if not a prereq-uisite for, equity. Roughly, the most common ar-gument for this position is that explicit standardsdemystifl the system by making public exactlywhat children must know and be able to do ineach curricular domain. In being public, stan-dards enable the transformation of a system puta-tively based on ability to one that is manifestlybased on effort. The needed catalyst (which, ofcourse, is rarely present) is adequate resources sothat the playing field is approximately level forall students, no matter their family’s financial oreducational resources.

In Texas there is a particular and, in my view,all-important feature of the state’s educationalaccountability system that may turn out to be anecessary and sufficient condition for greater eq-uity to be an outcome of standards-based reform.Specifically, in Texas, schools and districts aregiven one of four ratings ranging from “low per-forming” to “exemplary,” based mostly on stu-dent performance on state examinations (TexasAssessment of Academic Skills) whose content isdirectly defined by the state curriculum fiame-works. School data are reported for all studentsand separately for four subgroups: AfricanAmerican, Hispanic, White, and EconomicallyDisadvantaged students. To achieve a given rat-ing, the scores of every subgroup must exceed aparticular cut-off score, which rises over time ona predetermined schedule. Sanctions for low per-

1 7

formance include public hearings and possibletakeover by the state. In short, school and districtratings depend on the success of everybody’schildren.

The effects of this rating system in Texashave been extraordinary and are increasinglywell known. Ten years ago, Texas student perfor-mance mirrored that of the South-it was abys-mal. Today, the average scale scores of each ofthe above mentioned populations on NAEP math-ematics tests are near or at the top of nationalrankings. Indeed, on NAEP Grade 4 Mathematicsscores, Texas is tied for sixth place-onlyslightly behind Maine, Minnesota, Connecticut,Wisconsin, and North Dakota. Texas fourth-grade scores now exceed those of such tradition-ally high performing states as New Jersey, Mon-tana, Michigan, Colorado, Vermont, and Utah.Amazing. The mechanism of action is clear andeasily visible from the ground floor where Ispend most of my days. Today, building adminis-trators in Texas are far more likely than in thepast to direct their resources to supporting highachievement among groups of students who theybelieve might endanger their school accountabil-ity ratings. The focus on mathematics (still thedominant reason for low school ratings) startlesvisitors from out of state. Indeed, this feature ofthe Texas accountability system, in concert withthe recent changes in the Improving America’sSchools Act, has set a new benchmark for thespeed at which a large state’s performance datacan change.

In systems in which school ratings depend onaverage (nondisaggregated) scores, the naturalstrategy for a building administrator would likelybe to direct resources to those children whom heor she believes can be most easily educated. Thisstrategy would keep the performance gap amongethnic groups large and might increase it. Indeed,Texas is one of the few states where differencesin performance in mathematics among ethnicgroups is decreasing-at least at the K-8 levels,where the accountability system is strongest.

In short, there is no evidence that standards-based accountability by itself is a friend of eq-uity. There must be significant incentives for ad-dressing equity, but unless there are explicit andsubstantive consequences for not educating to

high levels everybody’s children, unless the partsof the accountability system that deal with equityhave teeth, it will continue to be minorities whoare underserved by the educational system.

II. Curriculum and Instruction

Perhaps the greatest surprise to emerge fromour work in the Texas SSI has come from our re-search on effective school responses to our high-stakes standards-based accountability system. Wehave now studied quite carefully 26 high pov-erty/high minority elementary schools whose stu-dents score well above the state average at everygrade level on state examinations in both math-ematics and reading. These are schools whoseperformance profiles are similar to those of typi-cal schools in the wealthiest suburbs of Texas.We have also studied the 13 Texas high schoolsin Title I feeder patterns, i.e., in high povertycommunities, with the highest average scores onthe state’s End-of-Course Algebra examination.

What did we find? First, we found enormousvariations both in how schools organized them-selves to succeed and in the particular curriculumand instructional approaches they adopted. Yes,some of the high-performing high schools useSaxon’s Algebra. Others use books that wouldmake any NCTM leader happy. At the elemen-tary level, we found as many examples of schoolsthat espoused direct instruction/explicit phonics-based approaches as we did schools that identi-fied themselves with “constructivist” approaches.The eclectic pragmatism of these schools im-pressed us. The teachers were clearly more com-mitted to nurturing students than to climbing onbandwagons.

It is, of course, one thing to espouse or toidentify with a philosophy and quite another toactually practi