DOCUMENT RESUME

ED 373 104 TM 021 999

AUTHOR Paulson, LeonTITLE Bridging the Gap: Applying Deming's "Quality by

Design" to Portfolios.PUB DATE Apr 93NOTE 12p.; Paper presented at the Portfolio Conference in

the Heartland (Des Moines, IA, April 1993).PUB TYPE Viewpoints (Opinion/Position Papers, Essays, etc.)

(120) Reports Evaluative/Feasibility (142)Speeches/Conference Papers (150)

MRS PRICE MF01/PC01 Plus Postage.DESCRIPTORS *Administration; Beliefs; Constructivism (Learning);

*Educational Assessment; Elementary Education;*Evaluation Methods; Formative Evaluation;Inspection; Learning Theories; Operations Research;*Portfolios (Background Materials); ProgramEffectiveness; Program Evaluation; *QualityControl

IDENTIFIERS *Deming (W Edwards); Deming Management Method

ABSTRACTThe gap between learning and assessment is apparent

both in the beliefs of children, who often see no connection betweenthe two, and in the practices of teachers, who do not choose to usethe most recently developed means of assessment. This gap stems frommanagement theory and from learning theory. Traditional management inthe United States has separated production and inspection, andtraditional learning theory has seen learning as passive. Portfoliospresent the opportunity to address the gap between assessment andlearning if educators recognize learning as an active process and ifthey apply the principles of management of W. Edwards Deming to theirconstruction and evaluation. The key to the Deming approach is inprocess-based evaluation: it builds quality into the process and doesnot impose it from the outside. Examples of elementary schoolstudents' portfolio building illustrate the use of portfolios withemotionally disturbed and learning-disabled students. Portfoliosoffer an opportunity for assessment that empowers learning, ratherthan assessment that polices learners. (Contains 17 references.)(SLD)

))7 *********'*******************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

*******************************************************************

U.S. DEPARTMENT OF EDUCATION(Ace ol Educational Research and Improvement

ED RONAL RESOURCES INFORMATIONCENTER (ERIC)

IP'Thrs document has been reproduced aswowed Item the parson or organizationoriginating it.

0 Minor changes have been made to improvereproduction quality

Points ol view or Opinions statedin this docu

ment 60 nol necessarily represent officialOERI positron Or policy

"PERMISSION TO REPRODUCE THISMATERIAL HAS BEFN ORANTED BY

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

Bridging the Gap: Applying Deming's

'Qualily by Design' to Portfolios

Leon PaulsonMultnomah ESD, Portland OR

My work with portfolios reflects acontinuing interest in making assessment information useful in the class-

room. We give many tests, but I see little evi-dence that teachers actually use the results.People in testing have a ready explanation. Theypoint out that few teachers receive formal train-ing on tests and measurement during teacherpreparation. The implication is that they woulduse test results if they knew more about them.

Let me tell you a story. Several years ago Iwas asked by a local district to develop a seriesof criterion referenced tests tied to the district'scurriculum. The district selected a team ofteachers and gave them release time to workwith me on the project. We worked together formaybe three years during which time : trainedthem in test development including developingblueprints, writing and editing items, conductingfield tests, interpreting statistics, and designingcustom reporting forms.

I felt the project had accomplished twothings. First, the district had locally designedtests tied to the curriculum which were reportedon teacherfriendly forms returned to the class-room in less than one week, all with buyin fromteachers. Second, the district had a trained cadreof sophisticated test developers who knew andunderstood testing.

About a year later the district hired a newsuperintendent who made giving the CRTsoptional. I was not surprised that teachers optednot to give the tests. What did surprise me wasthat all of the teachers I had so carefully traineddecided not to give the tests. I made some phonecalls and was told that the only reason they hadwritten the tests was because the district askedthem to. As teachers, they had no interest in theresults. There was a message in all this. Theirindifference could not be explained away bysaying they lacked training in tests and mea-sures. They were skillful test developers, yet

Keynote addrcss delivered at the "Portfolio Conference in the Heartland" sponsored by the Heartland AEA and NCREL,

Des Moines 1A, April 1993.

2BEST COPY AVAILABLE

Paulson: Bridging the Gap

they couldn't have cared less about the results.Clearly there was a gap between instruction andassessment, a gap which had little to do withteacher training.

But actually, experiences like this one haveled me to rethink many of the assumptionsunderlying assessment that go far deeper than theformat of the test. My theme today is that youcan't separate learning from assessment. Theyoccur simultaneously. In fact, I believe that themost effective instruction occurs when thelearner and the assessor become the same per-son. If we want students to take charge of theirlearning, they must also take charge of theirassessment. One way to do this is through thestudent portfolios.

About a year ago I met with a group of stu-dents in a program for the emotionally disabled.My task was to interest them in doing portfolios.I wasn't 100% successful. One listener said,

I hate portfolios because they're too much of ahassle.

But one found the portfolio a meaningfulpersonal experience.

I like portfolios because you put stuff in them and itmakes room in your desk.

Even some college students would agree withthe student who said portfolios are too much of ahassle. Lewis and Clark College accepts admis-sion portfolios as an alternative to their tradi-tional application procedures but only a smallpercentage. of applicants choose the portfoliooption. Their admissions director told me thatstudents find it easier to submit a transcript andSAT scores.

So why do portfolios at all if they are more ofa hassle than traditional methods? To answerthis, consider why we assess at all. Assessmentplays a role in education similar to the roleinspection plays in industry. We assess to ensurethat we are providing quality education whichmaintains high standards. Quality by Assess-ment! Certainly the justification for givingachievement tests reflects this view. Kids goforth with certificates stamped "Inspected by theIowa Test of Basic Skills, Form J," our guarantee

that a quality education has been received. Thequality by inspection model is simple, straight-forward, and powerful. It is widely accepted bothinside and outside of education.

But, how well is this system of ensuringquality through inspection working? There aregrowing indications that it may not be workingall that well.

The National Center for Research on Evalua-tion, Standards, and Student Testing recentlypublished a report by Lori Shepard (1992) thatasked Will National Tests Improve StudentLearning? After an extensive review of theliterature, Shepard concluded

Research evidence on the effects of traditionalstandardized tests when used as highstakesaccountability instruments is strikingly negative. Itwould not be far fetched to say that testing in thepast decade has aztually reduced the quality ofinstruction for many students rather than improvingeducation. (p. 16)

Tierney, Carter, & Desai (1991) in theiroutstanding book on portfolio assessment alsoreviewed assessment practices and came to thisconclusion:

As we reflected upon assessment procedares inplace in most schools, we realized that studentswere not involved. It seemed as if assessment wereused to police rather than enhance learning. (p. 108,abbreviated for brevity)

Here is yet another opinion. Kenneth Graywriting in the January Phi Delta Kappan arguesthat the testing itself may have actually taken ona life of its own having little to do with qualityeducational standards.

...a close examination of high schools suggests thattheir real purpose is not educating but rank orderingthe graduating class (p. 373).

My final quotation in this series comes from agroup of third graders who had just had their firstencounter with the district mandated CaliforniaAcKevement Test. The district's math curriculumencouraged children to work together in groupsto solve problems, a highly valued skill inAmerican industry. When the test came along,they couldn't understand why they weren'tallowed to work together to solve the problems."Why can't we work together," they wanted to

2 3

know, "That's how we always solve math prob-lems." To these children, there was a gap be-tween learning and assessment that made nosense at all.

But this gap is everywhere. Consider howcurriculum and assessment departments areusually administered and housed. Curriculum inone office, Assessment another.

Where does this learning/assessment gapcome from? I think it has two sources, one ismanagement theory, the other learning theory.

In the early years of this century FrederickWinslow Taylor introduced American industry tothe concept of setting output standards thatdefined accqtable performance. Taylor's ap-proach remained the guiding principle in Ameri-can industry for almost 75 years. Taylor's book,Scientific Management, was adopted directlyinto educational thinldng through the work ofMager, Tyler and others (Grey, 1993). Thelanguage is familiar: clearly defined objectives,measurable outcomes. The concept of measur-able outcomes is particularly important to scien-tific management. These are outcomes expressedin terms of numbers. We in education, havedeveloped a habit of thinking about outcomes assomething that can be expressed in numbers.Thus, the concepts of scientific managementhave become deeply entrenched in educationalthinking. Whether it be a dated term like 'man-agement by objectives' or the latest in buzzwords 'outcomesbased education', 'bench-marks' the focus remains setting standardsand inspecting the output in quantitative terms toensure that the standards are met. Production:Inspection. Curriculum: Assessment. Separatefunctions, each with a unique responsibility forthe success of the whole.

The second factor supporting the gulf is thatmany educators understand learning to be pas-sive. This means that teaching involves transmit-ting knowledge and learning involves receivingit (Shepard, 1991). When we generate laundrylists of skills and knowledge kids are supposedto acquire, and then design achievement testsfocused on those specific skills and knowledge,

3

Poulson: Bridging the Gc43

we assume learning is passive. Many of thetheories of learning developed during the firsthalf of the 20th century support this view explic-itly (Mayer, 1992). Thus, the classmom is a kindof educational service station where students lineup for a "knowledge fill up" while the evaluationdepartment's inspectors determine exactly howmuch knowledge actually "stays in the tank."Quality is assured by independent inspection of apredefined outcome.

But as there are two sources of the quality byinspection strategy, there are also two thingswrong with it. First, learning is active, notpassive. The second is industry itself has begunto abandon the quality by inspection strategy.

Let's start with learning. I like the way theneurobiologist Graham Hoyle (quoted in Calvin,1990) explthns the way the brain works:

What you've got to realize is that every cell in thenervous system is not sitting there waiting to be toldwhat to do. It's doing it the whole darn time. Ifthere's input to the nervous system, fine. It willreact to it. But the nervous system is primarily adevice for generating action spontaneously. It's anongoing affair. The biggest mistake that peoplemake is in thinking of it as a input-output device.

Contemporary learning theory assumes anactive learner, thanks to Piaget and many others.The emerging view is that knowledge is some-thing learners construct, not something theyreceive (see Meyer, 1992). That puts the studentin charge.

Knowledge as something the learner con-structs is not a new idea. John Dewey wastalldng about it 100 years ago. It means that whatsomeone learns and how they remember, under-stand, and represent that knowledge varies withmany factors including experience, culture, thelearning environment, and the anticipated assess-ment.

The second problem with quality by inspec-tion is that the principles of scientific manage-ment are themselves being called into questionby industry itself. Let me illustrate the problemswith a quotation from the Toronto Sun (April 25,1983, quoted in (3itlow & Gitlow, 1987, p. 32).

They're still laughing about this at IBM.

4

Paulson: Bridging the Gap

Apparently the computer giant decided to havesome parts manufactured in Japan as a trial project.

In the specifications they set out that the limit ofdefective parts would be acceptable at three unitsper 10,000.

When the delivery came in there was an accompa-nying letter.

"We Japanese have a hard time understanding NorthAmerican business practices. But the three defectiveparts per 10,000 have been included and arewrapped separately. Hope this pleases."

"Set high standards and test to ensure thosestandards are met." That statement has a familiarring. Phrases like that appear in almosteverybody's educational reform legislation. Yet,IBM's effort to set high standards and implementstrict quality control had the exact opposite ofthe intended effect. Instead of improving quality,the high standards in effect institutionalizedmediocrity.

Let's talk a little about quality control fromthe viewpoint of IBM's Japanese subcontractor.IBM's request probably looked strange to theJapanese because Japanese industry has a totallydifferent philosophy of quality control. If youthink their philosophy is from a book by anancient master called, say, the Tao of QualityConti you are mistaken. The book on qualitycontrol used in Japan industry was written by anAmerican with a PhD in statistics named W.Edwards Deming.

Let me say just a few words about Deming forthose of you who are unfamiliar with him,although he seems to be quickly achieving thestatus of folk hero (See Bonstingl, 1992).Deming began his career as a quality controlspecialist for the telephone industry where hedeveloped a system that replaced endoftheline inspection with enline quality control. Hisprocedures were hard for many to swallowbecause they required companies to change theway they were managed. At the time Demingbegan promoting his concepts, America was theleading industrial power in the post World War IIworld and the attitude was "don't fix it if it ain'tbroke." Eventually, Deming found his way toJapan whose industry had a well earned repu-

4

tation for producing 'cheap copies.' The Japa-nese industrialists jumped at what Deming wassaying and, as I'm sure you are aware, thequality of Japanese products has skyrocketedsince the late '40s.

Deming's ideas have now come home to helpAmerican industry. Several years ago Fordinitiated a major effort to implement Deming'sideas, an effort which is now paying off in asharp rise in the consumer satisfaction ratings ofFord products. General Motois has also imple-mented Deming's quality control principles in itsnew Saturn Division. Today, Ford sales are upand Saturn (recently ranked number three incustomer satisfaction) literally cannot keep upwith orders. Customer satisfaction is up becausethe companies are building better cars. Theybuild better cars because quality control isdesigned into the manufacturing process, notimposed by inspecting the end product.

To me, the key to Deming's approach is notoutcomes based, it is process based. Processbased evaluation is what I see as a major featureof portfolio assessment. Let me illustrate bydescribing a portfolio project done by a teacherenrolled in my class on portfolio assessment. Irequire teachers in my class to put together aportfolio. To me, the portfolio is a personallearning environment. Therefore, I do notspecify contents. Rather, I tell them I'm lessinterested in WHAT is in their portfolio thanWHY it is there. Thus, deciding on the purposeof their portfolio and what it should containbecomes the major challenge. One of my favor-ite quotations from Deming's book is,

Improvement of quality is a method transferable todifferent problems and circumstances. It does notconsist of cookbook procedures on file ready forspecific application to this or that kind of product(Deming, 1986, p. 128).

In other words, no cookbooks. I require classmembers to reinvent the portfolio wheel and tofit it to their particular circumstance. Peopleoften ask why I require each person to reinventthe wheel? My answer? Then it's THEIR wheel.

Darlene Frazier was one of the teachers in this

class (see Frazier & Paulson, 1992, Paulson,Paulson & Frazier, in press). Frazier had a

special education pullout class of learningdisabled fourth graders struggling with writing.

She decided to do her portfolio "...to track the

learning of my class of learning disabled stu-dents." She then decided that an important part

of her portfolio would be examples of her stu-

dents writings collected over time. Then Frazier

did an unusual and inspired thing when the time

came for her to start collecting examples for herportfolio. She went to her students and asked

THEM to select the examples to put into herportfolio. She told them she was taking a class

and as part of that class she needed to prove that

they were learning to write better and challenged

them to provide the evidence. The kids loved the

idea and jumped on the opportunity of making apositive contribution to their teacher's portfolio.

Frazier was, in effect, taking the stance of asupervisor in a Deming shop who was asking the

workers to make quality control decisions.

It is difficult to take an idea from one field

like manufacturing and transfer it to another like

education (See Educational Leaderslzip, Novem-

ber 1992; and Phi Delta Kappan, January 1993)especially when the approach requires a compre-hensive change in the way organizations arestructured and managed. I will limit my com-ments to portfolios in the classroom. Think ofthe classroom as an enterprise with a supervisor

and several workers. In doing so, I will further

limit myself to two of Deming's areas of con-

cern: quality by design; and the supervision of

workers.

In the Deming approach, management's role

is more to facilitate than control and workers

assume a much larger role in determining goals,production schedules, controlling quality, and

engaging in self-management. Usually, when

Deming's principles are applied to education,

administrators are considered the facilitators,teachers the worker-managers, and the child as

the ultimate customer of the service.

My own version differs a little. I don't think

of students as customers and schools don't exist

5

Paulson: Bridging the Gap

to serve students. Schools exist to serve societyin a very broad sense. Rather than customers, I

see the students as the production workers (SeeBrandt, 1992) whose job is to get an education.Remember the new learning theory learning

is what the learner does, knowledge is what the

learner constructs. Teaching involves facilitating

learning and managing the learning environment,

an enormously challenging task but a supervi-

sory one nonetheless.

If you think of students as workers and teach-

ers as supervisors, then the Deming principles ofsupervision make a lot of sense when applied to

the classroom. Let's return to Darlene Frazier'sclassroom. Frazier's students were so enthusias-

tic about contributing to her portfolio that bymidyear they asked her to do portfolios of their

own. I'd like to share one of those portfolioswith you because it illustrates how quality by

design began to permeate the classroom. Sara

chose to use Darlene's portfolio as a model, thus

its rather formal organization (see F. Paulson &

Paulson, 1992-b).This appears as the first page of Sara's port-

foli:, it's what Sarah herself wrote when she gave

her reason for having a portfolio.

Rationale

I want to use this portfolio to improve my writing.

Think of the Deming principle of building

quality into the process, not imposing it from

outside. Sarah is not concerned with outsideinspection. She did not write "I want to use this

portfolio to pass writing," or "to get a good

grade," or even "to satisfy state requirements."Sarah said, "I want to use this portfolio to im-

prove my writing." To me, this one simplestatement is more powerful than all the laundry

lists of mandated knowledge and skills so com-

mon in education. This statement incorporatesquality by design, the force behind Deming's

goal of "never-ending improvement."

But having a purpose for a portfolio is only

Poulson: Bridging the Gap

the first step. How does Sarah use her portfolioto improve her writing? Well, Sarah, withFrazier's help, developed a plan. For one thing,Sarah identifies the issues that she wants toaddress that will make her a better writer. Sarahsets goals.

Notice the difference between this and thetraditional managementbyoutcomes view thatthe curriculum tells the child what to learn inorder to be a good writer. But Frazier sat downwith her students in what Deming might call aQualityControl Circle and they, teacher andstudents as a group, talked about good writingand what they could do tc improve. Here is thesecond page of Sarah's portfolio where she listsspecific goals she chose to work on. By the way,this is but one example. Each of Sarah's class-mates developed a unique list.

Intenti

I want to improve my punctuation.

I want to improve my AWA scores.

I want to improve my imagination in writing storiesto make them better.

But a part of setting goals is to considerstandards. How do we know when we reach ourgoals? Here is the next page in Sarah's portfolio.Part of the discussion in the QualityControlCircle concerned standards how peopledecided what constitutes good writing.

5tandards

There will be more checks on the punctuation checklist.

Scores on AWA this year will remain the same or goup.

Scores on "word choice" should go up.

I am convinced that children learn best whenthey also learn to set their own goals and stan-

6

dards and develop skill at judging their ownperformance. This is consistent with the newlearning theory (see Zimmerman, Bandura, &Pons, 1992). But, that doesn't mean rejectingsomeone else's standards. Some of the studentsin Frazier's class referred to standards thatappeared in the State Department of Education'sanalytic scoring rubric ("I want to improve myAWA scores").

However, the way Frazier structured theexperience put the students, not the State ofOregon, in control. First, she presented the rubricto show them what the State's inspector wouldbe looking for, not so much (in my opinion) todefine good writing but to help them understandhow they would be judged by others. But, whenshe taught them to use the rubric, and they did soon a regular basis, Frazier allowed them toestablish their own standards for making judg-ments. They worked together in little QualityContiol Circles to negotiate the scores they gavethemselves and each other. Their negotiated '3'might be very different from the '3' awarded bya trained inspector, but that doesn't matter. Whatdoes matter is that they were learning to judgethe quality of their work and that today's '3'becomes the basis against which they judgetomorrow's work. The goal is "NeverEndingImprovement," not passing a test, satisfying thestate, or pleasing a third party. I think of this as away Frazier found to encourage quality workdespite state mandated standards.

But Frazier also encouraged the students todevelop their own personal goals and standardsfor good work. One of the things Sarah chose towork on was punctuation. Here is a checklist thatFrazier and Sarah developed together. Sarah usedit regularly. It is one of the ways Frazier workedwith Sarah to help her learn to recognize qualitywork. The key to quality by design is that Frazierand Sarah worked together to develop thischecklist. It would have been far less effective ifFrazier had pulled out an already existing check-list and impsed it. Building the quality controlprocedures themselves was a major feature inFrazier's instructional program.

punctuation Pals

Period at the end of sentence

Question mark at the end of question

Exclamation marks to show excitement

Quotation marks when someone is speaking

Commas for pauses

This is part of an area called studentnegoti-ated standards (See P. Paulson & Paulson, 1992-c) that has major implications for portfolioassessment. Let me share another example, thisone is from Wyoming, Michigan. Last year,when Marie Spaak started her short story unit,she distributed examples of short stories includ-ing published stories and stories written bystudents in last year's class. The children dis-cussed the examples and brainstormed aboutwhat characterized a good short story. After thediscussion, she handed out empty rubric formsand asked each child to write personal standardsfor a well written short story. These studentnegotiated standards became the basis for evalu-ating each student's performance on the unit.

Short Story

Character

Is there one character that plays the most impor-tant role?

Plot

Is there a problem?

Does the action build to a climax?

Is the problem solved?

Is there a conclusion?ThemeDoes the story present an important idea aboutlife?

OtherIs the paper neat?Is the paper free of run on sentences?Is the paper free of spelling and grammaticalerrors?

7

Paulson: iridging the Gap

Here is one student's rubric. Again, the classand the teacher form a QualityControl Circlethat helps each student assume responsibility forproducing quality work.

Now lets turn to the question of what contentsSarah chooses for her portfolio. Although shediscussed her selections with her teacher, Sarahchose all the items herself. She included finaldrafts, rough drafts, and copies of the punctua-tion checklist. She also kept a record of her selfassessments using the analytic writing rubric.She knew why each piece was there and she hada plan for using that knowledge to reach hergoal. She turned her portfolio into a stepbystep record of her growth as a writer. Quality bydesign, not quality by inspection.

Another part of putting together a portfolio isreflecting, making judgments about the contentsthemselves. These reflections are an importantpart of quality control. Sarah developed anevaluation plan for her portfolio (I suspect Sarahmay have been familiar with IEPs).

Judgment

I will write a cognitive letter discussing how well Imeet my intents.

A review of my punctuation checklist plus a reviewof my AWA scores, plus a review of my cognitiveletter will help people look at my portfolio to makea judgment.

Sarah did not actually finish her portfolio. Shewas judged ready to return to her regular class.Here are some final judgment statements thatsome of the other children in her class wrote.Remember, at thy., beginning of the year theywere defeated unwilling to write. Here iswhat some of them wrote at the end.

Dear People,

I think I improved in my cursive writing and myAWA scores. If you do not believe me look in myportfolio, it has proof. Just read my first story andmy last....

Paulson: Bridging the Gop

Dear Reader,

....I am doing good in Voice. I just don't say "I don'tlike voice very much." That's not the way Voice is."I hate Voice." That's the way Voice is.

Thank you for reading my portfolio. I hope you reehow I improved. I am still trying. Thank you.

If quality is to be built into the system ratherthan imposed from outside, then the workersbecome the essential element of quality control.Saturn builds quality automobiles becausemanagement trains workers in quality controland holds them responsible for it. If Saturn'semployees were students, they would be settingtheir own goals, negotiating their own standards,designing their own tests which is exactlywhat was going on in Sarah's class.

In Deming's approach, workers are not trainedjust to produce products, they are trained toproduce quality products in an atmosphere wherestandards are not fixed, but are constantly beingraised through the efforts of the productionworkers themselves. Rather than address fixedstandards set externally, workers are concernedthat today's standard is higher than yesterday'sstandard, and tomorrow's standard is higher still.Deming calls this neverending improvement.Setting standards is a necessary part of qualityand must not be imposed externally. Demingwrites

Goals are necessary for you and for me, butnumerical goals set for other people, without a roadmap to reach that goal, have effects opposite to theeffects sought (Deming, 1986, p. 69).

To Deming, the strategy "set high standardsand test to be sure they are reached" is a loserlikely to yield exactly the outcome Shepard(1992, op. cit.) found in her research on achieve-ment tests used for highstakes accountability.

Recently I was videotaping a second gradernamed Colin as he showed me his portfolio.Colin's teacher, Cathy Ryan, believes that stu-dents should choose everything that goes intotheir portfolios. Thus, in Ryan's classroom the

8

students have a primary responsibility for con-trolling quality. One of her major responsibilitiesis helping Colin and his classmates becomebetter evaluators of their own work. I askedColin, "What is a portfolio?" He told me, "It'swhere I keep my best work." Then he said, "Mygoal is to have all my work in the portfoliobecause it's all my best work."

If the worker's job is quality production, thenin Deming's approach, it is management's job tolisten, respond, support, and remove obstacles.In all Deming companies, workers are encour-aged to make recommendations for improvingquality. Frequently, as many as 90% do so on aregular basis. But here is the thing that sets theapproach apart Management, in these compa-nies, implements 90% of the suggestions re-ceived (Gitlow & Gitlow, 1987). This manage-ment strategy does two things. First it leadsdirectly to improvement of the quality of thegoods produced. But it alsosends a message toworkers that the company truly values qualitywork. Remember the third graders who were notallowed to use group problem solving skills onthe district's achievement test. I wonder whatmessage they got about what the district values.

Howard and Shelly Gitlow in their book onDeming have this to say about supervision:

Instituting modern methods of supervision ismanaging for success. It generates a positiveatmosphere because supervisors are not policingworkers and blaming them for the handicaps of thesystem. The worker is treated with dignity, trainedand supervised properly, and knows what the job isso that performance can improve. (Gitlow &

Gitlow, 1987)

In my closing example, I return to the sameschool where one student told me he hatedportfolios because they were a hassle and an-other valued them only as a way to make roomin her desk. While Darlene Frazier's studentsmay have been academic rejects, the Wynn Wattsstudents are society's rejects. The school serves atroubled population of junior and senior highaged pupils. Most have been abused, some arewards of the court, some have police records,many have a history of inappropriate social

9

behavior that requires separation from regularclasses. Some live in dormitories on site. Theysee themselves as victims of forces over whichthey have no control and typically react toanything new or out of the ordinary (such asportfolios) with passive hostility (See F. Paulson& Paulson, 1992-a).

The teachers at the school (Jim Allison, MikeCenedella, Christine Downs, Jo Ann Michaelis,and Karen Zabel) had worked for about a year toadapt portfolio assessment for these students inthe hope that through portfolios, these passive,turned off students would begin to take respon-sibility for their own learning. Their approachwas to make the students responsible for theirown portfolios and the choice of items to in-

clude.

Earlier I mentioned giving a pep talk to thesestudents. This was one of their efforts to get aportfolio program started. Although I earned thetitle "Mr. Portfolio Dude," my efforts did little toactually get them interested in portfolios. Theywere unwilling to do something simply becausetheir teachers or anyone else wanted them to. Setstandards and test to ensure they dre met? Getserious! These kids were, by choice, 'out of theloop.'

But their teachers did find a way to draw theminto the loop. They uncovered an article in theMay, 1992 issue of Educational Leadership byBeth Hebert describing portfolio evening, a timewhen students present their portfolios to theirparents. The concept is similar to the "studentled parentteacher conferences" that originatedin Canada. Hebert writes that portfolio evening"is really the children's evening, and they needto 'run the show' as much as possible. Parentsand teachers have been impressed with theleadership and independence that even ouryoungest students have demonstrated in thissetting." (Hebert, 1992, p. 61).

The teachers were fascinated with the ideaand decided to give it a try at their school. Theymodified Hebert's approach only slightly to fitWynne Watts' far different setting. They choseafternoon instead of evening, and changed the

9

ikluison: Bridging the Gap

name to the more familiar open house.

They told the students about the open houseand that they would be expected to show theirportfolios to visitors. Several weeks later, invi-tations were sent to parents, friends, and family.In anticipation of a low parent turnout, theschool's teaching staff invited the residential andthe day treatment staffs to play the role of thesurrogate parents. These were the people towhom many of the kids "came home" afterschool. In all, 37 adults attended includingfamily members of 7 of the school's approxi-mately 25 students.

The visitors met briefly with the teachers priorto going to the classrooms. A staff spokespersonexplained that the open house was an opportu-nity for the students to show portfolios of theirschool work. The teachers encouraged thevisitors to be honest but emphasized that theyshould tell the children if they liked what theysaw. Privately, one teacher confided, "The kidsare really scared I don't know if this is goingto work or not."

The event was memorable as animated discus-sion filled the rooms. One teacher described itlike this, "I heard buzzing sounds, people talkingall the time. I saw a lot of eye contact and anincredible number of smiles. Some people camein looking pretty bummed out and nervous andsmiled a whole lot before it was over."

The portfolios themselves were impressive,considering that they didn't even exist a fewweeks prior to the open house. Obviously theopen house had provided an impetus to bothstudent and teacher. But more was going on thanjust motivation.

We debriefed the kids afterwards. While thegeneral response during the open house itselfwas positive, many students expressed a differ-ent opinion afterward. One group went on theattack, refusing to talk about the open house ortheir portfolios.

"How can you expect us to do high school work ifyou give us baby work?"

Paulson: Bridging the Gap

"Why don't we get grades like in real school?"

The teachers judged the open house a quali-fied success, and scheduled a second open housethree months later. The drill was the same andthe only thing really different was that the kidshad now been through one open house. Thestudents were given free reign over what to putinto portfolios and the open house was an-nounced well in advance. In the classrooms, thesecond open house felt very different from thefirst. The rooms were much quieter although theintensity remained. The students acted more "incontrol," more ready to engage the visitors, morearticulate, and more willing to talk about theirportfolios and explain what was going on. Iheard one student explain her portfolio to avisitor

"It's like a story. We put an old paper in and one ofour last ones, it shows how we've improved."

Another showed subtraction papers fromJanuary and June, pointing out (with the pridethat discovery brings) how much he hadchanged. This child had an IQ documented to be70, leading his teacher to comment, "It kind ofmakes you wonder about the tests, doesn't it."

The debriefing also contrasted sharply withthe first open house. Rather than react to theevent itself as they had the first time, they tendedto be more reflective, more concerned with thequality of the work they were sharing. Onestudent said that the visitors were not criticalenough,

--"ICITinoTrivird-Tva-E7gsTleriicWrala thetruth."

Another commented

"I've finally begun to learn what portfolios areabout, I didn't understand last time."

The class that had complained of 'baby work'focused on the portfolio process. One said,

"Last time I didn't care, I just threw anything in.This time I had more reason for choosing."

Another said,

"It was easier this time because the first timenobody knew what was going on. It was moreconfusing the first time."

*1.1011011MNIMOI1*IMMEMIIIII,.

A boy commented,

"This portfolio took more work. I knew what I wasdoing this time. I had more stuff in my portfolio."

The reactions of this class deserve specialcomment. At the first open house, they claimedto have produced poor quality work becausesomeone else had given them 'baby work.' Buttheirs was a vulnerable position with no place tohide. No one had graded or scored their port-folios. They had written the papers, filled out thework sheets, drawn the pictures, and ABOVEALL had chosen what to show. And, when theylooked into the mirror of portfolio ownershipthey saw poor quality they saw 'baby work.' Isuspect they began to think about the quality oftheir work realistically for the first time. Thus,although they tried really hard, it was difficultfor them to blame someone else for the workthey chose to place into their portfolios. Thesecond time around they chose a different strat-egy, one in which they began to assume control.As one girl said,

10

"It was easier to do the reflections [this time], I hadmore reasons for choosing the work I put into myportfolio."

I've described two portfolio programs thatillustrate the close tie between learning andassessment. The emotionally disabled students atWynne Watts with their long histories of beingvictimized by their own poor choices began torealize that they could make choices that hadpositive consequences. In their experienceassessment was something always done to them.Poor grades and test scores had nothing to dowith them personally they were from theoutside, awarded by authority. These studentswere taking the first step in controlling qualityby realizing that quality can be controlled. Thismay not be news to most of us, but to these kidsit was a revelation.

The learning disabled students in Frazier's

class were starting from a different place. Theywere the rejects that the quality control proce-dures had kicked out. Their challenge was toovercome a history of failure by taking directcontrol of the quality of their work. They did thisby taking charge of assessment as well as learn-ing. They set goals, negotiated standards, andjudged the quality of their work, all the thingsthat are usually done by the evaluation depart-ment.

If I were to characterize portfolios and port-folio assessment in a single phrase, I wouldwould echo Tierney, Carter, and Desai (1991),saying thatportfolios offers the opportunity forassessment that empowers learning rather thanassessment that polices learners. I was standingon the steps of the Wynne Watts school as thevisitors were leaving after the first open house.One man, a grandfather, looked at me as hepassed and said, "I sure wish we had somethinglike that when I was a kid."

ReferencesBonstinl; J. J. (1992, November). The quality revolution

in education. Educational Leadership, 50(3). 4-9.

Brandt, R. (1992, November). On Deming and schoolquality: A conversation with Enid Brown. Educational

Leadership, 50(3), 28-31.

Calvin, W. H. (1990). The cerebral symphony: Seashore

reflections on the structure of consciousness. New

York: Bantam.

Deming, W. E. (1986). Out of the crisis. Cambridge, MA:MIT, Center for Advanced Engineering Study.

Frazier, D. M., & Paulson, F. L. (1992, May). Howportfolios motivate reluctant writers. EducationalLeadership, 49(8), 62-65.

Paulson: Bridging the Gap

Gray, K. (1993). Why we will lose: Taylorism inAmerica's high schools. Phi Delta Kappan, 74(5), 370-374.

Gitlow, H. S., & Gitlow, S. J. (1987). The Dem' ig guide to

quality and competitive position. Englewoou Cliffs, NJ:Prentice Hall.

Meyer, R. E. (1992). Cognition and instruction: Theirhistoric meeting within educational psychology.Journal of Educational Psychology, 84(4),405-412.

Paulson, F. L., & Paulson, P. R. (1992-a, Summer).Afternoons to remember: Portfolio open houses foremotionally disabled students. Porffolio News, 3(4),

3,7,8-10.

Paulson, F. L., & Paulson, P. R. (1992-b). The making of aportfolio. Unpublished manuscript, revised.

Paulson, Pearl R. & Paulson F. L. (1992, November).Student-negotiated standards for portfolios. Paperpresented at a workshop on the evaluation of writing,National Council of Teachers of English, Louisville,KY.

Paulson, Pearl R., Paulson, F. L., & Frazier, D. M. (Inpress). Sarah's portfolio. In Yancey, K. (Ed.). Poryb-lios: Voices from the classroom. Urbana, IL: NationalCouncil of Teachers of English.

Schmoker, M, & Wilson, R. B. (1993). Transformingschools through total quality education. Phi Delta

Kappan, 75(5), 407-410.

Shepard, L. A. (1991). Psychometricians' beliefs aboutlearning. Educational Researcher, 20, 2-16.

Shepard, L. A. (1992). Will national tests improve student

learning? (CSE Technical Report 342). Los Angeles:UCLA Center for the Study of Evaluation.

Tierney, R. J., Carter, M. A., & Desai, L E. (1991) Portfo-lio assessment in the reading-writing classroom.Norwood., MA: Christopher-Gordon.

Zimmerman, B. J., Bandura, A., & Martinez-Pons, M.(1992). Self-motivation for academic attainment: Therole of self-efficacy beliefs and personal goal setting.American Educational Research Journal, 29, 663-676.

12