8/3/2019 Salzman and Lowell - 2008 - Making the Grade

1/3

C.E

MIABATA

Its a familiar story. Children around theworld have been tested, and the UnitedStates is in trouble. A US Department of

Education report1 from March concludes thatwithout substantial and sustained changesto the educational system, the United Stateswill relinquish its leadership in the twenty-first century. The panic plays out in countlessnewspaper articles and policy reports, recentlyleading to legislative responses such as theAmerica COMPETES Act, which contains alist of measures to boost average mathematicsand science test scores.

A countrys place in the new global economyis, according to these reports, determined by itsrank in the maths- and science-score hierarchy.Following this reasoning, one would conclude

that the US economy is threatened not only byJapan and South Korea, but also by Finland, Sin-gapore, New Zealand and the Czech Republic.The rankings that engender these fears are pri-marily based on two tests administered to mid-dle- and high-school students since 1995: theProgramme for International Student Assess-ment (PISA) and the Trends in InternationalMathematics and Science Study (TIMSS).

Improving education should be a priorityfor the nation, but erroneous interpretationsof international test scores may drive economicand competitiveness policy in the wrong direc-tion. When we consider that education testing

shows formidable US strength as the largestproducer of top-scoring students alongside asignificant problem at the bottom, the threat

to future competitiveness seems to besomething quite different from the

headlines2. Caution is neededso we neither create

policies that

overstock the science and technology work-force nor unthinkingly implement the educa-tion and social practices in other high-scoring

countries. A full grasp of the meaning of test-score differences should lead the next presi-dent to address education and competitivenessproblems more effectively than the recentAmerica COMPETES legislation, which isnow languishing for a lack of funding. Focus-ing the great consternation about education onreal rather than imagined problems requires acareful assessment of the evidence.

Lagging behind?It is misleading to gauge the relative position ofthe United States in the world based on a sim-plistic ranking of its students

test scores. This is much likemeasuring shoe size to pre-dict runners future racetimes while ignoring theirpast performance. There aresubstantial methodologicallimitations in using thesetests to compare nations,including reporting rankings that are basedon minute differences that are not statisticallysignificant3. For example, when consideringstatistically significant differences, nationaltest scores can be clustered into three mean-ingful levels and the United States consistently

ranks in a middle group on maths and sciencewhile being top ranked in civics4 the studyof citizenship and government. Overall, aboutone-fifth of other nations rank better and two-fifths rank underneath the United States.

Still, average performance tells us nothingabout the distribution of students with the verybest test scores. In maths and science, whenlooking at average scores, the United States isoutranked by countries such as Finland andSouth Korea. But the rankings change when weexamine the percentage of students who per-form at the top, those most likely to be tomor-rows innovators. The South Korean average

places it in the top-ranked group of nations,yet its relative proportion of top performing

students is 30% lower than that of the UnitedStates. In fact, the United States has a higherpercentage of top-performing students than

5 of the 14 others in the top-ranked group ofcountries with high average scores.

Moreover, it would seem inappropriate toconsider the United States, a country with apopulation of more than 300 million, in com-petition with Singapore, a country of 4.5 mil-lion, or with even smaller New Zealand. Theeconomies in these countries range from a grossdomestic product (GDP) of $124 billion in NewZealand to $236 billionin Finland, comparedwith the $14-trillion GDP of the United States.Perhaps a more apt comparison would be Mas-sachusetts with a population of 6.4 million and

a gross state product of $338

billion, or Colorado with 4.8million residents and a $230-billion state product. Althoughthe top group also includeseconomic powerhouses SouthKorea and Japan, which comein at under a fourteenth andless than a third, respectively,

of the size of the US economy, for the mostpart it makes more sense to compare US stateeconomies with other countries because it isMassachusetts or California that is competing,for example, with Singapore in developing theirbiotech industries.

If, as we argue, average test scores are mostlyirrelevant as a measure of economic potential,other indicators do matter. To produce leading-edge technology, one could argue that it is thenumbers of high-performing students that ismost important in the global economy.

These are students who can enter the sci-ence and engineering workforce or are likely toinnovate whatever their field of study. Remark-able, but little noted, is the fact that the UnitedStates produces the lions share of the worldsbest students (see graph opposite).

At the same time, low-performing studentscan hamper productivity and here, unfortu-

nately, the United States also stands out. TheUnited States produces more than one million

Making the gradeInternational testing that is used to predict the grim future of US science and technology

is being vastly misinterpreted, say Hal Salzman and Lindsay Lowell.

Average test scores

are largely irrelevant

as a measure of

economic potential.

28

Vol 453|1 May 2008

COMMENTARY

8/3/2019 Salzman and Lowell - 2008 - Making the Grade

2/3

TOP-PERFORMINGSTUDE

NTSBYCOUNTRY

low-performing maths and science studentseach year, more than any other country in theOrganisation for Economic Co-operation andDevelopment except for Mexico (see graph

overleaf). Although programmes to improveeducation for low-performing students andschools are included in the various policyreports, they are shunted to the backgroundwhen the headlines focus on increasing thenumbers of those at the top, and overall seem tocarry little weight when they are diluted as partof a long laundry list of recommendations.

Market mathsWithout a doubt, science, maths and technol-ogy education is needed in todays society,whether for its citizens to understand enoughto participate in public debate or just to operate

the technology of everyday life. However, someargue for more advanced courses as if theywant to prepare all students to be scientists orengineers. We believe that there is somethingfundamentally wrong with such an approach.

History suggests that policies designed tostockpile scientists and engineers arecounter-productive. Thespace race is typically citedas a success story of Ameri-can technological prowess,but less often discussed is theimpact of the workforce build-up on US engineering and sci-

ence in the years that followed.Following a spike in the num-bers of science and engineer-ing college graduates in thelate 1950s and early 1960s, aspectacular bust followed thatled to high unemployment inthese fields. For many yearsafterwards, fields such as physicswere thought of as poor careerchoices5. Similar boom-and-bust cycles have continued forthe past four decades, in engi-neering, in information technol-

ogy (IT) and in science.When demand is translated into

increased salaries and job openings, studentsrespond. When the IT industry was growing,the number of graduates in computer sciencekept pace, doubling over six years. Follow-ing the collapse of the IT industry bubble, thenumber of graduates fell by 17% between 2003and 2005. Employment in this field is just nowreaching the levels of the boom years but, withlittle prospect of rapid growth, students seem

to be wise in choosing other fields. Or, considerpetroleum engineering. This is an industrythat has had slow growth for two decades and,correspondingly, undergraduate enrolmentsdeclined 85% during that period, and mastersprogrammes instead attracted students fromareas of the world with fast-growing oil explo-ration. Today, 75% of US masters graduates inpetroleum engineering are foreign students ontemporary visas. Now, theUS industry has a real needfor more engineers becauseof increased demand foroil and new exploration

coupled with 20 years ofminimal hiring and anageing workforce. The oilindustry has responded byincreasing entry-level sala-ries 3060%over the past four to five years,far greater than in other fields. As a result,petroleum-engineering graduates have dou-bled in the past five years and freshmen enrol-ments, at Texas Tech University in Lubbock forexample, have increased more than sixfold.

When supply far exceeds demand, the bustthat follows reverberates for many years anddiscourages students even when demand does

increase later. As Michael Teitelbaum of theAlfred P. Sloan Foundation in New York notes,not only is there no evidence of any widespreadshortages but substantially more scientistsand engineers graduate from US universitiesthan can find attractive career

openings in the US work-force. Teitelbaum adds thatthis overproduction, whichleads to increasingly longerpostdocs in many sciencefields, makes our universi-ties look more like a system toproduce a pool of low-cost research labworkers with limited career prospects than a

high-quality training program for soon-to-beacademic researchers.

Social choicesThe beauty of brandishing a simple numberor a few facts is that they fit in a single head-line and focus the readers attention. However,before we send teams of educators to discoverthe educational secrets of Finland, Singapore,

New Zealand, South Koreaor Japan, we should domore study into the natureand context of their educa-tion systems. As the PISA

report6 notes, the tests donot evaluate schooling,per se, but the cumula-tive impact of learningexperiences ... starting in

early childhood and up to the age of 15 andembracing experiences both in school and athome. That is, much is made of a few selectschooling practices in each country and recom-mendations are made to emulate them withoutconsidering what the effect is both on the livesof these children and on the economy.

Sending children to classes six days a week,extra preparation courses nights and weekends,

and having a single examinaton that decidestheir fate, as is done in Japan, is not a choicemost US parents would make. Nor is the socialdiscipline in Singapore that seems to keep stu-dents on the straight and narrow path: death

for drug pushers, prohibitions on spit-ting and, for offences in between such asrobbery after 7 p.m., at least 12 strokesof the cane. Although South Koreasspectacular economic rise is held in awe,and its tenfold-per-capita GDP increaseover the past 20 years is widely praised,rarely noted is the close to 250% rise inthe incidence of suicide over the same

period, with suicide becoming a lead-ing cause of death among young peo-ple. With South Korea are Finland andNew Zealand at the higher end of theglobal rankings of test scores and sui-cide rates. No single factor is respon-sible for either high scores or suicides,but mental-health experts cite thepressure leading to one outcome asa factor leading to the other in manyhigh-scoring nations7.

The future educational path forthe United States should come fromlooking within the country rather

than lionizing faraway test-scorechampions. Our analysis3 of the

History suggests that

policies designed to

stockpile scientists andengineers are counter-

productive.

29

NATURE|Vol 453|1 May 2008 COMMENTARY

8/3/2019 Salzman and Lowell - 2008 - Making the Grade

3/3

LOWEST-PERFORMINGSTUDENTSBYCOUNTRY:

NUMBERANDSHAREOFTOTAL(15YEAROLDSIN

ALL30ORGANISATIONFORECONOMICCO-OPERATION

ANDDEVELOPMENTCOUNTRIES)

data suggests two fundamental problems thatrequire different approaches. First, pedago-gies must address science literacy for the largenumbers of low-performing students. Second,education policy for our highest-performingstudents needs to meet actual labour-marketdemand.

In the United States, a decades worth ofinternational test rankings based on slender

measures of academic achievement in scienceand maths have been stretched far beyondtheir usefulness. Perhaps policy-makers feel itis better to motivate policy by pointing to high-scoring Czechs with fear, instead of noting ourhigh-scoring Minnesotans as examples toemulate. But looking within the United Statesmay be the best way to learn about effectiveeducation. As the PISA authors emphasize intheir report, 90% of the variance in the scoresis within countries rather than between coun-tries. Therefore, most ofwhat one can learn abouthigh performance is due

to the variation in factorswithin the nations borders.It would seem far moreeffective to transfer best practices across cityand state lines than over oceans.

Chasing tailsIn America, little about the nations condition

can be gleaned from averages, whetherby assessments of income or edu-

cation. Our great opportunities aswell as our great limitations seemto be accompanied by great dis-parities. It is these extremes, the

tails at either end of the distribu-tion, that require much more

discussion than the averages. Sowe need to look beyond the meanto consider size and quality of theworkforce and the content of oureducation that will be drivinginnovation. In a country thathas a long history of innova-tion and high productivity, weshould start by looking at how

our best schools educate topperformers. It is unlikely thatthey do so by the types ofeducation heralded in othercountries.

Paying attention to theproblems at the bottom is asimportant, if not more so, than focusing onthe top. The most innovative technology haslimited use if the more than 70 million work-ers without college degrees do not have the

skills to use it effectively.The nations low perform-ers and schools should be

a headline concern andthe remedies are often tobe found in schools only

a neighbourhood or town away. It will be farmore effective to take the best that Americahas to offer before seeking elusive and poorlyunderstood practices found in a diverse collec-tion of small countries around the globe.

As advocates of evidence-based policy, weargue that competitiveness and educationpolicy should use the best available evidenceas a guide and not be driven by impressionsand rhetoric. Our analysis suggests that a betterunderstanding of the education data will lead to

better and, in many cases, different policy direc-tions from those now being advocated.

Hal Salzman is at the Urban

Institute, 2100 M Street NW, Washington, DC

20037, USA. Lindsay Lowell is at the Institute

for the Study of International Migration at

Georgetown University, 3300 Whitehaven Street,

Washington, DC 20007, USA.

1. www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf

2. Lynn, L. & Salzman, H.Issues in Science and TechnologyWinter, 7482 (2006).

3. Lowell, B. L. & Salzman, H.In the Eye of the Storm: Assessingthe Evidence on Science and Engineering Education, Quality, and

Workforce Demand (Urban Inst., Washington DC, 2007).4. Boe, E. E. & Shin, S.Phi Delta Kappan86, 688695 (2005).5. Kaiser, D. Social Res. 73, 12251252 (2006).6. Learning for Tomorrows World: First Results from PISA 2003

(OECD, Paris, 2004).7. Lim, M.Asias Ongoing Struggle with Suicide International

Affairs Journal at UC Davis (29 June 2007).8. PISA 2006 Science Competencies for Tomorrows World

(OECD, Paris, 2008).

To discuss this article or any of our education

material this week, visit http://tinyurl.com/6ndqko

A better understanding

of the education data will

lead to better policy.

30

NATURE|Vol 453|1 May 2008COMMENTARY