S C I E N C E N E W S2 9 6 N O V E M B E R 5 , 2 0 0 5 V O L . 1 6 8

PUSHING THE LIMITDigital-communications experts are zeroing in on the perfect code

BY ERICA KLARREICH

When SMART-1, the European SpaceAgency’s first mission to the moon,launched in September 2003, astron-omers hailed it as the testing ground fora revolutionary and efficient solar-elec-

tric-propulsion technology. While this technologicalleap absorbed the attention of scientists and the newsmedia, a second, quieter revolution aboard SMART-1went almost unheralded. Only a small band of math-ematicians and engineers appreciated the quantumleap forward for digital communications. Incorpo-rated into SMART-1’s computers was a system forencoding data transmissions that, in a precise math-ematical sense, is practically perfect.

Space engineers have long grappled with the problem of howto reliably transmit data, such as pictures and scientific meas-urements, from space probes back to Earth. How can messagestravel hundreds of millions of miles without the data becominghopelessly garbled by noise? In a less extreme situation, as anycell phone user can attest, noise is also an issue for communi-cation systems on Earth.

Over the past half-century, mathematicians and computer sci-entists have come up with clever approaches to the noise problem.They’ve devised codes that intentionally incorporate redundancyinto a message, so that even if noise corrupts some portions, therecipient can usually figure it out. Called error-correcting codes,these underlie a host of systems for digital communication and datastorage, including cell phones, the Internet, and compact disks.Space missions use this technique to send messages from trans-mitters that are often no more powerful than a dim light bulb.

Yet coding theorists have been aware that their codes fall farshort of what can, in theory, be achieved. In 1948, mathematicianClaude Shannon, then at Bell Telephone Laboratories in MurrayHill, N.J., published a landmark paper in which he set a specificgoal for coding theorists. Shannon showed that at any given noiselevel, there is an upper limit on the ratio of the information to theredundancy required for accurate transmission. As with the speedof light in physics, this limit is unattainable, but—in theory atleast—highly efficient codes can come arbitrarily close.

The trouble was that no one could figure out how to constructthe superefficient codes that Shannon’s theory had predicted. Bythe early 1990s, state-of-the-art codes were typically gettinginformation across at only about half the rate that Shannon’s lawsaid was possible.

Then, in the mid-1990s, an earthquake shook the coding-the-ory landscape. A pair of French engineers—outsiders to the worldof coding theory—astonished the insiders with their invention of

what they called turbo codes, which come within a hair’s breadthof Shannon’s limit.

Later in the decade, coding theorists realized that a long-for-gotten method called low-density parity-check (LDPC) codingcould edge even closer to the limit.

Turbo codes and LDPC codes are now coming into play. Inaddition to being aboard the SMART-1 mission, turbo-code tech-nology is on its way to Mercury on NASA’s Messenger mission,launched last year. In the past couple of years, turbo codes havealso found their way into millions of mobile phones, enablingusers to send audio and video clips and to surf the Internet.

LDPC codes, meanwhile, have become the new standard fordigital-satellite television. Hundreds of research groups are study-ing potential applications of the two kinds of codes at universitiesand industry giants including Sony, Motorola, Qualcomm, and

Samsung.“In the lab, we’re there” at the Shan-

non limit, says Robert McEliece, acoding theorist at the California Insti-tute of Technology in Pasadena.“Slowly, the word is getting out, andthe technology is being perfected.”

The closing of the gap betweenstate-of-the-art codes and the Shan-non limit could save space agenciestens of millions of dollars on everymission because they could uselighter data transmitters withsmaller batteries and antennas. Forcell phone and other wireless tech-

nologies, the new codes could reduce the number of dead spotsin the world and lower the cost of service.

NOISY COMMUNICATION Most engineering fields startwith inventions; researchers then gradually figure out the lim-its of what can be achieved. Coding theory experienced thereverse. The field got launched by Shannon’s result, which usedtheoretical insights to place an absolute limit on what the clevercoders of the future could achieve.

Shannon considered how much redundancy must be added to amessage so that the information in it can survive a noisy transmis-sion. One way to fight noise is simply to send the same message sev-eral times. For example, if a friend can’t hear what you’re saying ata noisy party, you might repeat yourself. However, it might be moreeffective to rephrase your message. Coding theorists look for effec-tive ways to rephrase the information in a message to get it acrossusing the smallest number of bits—zeros and ones.

For example, to protect a message against a single transmis-sion error, one solution is to send the message three times. How-ever, that requires three times as many bits as the original did.In 1950, Richard Hamming of Bell Laboratories came up witha much more efficient approach that adds only three redundantbits for every four bits of the original message.

“Slowly, theword is getting out, and thetechnology is beingperfected.”— ROBERT McELIECE,CALIFORNIA INSTITUTE OF TECHNOLOGY

EK BOB-BB BOB 11/2/05 2:25 PM Page 296

S C I E N C E N E W S2 9 8 N O V E M B E R 5 , 2 0 0 5 V O L . 1 6 8

EUR

OP

EAN

SPA

CE

AG

ENC

Y

In Hamming’s approach, each of the three added bits says some-thing about a particular combination of bits in the original mes-sage. Given a four-bit message, bit number 5 is chosen in such away that bits 1, 2, 3, and 5 will have an even number of ones. Bit6 is chosen to make bits 1, 2, 4, and 6 have an even number ofones, and bit 7 is chosen to make bits 2, 3, 4, and 7 have an evennumber of ones. Chase through the possibilities, and you’ll find thatif noise flips one of the seven transmitted bits from 0 to 1, or viceversa, even-odd checks reveal which bit is wrong.

Over the decades, coding theorists have come up with evenmore-efficient schemes and ones that can cope with a higher errorrate. Shannon’s law, however, says that there is a limit to how goodthese codes can get—whether thecommunication channel is a fiber-optic cable or a noisy room. Try tosend more information and lessredundancy than the channel cansupport, and the message won’t sur-vive the channel’s noise.

“Shannon showed that even withinfinite computing power, infinitelybright engineers, and no constraintson your budget, there is thisabsolute limit on what you canachieve,” McEliece says.

Shannon’s law has a bright side,however. He proved that almostevery code whose efficiency rate isbelow the Shannon limit would per-mit the recipient to recover the orig-inal message almost perfectly.

But there’s a catch. Shannondidn’t provide a practical recipe forconstructing reliable codes thatcome close to the Shannon limit. In the decades that followed,researchers tried but failed to develop such codes. Coding theoristsbecame fond of saying that almost all codes are good—except theones they know about.

TURBO BOOST In 1993 at the IEEE International Conferenceon Communications in Geneva, a pair of French engineers madean incredible claim. They reported that they had come up with acoding method that could provide almost perfect reliability at a ratebreathtakingly close to the Shannon limit. These “turbo” codes,the researchers asserted, could transmit data about twice as fastas other codes do or, alternatively, could use only half as muchtransmitting power to achieve the same rates as other codes do.

“Their simulation curves claimed unbelievable performance,way beyond what was thought possible,” McEliece says.

Many experts at the conference scoffed at the claims and did-n’t bother attending the engineers’ talk. “They said we musthave made an error in our simulations,” recalls Claude Berrouof the French National School of Telecommunications in Brest,who invented turbo codes together with his colleague the lateAlain Glavieux.

Within a year, however, coding theorists were reproducing theFrench engineers’ results. “Suddenly, the whole world of coding the-ory started twinkling and scintillating with the news,” says MichaelTanner, a coding theorist at the University of Illinois at Chicago.“Turbo codes changed the whole problem.”

“The thing that blew everyone away about turbo codes is not justthat they get so close to Shannon capacity but that they’re so easy.How could we have overlooked them?” McEliece says. “I think itwas a case of fools rushing in where angels fear to tread. Berrouand Glavieux didn’t know the problem was supposed to be hard,so they managed to find a new way to go about it.”

Turbo codes use a divide-and-conquer approach in which eachof two decoders gets a different encoded version of the original

message and the decoders then collaborate. Berrou likens the codeto a crossword puzzle in which one would-be solver receives the“across” clues and another receives the “down” clues.

In the first round, each decoder uses its own clues to guess thebits of the original message, keeping track of how confident it isabout each guess. Next, the two decoders compare notes and eachupdates its guesses and confidence levels. In the analogy withcrosswords, if you guessed that an across word was wolverine, anda friend told you that a down clue also put a w in the first square,your confidence in your guess would grow. However, if you learnedthat the down clue put, say, a p in the first square, you would feelless confident about wolverine or possibly switch your guess to

another word.After the two decoders update

their proposed solutions, they com-pare notes again and then updatetheir solutions again. They continuethis process until they reach a con-sensus about the original message.This typically takes 4 to 10 passes ofinformation back and forth.

Rather than give two decoderspartial information, it might seemmore efficient to send all the cluesto a single decoder, in closer anal-ogy with how crosswords are usuallysolved. In principle, this wouldresult in more-accurate decoding ofthe message than the two turbodecoders could produce. In prac-tice, however, as the number of cluesincreases, the number of computa-tions needed in the decodingprocess grows exponentially, so

sending all the clues to one decoder would result in too slow adecoding process.

In hindsight, McEliece says, coding theory before 1993 wasoverly focused on the optimal, but inefficient, decoding that a sin-gle decoder offers.

“The genius of turbo codes is that Berrou and Glavieux said,‘Let’s not talk about the best possible decision but about a good-enough decision,’” McEliece says. “That’s a much less complexproblem, and they got it to work.”

PARITY REGAINED Although turbo codes quickly becameestablished as the fastest codes around, they soon had to sharethat title. Within a few years, coding theorists realized that a slightlymodified version of an old, obscure code could sometimes pro-duce even better results.

In 1958, Robert Gallager, then a Ph.D. student at the Massa-chusetts Institute of Technology (MIT), created a class of codes that,like Hamming’s codes, add redundant bits that permit decodersto do even-odd checks to guess which bits of the original messagehave been corrupted by noise. As with turbo codes, Gallager’s LDPCcodes use cross talk between decoders to gradually establish con-fidence about the likely bits of the original message. However, incontrast to turbo codes, which use just two decoders, LDPC codesuse a separate decoder for each bit of the message, creating a giantrumor mill. That requires thousands, or even tens of thousands,of decoders.

“It’s like turbo codes balkanized to the nth degree,” McEliece says.Gallager’s simulations showed that LDPC codes come very close

to the Shannon limit. However, his method was impractical at thetime because it presented a computational problem far too com-plex for the computers of the day.

“The very biggest computer at MIT in the early sixties had anamount of computer power only about equivalent to what you get

SMART TALKER — The European Space Agency’s SMART-1mission to the moon, depicted in this drawing, is testing asolar-electric-propulsion technology. The bigger news formathematicians is that the craft is sending those data toEarth via a digital code that’s exceptionally efficient.

(continued on page 301)

EK BOB-BB BOB 11/2/05 2:25 PM Page 298

W W W. S C I E N C E N E W S. O R G N O V E M B E R 5 , 2 0 0 5 V O L . 1 6 8 3 0 1

S.N

OR

CR

OSS

in a cell phone today,” recalls Gallager, now an MIT professor.Because of this limitation, most coding theorists regarded LDPCcodes as intriguing but quixotic—and promptly forgot about them.

LDPC codes are “a piece of 21st-centurycoding that happened to fall in the 20th cen-tury,” says David Forney, a coding theorist atMIT. In the 1960s, he worked at Codex Corp.in Newton, Mass., a Motorola subsidiary thatheld the patents to LDPC codes. “I’m embar-rassed to say we never once seriously consid-ered using those patents,” he admits.

In the mid-1990s, however, with the cod-ing-theory world buzzing about turbo codes’cross talk approach, explorations of severalresearch groups independently led themback to Gallager’s work.

“Gallager’s paper was almost clairvoyant,”McEliece says. “Every important concept aboutLDPC codes is buried in there, except one.”

That one concept is irregularity, in whichthere are more even-odd checks on certainbits of the original message than on others.For a crossword puzzle, this would be likehaving a particular letter of the grid bedefined not just by across and down cluesbut also by a diagonal clue.

The extra clues of irregular LDPC codespermit the decoders to guess the value ofcertain bits with an extremely high confi-dence level. The decoders can then use thesehigh-confidence bits to become more confident about the otherbits of the message, creating a “ripple effect,” says Amin Shokrol-

lahi of the Federal Polytechnic School of Lausanne in Switzer-land, one of the coding theorists who pioneered the concept ofirregularity. With this innovation, LDPC codes sometimes edgeout turbo codes in the race toward the Shannon limit.

The ideas behind turbo codes and LDPCcodes have rendered much of the preceding50 years of coding theory obsolete, says DavidMacKay of Cambridge University in England,one of the coding theorists who rediscoveredLDPC codes. “Future generations won’t haveto learn any of the stuff that has been thestandard in textbooks,” he says.

To some coding theorists, turbo codesand LDPC codes represent the end of cod-ing theory. In the lab, they’ve reachedwithin 5 percent of the maximum efficiency,and commercial systems are within 10 per-cent, says McEliece.

“We’re close enough to the Shannon limitthat from now on, the improvements will onlybe incremental,” says Thomas Richardson, acoding theorist at Flarion Technologies inBedminster, N.J. “This was probably the lastbig leap in coding.”

In 1,000 years, McEliece whimsicallysuggests, the Encyclopedia Galactica willsweep past most of the past half-century ofcoding theory to say simply, “Claude Shan-non formulated the notion of channelcapacity in 1948 A.D. Within severaldecades, mathematicians and engineers

had devised practical ways to communicate reliably at data rateswithin 1 percent of the Shannon limit….” ■

CODING POWER — Thanks to digital-coding advances, cell-phone towerssuch as this one in Arlington, Va., arecarrying audio and video clips and information from the Internet.

cognitive therapists usuallyblended psychodynamic tech-niques into their treatment, whilepsychodynamic therapists oftenexamined faulty thinking and irra-tional beliefs just as cognitivetherapists did. Moreover, thera-pists who relied most on psycho-dynamic techniques, regardless ofhow they labeled themselves, dis-played the greatest success in alle-viating depression.

In a 2002 paper that Ablon calls“a shocker,” clinicians and psychol-ogy graduate students rated video-taped sessions of therapists prac-

ticing what they considered either cognitive or interpersonaltherapy. The researchers found that, at least in the sessions withdepressed patients, both treatments fit the definition of cognitivetherapy, suggesting that a single therapy had been compared withitself. The two sets of therapists were similarly effective.

Q-set studies of other therapy sessions, including psychoanalytictreatment, typically reveal recurring themes in patient-therapistinteractions, Ablon says. Therapists who recognize these themes andfind ways to deal with them enjoy the most success, he adds.

For instance, in a psychoanalytic case videotaped over a 6-yearperiod, the analyst gradually realized that his patient regularly becameconfused when the conversation shifted to her disturbing sexualfeelings and relationships. At first, the analyst offered explanationsin an attempt to clarify matters. After recognizing that pattern, hehad more success by helping the woman confront her tendency to“play dumb” to avoid thinking about her sexuality.

Many proponents of randomized controlled trials regard Q-setstudies as a swamp of correlations that can’t establish what actu-

ally helps a patient. Moreover, many psychoanalysts frown on whatthey consider to be superficial attempts to measure what they do.

Ablon sees a future for his approach, though, as a guide for main-stream research. “We need to study treatments in randomized tri-als that resemble what clinicians do in the real world,” he says.

MONOPOLY BREAKER Brent D. Slife stood before an audienceat the annual APA meeting held in Washington, D.C., in August,and filed the equivalent of a philosophical antitrust suit against psy-chotherapy researchers.

Slife, a psychologist at Brigham Young University in Provo, Utah,bemoaned what he called “the almost dogmatic status” of the phi-losophy of empiricism in guiding examinations of psychotherapy.The conviction that scientific knowledge comes only from observ-able experiences is a value, not a “fact of the world,” Slife said. It isas if scientists studying reading would examine only how well indi-vidual readers identify the words that they see and ignore how read-ers get meaning out of a sentence.

Scientists should explore psychotherapy through qualitativemethods that don’t involve statistical analyses, Slife suggested.One type of qualitative study might use transcripts of psychotherapysessions to track changes in a patient’s conflicts and concerns.

Qualitative research provides an opening to explore approachesthat have been ignored in randomized controlled trials, such ashumanistic and existential psychotherapy, Slife says.

Proponents of evidence-based psychotherapy see little value inqualitative research. “If we took Slife’s approach, we’d quickly getbooted out of the health care system,” says Barlow.

In the quarrelsome world of psychotherapy studies, there’sone issue that everyone agrees on: Psychotherapists are fightingan uphill battle to procure more than minimal health insurancecoverage for their services. Norcross remarks, “The sad reality isthat insurance companies largely respond to financial consider-ations, not psychotherapy research.” ■K

EITH

SK

EEN

(continued from page 298)

EK BOB-BB BOB 11/2/05 2:26 PM Page 301