Save paper and follow @newyorker on Twitter

MAY 14, 2015

What a Dinosaur’s Mating Scream Sounds LikeBY RAFFI KHATCHADOURIAN

TCOURTESY HELLO GAMES

wo years ago, Sean Murray, a video-game developerfrom the town of Guildford, outside London,announced an ambitious game that he had beenworking on in secrecy with a small team: a fully

explorable digital cosmos, called No Man’s Sky(http://www.newyorker.com/magazine/2015/05/18/world-without-end-raffi-khatchadourian) (which I wrote about in themagazine this week). When the digital universe is complete, it will contain more than eighteenquintillion planets; it would take billions of years, Murray has calculated, for one player to explore themall. To create a representation of space that is so vast, Murray’s studio, Hello Games, is using a techniquecalled procedural generation: equations that draw upon random numbers to build naturalistic features,such as solar systems, planets, flora, and fauna.

When I visited Hello Games earlier this year, I expected to meet people working to figure out how theuniverse would look, and how it would function. One thing I hadn’t considered was the game’s audio. But,it turns out, No Man’s Sky poses a complicated question for its designers in this regard: How can youimbue this universe with naturalistic sound, from dinosaur grunts to whirring star-ship engines? Theanswer, initially, was unclear. Sound design in games, as in films, typically begins with the known; onceSteven Spielberg decided that “Jurassic Park” would contain velociraptors and the Tyrannosaurus rex, hissound technicians knew they had to create plausible voicing for such creatures. But No Man’s Sky’scosmos will be inhabited by countless imaginary organisms, based on random morphology, inenvironments shaped by chance. It will contain modular ships and architecture of unpredictable size anddesign. And everything in the game will be rendered only at the moment of encounter.

The job of solving the game’s sound problem, I learned, had fallen to Paul Weir, a composer and sounddesigner who specializes in the creation of audio for films, games, banks, and large stores, like Harrod’s.One morning at Hello Games, I climbed a narrow set of wooden stairs to the studio’s second floor, wheremembers of the audio team were working in a tiny trapezoidal room. Weir was seated at a computerflanked by speakers on pedestals. He has a compact, wiry body, which he maneuvers with easy formality.(He was the only person in the studio wearing a tie.) He maintains an affiliation with the London officeof Microsoft Studios, where he works as its audio director, and he is also well-versed in science fiction.Among other projects, he has worked on a BBC radio drama based on “The Hitchhiker’s Guide to theGalaxy.”

That morning, two programmers were with Weir, and a discussion was under way about how to fixunanticipated reverb that was appearing in the game. Like many parts of No Man’s Sky, Weir’s systemwas evolving through cycles of disassembly and reassembly. Working on his own for months, Weir hadbeen making adjustments to game sounds divorced from their graphical embodiments, but that week hewas trying to stitch key elements of his system back into No Man’s Sky’s master build. When I walked in,he was testing audio for a spaceship. The sound was multi-dimensional; layered with noises thrummingat alternate pitches, and rich in overtones, it seemed to be the byproduct of a genuine mechanism—combustion at the intensity of high-energy rocket thrust. “I love harmonic complexity,” Weir said. Thesource of the sound, which he had manipulated, was a hand dryer. “I always carry a recorder. For a lot ofjobs, particularly for a game like this, I have this rule: the sounds have to be a hundred-per-cent original.We haven’t sourced anything from sound libraries, but a lot of games would.”

No Man’s Sky will make use of a wide range of atmospheric sound. Fly past a cluster of stars in its 3-Dgalactic map, and you will hear a shimmering noise that gives the universe a bejewelled quality. The gamewill also contain a soundtrack by the band 65daysofstatic, recorded in an old church. Some of the band’smusic will be fragmented into micro-segments, which the game’s algorithms will recombine into ambientsoundscapes uniquely tailored to what players see.

Weir told me that No Man’s Sky’s biggest audio challenge was the creatures. Vocalizations for thedinosaurs in “Jurassic Park” were amalgams of field recordings, too: distorted utterances of whales orterriers. (The Tyrannosaurus rex’s roar was from a baby elephant.) But Weir did not have the budget tomake high-quality field recordings of exotic animals, nor did it make sense to do so, since there was noway to predict what individual animals in the game would look like, or do, or even what environment they

Enter your email address.

would be in. Instead, he decided to bestow each creature with a unique digital vocal tract, to simulatesounds consistent with its appearance. Rather than working against the game’s algorithmic chaos, hewould embrace it. Weir knew of no other game that did such a thing, and he thought that a fewprogrammers would be required to complete the coding on schedule. Then he reached out to a Scottishprogrammer and game designer named Sandy White. (In 1983, White’s game Ant Attack(https://www.youtube.com/watch?v=nAZnXa1Eamk) was among the first to use 3-D graphics.) In twomonths, White wrote the necessary software.

White had flown down from Edinburgh for the week to work on the game, and he was seated next toWeir. “The whole issue is: how do you synthesize creature vocals without them sounding synthetic?” hesaid. “Because the danger is that they will sound like Stephen Hawking.”

Our brains are very adept at detecting patterns, and the reason synthetic voices typically sound artificial isthat they are carried on sound waves that have a regular frequency: unvarying up-down-up-downmodulations that are unmistakably inorganic. White suspected that if he built digital vocal chords(stimulated by columns of mathematically simulated air) the system would achieve naturalism. “The firstresults were a bit like the squeaker out of a dog toy,” he said, which wasn’t surprising: blow through themouthpiece of a clarinet without the instrument, and the effect is similar. White then added a digitalversion of the pharynx, which sits behind the mouth and nasal cavity; it served as a resonator, amplifyingsounds produced by the vocal chords, but also altering their texture. The squeaks became elongated. Hecalled this the system’s “trumpety-chicken-duck-whale-car-horn phase.” By the end of January, severalweeks after he had started programming, he added a digital mouth—the final component necessary for arudimentary virtual vocal tract. Then he set about giving his creation a voice.

Every vowel is defined by a narrow band of frequencies, known as a formant, which are created by thevocal tract as a whole—the way sound resonates throughout all its parts. White found a paper from 1962,titled “A Study of Formants of the Pure Vowels of British English.” The paper, based on recordings oftwenty-five male subjects, contained a table of the relevant data. Late one night, alone in his Edinburghstudio, he copied the values for a vowel labeled “/a/ hard” and plugged them into his system. The digitalresonance that White had created—with vocal chords, pharynx, and mouth all affecting each other—caused the utterance to take on human character, and the result was a blood-curdling scream. The voicebroke, twisted, and grew hoarse during moments of high intensity. White gave me an MP3 of it, and Ilater played it for two people without telling them what it was. Both thought it came from an animal; one

wondered if it was a person being tortured, and the other wondered if it was a goat. White recalled, “Twoo’clock in the morning, headsets in, and the thing went ‘Aaaaahhhhh.’ I was sweating because it was soscary. But I was also like, This is working!”

From there, White set about making the system adaptable, so that it could be used for animals of myriadspecies and sizes. The week before I visited Guildford, he had designed a tool—using an off-the-shelfapp on an iPad—to allow Weir to implement his software to generate the creature sounds, which wouldthen be incorporated as code into the game. Sitting in his office, Weir held the iPad in his hand. Thedevice had a hybrid user interface: part laptop trackpad, part midi board—the kind you might see in asound studio, with many sliding levers—and part theremin. “A player would never see this,” he said. Weirexplained that he would use the iPad to perform vocalizations only for creature archetypes; then a set ofalgorithms would mutate each performance, to adapt it to the countless variations of that creature in thegame. The process is analogous to the way the creatures are designed graphically: using tools likePhotoshop, artists at Hello Games create archetypes that algorithms then transform.

Raising the iPad, Weir said, “It feels like an instrument.” He offered to play it. Drawing his finger acrossthe screen, he nudged the lever bars to indicate attributes like body mass, aggressiveness, windpipe length,wetness, screechiness, harshness. (The software makes sounds based on roughly a hundred differentparameters.) Then, while moving his thumbs across two graphical boxes on the iPad—one labelled “vowelmap,” the other “pitch”—and simultaneously twisting the device in space, he generated a vocalization.The iPad’s physical movement determined the energy behind the utterance: the arc of the motionshaping the sound’s arc.

Out came a tired, yawn-like rumbling, the deep grunt of a lumbering multi-ton herbivore. “I can changethe size,” Weir said. He tinkered with the iPad, and moved it differently, and the vocalization’s over-allfrequency become higher; the texture became rougher and wetter. After a few seconds, Weir gave it anupturn in pitch and intensity. The sound resembled the mating call of some tropical species.

“You see, you have to perform it,” Weir said. He is collaborating with an engineering Ph.D. candidate tobuild bespoke hardware to replace the iPad; using this, he hoped, would give the vocalizations morefluidity and, thus, realism.

“We are trying to convey emotion,” White said.

Raffi Khatchadourian became a staff writer at The New Yorker in 2008.

Weir adjusted the iPad again. “If I go way up—that’s more like a bird sound,” he said. Out came a longshrill bird call. White furrowed his brow. “That sounds slightly synthy,” he said. “There is a vibrato inthere that is very regular. Its experimental.” He had used the vibrato as a placeholder. It would later bereplaced by Perlin noise, an algorithm often used in movies and in video games to create natural-seemingtexture on computer-generated images.

Weir tinkered with various settings, and performed more sounds: the voices of an alien menagerie. Therewas an ungulate experiencing sudden sharp pain; the vocalization descended from a screechy high pitchto a lower, more guttural, exhalation. It sounded very real.

“That’s it falling off a cliff,” Weir said, laughing.

“I’d save that one,” White said. “It’s like a horse.”

The scope of No Man’s Sky, along its openness, made it easy to get carried away. At one point, Weirsuggested, “We could potentially have a situation where every tree in a forest could have a bird in it,which you would only hear if you are close enough to it. Then we would be automatically populatingbirds in forests, and spatially it would always sound correct.” He became more excited, and White added,“They are fairly musical. Why couldn’t they be musical—or at least in harmony?”

Since I visited Hello Games, Weir had integrated creature sounds into the master version of No Man’s Sky. Aboveis a sample that he created for The New Yorker using the game’s software.

Raffi Khatchadourian will participate in a Reddit Q. & A. about his piece on No Man’s Sky today, Thursday, at10 A.M.