TELEVISION

To TV sports fans,the yellow first-down line appearson stadium grassas if by magic,thanks to the wizardry of agroup of formerdefense engineers By Tekla S. Perry

ALL IN GAMETH

E

ALL IN GAMETH

E

31

IEE

ES

PE

CT

RU

M•

No

vember 2003

Stan Honey, the force behind the creation of Sportvision’s yellow football line, poses

on a miniature football field in the company’s laboratory. The yellow line on the live

video feed is generated to appear as if painted on the turf behind Honey.

TELEVISION

32

IEE

ES

PE

CT

RU

M•

No

vem

ber

2003

If you’ve watched a pro football broad-cast in the United States lately, you’veseen what Gaudelli is talking about: aneerily realistic bright yellow line createdon the playing field that shows you exactlyhow far the offense has to carry the ball toget a first down. “It makes it easier towatch the game, and that is what our jobis all about,” Gaudelli tells IEEE Spectrum.

That yellow line has become such astaple in U.S. football that no self-respect-ing network would think of televising agame without it. It even won an Emmyaward, for technical innovation. The tinycompany that pioneered the technology,Sportvision Inc., with offices in MountainView, Calif. (headquartered in Chicago),now covers up to 300 games a year with 18 crews. The state-of-the-art workstationsthe crews use, along with sensors andother hardware and software, solve a bar-rage of fiendishly difficult image-process-ing problems in a fraction of a second.And some casual viewers have no ideathat the yellow line they see on the field isa computer graphics figment no more“real” than the weather maps that seem toswirl behind television meteorologists.

The story of the yellow-line system,which is called 1st & Ten, is a classically cir-cuitous one that begins in the world ofmilitary technology. It goes into the hockeyarena, where an early version of Sport-vision’s tracking technology turned thepuck into a flashy orb that streaked acrossthe ice with a fiery tail. It moves onto thefootball field. And most recently, it hasbranched out into car racing sponsored byNASCAR (for National Association ofStock Car Automotive Racing, based inDaytona Beach, Fla.), where broadcastersconjure up graphics to give detailed racedata in real time. A future version of thetechnology will even take NASCAR dataand channel it to homes, where it will con-trol the movements of virtual cars in inter-active video games so couch potatoes cantest their racing skills against the pros.

Faster than a speeding puck

Flash back to 1994. IEEE Member StanHoney [preceding page] is executive vicepresident of technology for Rupert Mur-doch’s vast media and entertainmentempire, News Corp. (Sydney, Australia).It’s June, and Honey is in a meeting withDavid Hill, then president of News Corp.’sFox Sports (now chairman and CEO of FoxSports Television Group). Honey is rhap-sodizing about the possibilities of virtualbillboards, which would let techniciansinsert any graphics at will into the imagesof actual billboards in a stadium. Hill rejectsthe idea (since implemented by Sportvisionand others), but suddenly asks, “Could youtrack and highlight a hockey puck?”

Honey responds, “I tracked things alot harder than a hockey puck for the mil-itary, David, but you couldn’t afford it.”

“Just how much would it cost?” Hill asks.

“It would take two years to develop andcost about $2 million,” is the reply.

“You don’t understand the econom-ics of sports,” Hill tells Honey. “Writea memo.”

A few days later, back in his officenear San Francisco, Honey gets a callfrom Murdoch, who gets right to thepoint. “David says you can track andhighlight a hockey puck, and you canget it done by the 1996 January All-Stargame, and it’d only cost $2 million,”Murdoch says. “That is now your high-est priority. If anybody asks you aboutthe money, tell them to call me.”

With just 18 months to go before thegame, Honey immediately starts assem-bling his team, relying heavily on engi-neers he’d worked with at SRI Interna-tional in Menlo Park, Calif., in the early1980s. The group had developed an over-the-horizon radar, underwater sensors,and an ultraprecise radio-positioning sys-tem for the military. Some of them hadgone on with Honey when he foundedEtak Inc., the company that pioneered in-

car vehicle navigation; others had stayedat SRI. With the defense industry slump-ing, Honey has little trouble arousinginterest in his project to break newground in TV sports broadcasting.

The goal is simple. Make the televisedimage of a hockey puck glow so it’s eas-ier for the viewer to spot, and, when it’sgoing really fast, put a tail on it showingits path. Murdoch and company hope thesystem can overcome the main complaintabout televised hockey: the trouble thatcasual viewers have in following the fast-moving puck on TV screens.

Honey lays out his proposal. To sumup: it’s never been done before, it willundoubtedly be hard, but it’s doable.In other words, it’s “just the perfectproject.” He quickly gathers a team of10 and enlists help from Vista ResearchLLC (New York City), a group ofdefense engineers, and Shoreline Stu-dios Inc. (Vancouver, B.C., Canada), aspinoff of Silicon Graphics Inc. (Moun-tain View, Calif.). The project becomes“a taste of Camelot” for the engineersinvolved, Honey says.

To track the hockey puck, a number ofparameters have to be fed into a computersystem and updated continually. First, thesystem has to know exactly where thebroadcast cameras are focused. It also hasto have some idea of how each cameralens distorts the image; different brands oflenses vary. It then has to figure out whichcamera’s feed is being displayed to viewersat any moment.

Meanwhile, the system also has toknow exactly where the puck is and howfast it is traveling, and then it has to cre-ate a graphic based on that data and over-lay it onto the video image 60 times a sec-ond. All these things have to be exactlysynchronized, as the cameras are zoom-ing and panning and the puck is travelingat up to 160 km an hour.

Making it work requires puttinginfrared transmitters in each hockeypuck. Calibrating the system proves dif-ficult. It is finally accomplished bydrilling holes in the ice and filling themwith blue dye to enable the system toaccurately calculate locations on the ice.“It took some convincing to be allowed todo that,” recalls Marvin White, nowSportvision’s chief technology officer.

PH

OT

OG

RA

PH

S: R

OB

ER

T H

OU

SE

R

It is “one of the most significant innovations to football cover-

age since instant replay,” pronounces Fred Gaudelli, executive

producer of football for the ABC TV network. And yet it

stemmed from a failure in hockey broadcasts that was rejected

by die-hard fans as a garish gimmick that defaced their game.

33

IEE

ES

PE

CT

RU

M•

No

vember 2003

Technically, the project is a success, andit comes in on budget and on time.

The technology works fine but man-ages to offend even hockey fans with itslack of subtlety. Fox Sports TelevisionGroup (Los Angeles) chooses to highlightthe puck with a large, bright, fuzzy bluespot. A red rocket trail appears, paintedright over the players, when the puck istraveling at high velocity. “It was comic-strip-like, Flash Gordon,” Honey says.Called “FoxTrax,” it makes its debut at the1996 All-Star Game with a series of televi-sion commercials and a huge fanfare.Stacks of newspaper articles debate its pros

and cons, the “Late Show with David Let-terman” spoofs it, and hockey ratings jumpto their highest levels ever.

But serious hockey fans hate it. Afterthree years, pro hockey broadcasts in theUnited States switch networks, and thesystem dies a quiet death. “There are twoways for a product to fail,” Rick Cavallaro,Sportvision’s vice president of productdevelopment, tells Spectrum. “One is forit not to work; the other is for people tosay it works great and we hate it.”

In hindsight, Honey says, the systemwould have been better accepted had itbeen subtler. Better to have placed theimage of a gray disk under the puck—and underneath the players—and shownthe puck’s track only when it was movingfaster than a preset threshold, he says.That track could have been a clean blackline, drawn as if by a drafting tool, making

it clear that it was to provide information,not entertainment. “It’s possible to imag-ine,” Honey says, “that the diehard fanswould have seen that as an advantage.”

Off the ice, onto the field

Meanwhile, unwilling to disband hiscrack team, Honey needs another proj-ect—fast. An idea for a fancy newtelestrator for commentator John Mad-den’s play diagrams goes nowhere, butonce again, Hill has another idea. “Whydon’t you just do the first-down line?” heasks Honey. “It’s clean, it’s simple, andit’s important.” (The telestrator as

described by Honey is, nevertheless,developed later and introduced in 2002.)

This time, though, Murdoch does notbring out the checkbook. Honey and histeam, along with two News Corp. execu-tives, Jerry Gepner and Bill Squadron,spin out a new company, Sportvision, tak-ing rights to use all the patents and otherintellectual property they have from thehockey puck project. In exchange, NewsCorp. gets 10 percent of the company.

The color map conundrum

Drawing a simple first-down line has gotto be much simpler than continuallytracking a puck bouncing around andtraveling at up to 160 km an hour, right?Wrong. While some elements of the first-down line problem are similar to some inthe hockey puck problem—you have toknow where the cameras are pointed at

all times and figure out where in theimage to do your drawing—a number offactors make the seemingly simple task ofdrawing a line actually more difficult.

For one, the line has to be drawn as ifit were under the players, not as an over-lay. Also, the distortion of the televisionlenses becomes more critical—if thepuck trail is a little off, it’s no big deal, butif the yellow line is curved incorrectly, itis immediately apparent next to the realwhite lines on the field. That lens distor-tion changes constantly as the cameraszoom in and out. Complicating all of thisis the fact that, unlike hockey rinks,which are flat, football fields are not—they have a crown down the middle toallow drainage.

But the biggest problem of all is thecolor “keying.”

Color keying is done all the time inbroadcast television and movies. Theclassic example is the weather forecasterin front of a blue screen; the image in theblue screen is later replaced by a videoimage of a weather map, and it looks asif the weather forecaster is in front of themap. This type of keying is simple; theprocessor simply replaces any blue pixelwith the second image. If a pixel is notblue, it doesn’t replace it. When this tech-nology is being used, actors simply donot wear anything blue or parts of themwould seem to disappear.

But a football field is not blue; it ismany shades of green, and the exact colorof the green changes every time a cloudpasses over the field. Some fields havepatches of brown dirt as well. Some play-ers also have green uniforms, or brown-ish ones, or uniforms that becomestained with grass or mud. As a worst-case scenario, the engineers consider theSan Francisco 49ers playing the GreenBay Packers at sunset, after a recent rain:the image will have multiple colors ofbrown dirt, because there is dirt sitting inthe sun, dirt in the shade, dry dirt, andwet dirt. It will have multiple shades ofgreen grass. And it will have the 49erswearing brown pants and the Packers intheir green shirts [see photo, above].

Even with all that green and brown todeal with, the system would have to makesure that the yellow line is never drawnover a player. Determining the key thatwould define what is field (to be drawn

A custom-made colorkey enables the yellowline to appear only overthe green grass, notover the green jerseys.

TELEVISION

34

IEE

ES

PE

CT

RU

M•

No

vem

ber

2003

upon) and what is player (not to be drawnupon) is going to be a challenge.

Cavallaro, the project leader, fears that“in some cases, there just wouldn’t beenough pure color distinction,” resultingin some yellow line appearing incongru-ously on a player’s shirt, say.

So the first thing the team does is de-velop a more sophisticated method of colorkeying than had ever been done before,one that can be redefined quickly as light-ing changes. The group produces a hugetable of color definitions in multiple for-mats, the most common of which is RGB,specifying colors by their red, green, andblue components. An operator can usewhichever set of color definitions worksbest in a particular situation. Proving thata sophisticated enough color key can bedeveloped takes a month; then the teammoves on to solving its other problems.

Each television camera, it turns out,electronically measures the zoom andfocus position of its lens. This informationis, amazingly, not used for much of any-thing until Sportvision taps into it to makesure the yellow line is correctly laid out.But zoom and focus information is notenough; the team also has to add sensorrings under the camera that measure itspan and tilt and the attitude of the tripod.

In the latest version, the solution is acontroller mounted on the tripod headthat collects all this data for each frame ofvideo, synchronized by the camera’s ver-tical sync signal and then modulated ontoan audio frequency before being sentback to the production area via a micro-phone channel.

With the project about halfwaythrough, the team decides to go with yel-low, after considering and rejectingorange, blue, and red. “Yellow isn’t themost obvious choice,” Cavallaro admits.“Orange would make more sense becausethe chain gang [the officials who mark andmeasure first downs] has orange flags. Butorange looks bad and yellow looks good.”

Finally, it’s all done. The technology is first embraced by

ESPN, which introduces it in the fall of1998 and receives an Emmy for technicalinnovation. The introduction is done with-out fanfare, without teaser advertisements.It just appears one Sunday in September,with the intensity of the line turned downfar below what it typically is today; blades

of grass seem to peek through, makingthe line look like yellow chalk on the grass.Viewers are puzzled: is the line really onthe field or not? And that mystery initiallycreates a lot of interest.

Today, Sportvision crews crisscross thecountry during football season. A crewof two typically arrives the day before thegame, along with a small rack of com-puters, about the size of a dormitoryrefrigerator. They first measure the slopeof the field using a laser surveying sys-

tem, fit the rings on each camera, andcalibrate zoom positions. An hour beforethe game starts, the operator chooses thecolor key, based on the colors of the fieldin both shade and sun and those of theplayers’ uniforms. During the game itself,every time a new first down is made, anoperator looks at the screen and clickseither on the ball or on the chain gang;this positions the line.

Meanwhile, sensors on three maincameras continuously send camera posi-tion information down an audio chan-nel. While the video feed is delayed afraction of a second, the computers con-sider all the data, determine whether theline should be drawn in that image, and,if it should, consult the color key todetermine which pixels should beswitched to yellow—and repeats thisprocess 60 times a second.

Viewers almost immediately acceptedthe technology. Today, while Sportvision

creates the lion’s share of virtual first-down lines, others, like PVI VirtualMedia Services LLC, in Lawrenceville,N.J., have gotten into the act.

Next up, NASCAR

Sportvision’s latest conquest is NASCARracing, a sport in which Sportvision relieson a Global Positioning System (GPS)receiver in each racing car to send loca-tion information to a computer systemthat, during a two-second broadcast delay,

generates graphics and other data relatedto each car’s movement and position onthe course, including its speed, brake sta-tus (on or off), and tachometer reading.The technology was first used in a liverace in 2001.

The imaging problem was similar tothat in football, because the televisioncamera lenses have long zoom ranges.This problem had been solved before. Butat its inception, the NASCAR project wasfaced with a conundrum. It was clear thatto get accurate positioning of each car onthe track, a GPS receiver was needed ineach car. But the conventional wisdom ofthe time was that GPS could never workin a racing environment because of theamount of electrical noise generatedinside each car; the extreme multipathinterference due to all the metal nearby asa car raced around a track; the metal fencethat leans out over the track, blocking, ineffect, half the sky; and the fact that the

Sportvision’s technology createsvirtual dashboards in this screenshot from an NBC broadcast andone from InDemand [far right], a pay-per-view service. Also shownare the race car’s number and position on the track and the name of the driver.

35

IEE

ES

PE

CT

RU

M•

No

vember 2003

GPS receivers would have to provideaccurate positioning within centimetersinstead of the more typical meters.

The solution was to give the GPSreceivers a head start in figuring out theirposition by restricting the possibilities toa three-dimensional ribbon of track,measured in advance to within a few cen-timeters of accuracy.

Designing the telemetry, the methodby which the GPS information would besent back to the computer system from

the car, was also a challenge. PreviousNASCAR telemetry systems relied on atransmission control protocol/Internetprotocol (TCP/IP)–like approach ofacknowledge/retry. But the Sportvisionsystem didn’t have time for that; it had toget the information for all the cars, com-pute the graphics, and display them in lessthan the two seconds the live broadcastwas being delayed. So the engineersdesigned a customized telemetry systemthat used spread-spectrum modulation ofthe signals, and assigned each car two pre-determined time slots for transmission sothere would be no signal collisions.

After the design of the NASCAR sys-tem, RACEf/x, was complete, the technol-ogy proven, and the NASCAR authoritiespleased enough with it to mandate its instal-lation in all cars, there was a problem thatnearly stopped the technology in its tracks.

RACEf/x was designed to use lith-ium–manganese dioxide batteries to

power each car’s GPS and telemetrypackage. During the 2001 Watkins Glen(N.Y.) International Race, one batterypack overheated and started a fire in thebattery enclosure, emitting smoke intothe car. (The fire was extinguished with-out injury, though the car did not finish.)

Engineers determined that the bat-teries chosen were not stable under highvibration, and reported this to NASCAR.They then designed and manufacturedreplacement alkaline battery packs that

were much larger and heavier but lessflammable, and had those ready to installin time for the next weekend’s race.NASCAR, which had been mandatingthe use of Sportvision’s technology, gavedrivers a chance to opt out; none did.

Internet in the future

Today, the Sportvision crew is expandingthe NASCAR tracking system to interac-tive television and video games. In thesegames, an actual race is virtually recon-structed with slightly smaller vehicles anda home player’s car is inserted, allowingthe gamer to race against the actual driv-ers. The company has introduced an Inter-net application called PitCommand, inwhich paid subscribers can watch a virtu-al aerial view of the race and view the trackand data for any car they select, run instantreplays at will, and change perspective.

Ultimately, the Internet is where thecompany sees its future.

“When the time comes for people athome to make decisions about how to cus-tomize their viewing experience, they willneed data from the sporting event. And wewill have that data,” Honey says. The com-pany already has been granted over 20patents related to gathering data at anevent and sending it downstream todevices in the home.

But that future may take a while toarrive, and Sportvision isn’t holding itsbreath in anticipation.

The company, founded during thedot-com boom, was initially pressuredby investors to become an Internet com-pany instead of focusing on standardbroadcast television, which was then per-ceived as a technology dinosaur. ButSportvision resisted, arguing that TVbroadcasters have real money. Then,with its trucks and sensors at everysporting event, the company would bewell positioned to dominate interactiveInternet sports if and when that timecame. Today, Sportvision is a solid, ifstill small, privately held business. Itprojects that it will be profitable on anoperating basis in 2004.

Honey meanwhile in October steppeddown as company president to be the nav-igator of a sailing crew attempting to setthe record for the fastest passage aroundthe world. He remains on the board.

“We haven’t made a bazillion doingthis,” Honey told Spectrum, “but it’sthe most fun we’ve ever had. It’s notlike when we were building militarysystems. Everybody gets to see whatwe’re doing, and we’re doing some-thing that is both technically hard andis in good cheer.” •To Probe Further

For information about Sportvision products

and career opportunities, see http://www.

sportvision.com.

As a navigator of racing sailboats, Stan

Honey currently holds both the transat-

lantic and transpacific records. See the site

athttp://www.pegasus.com/Transpac-

philippe-kahnlightsurf-071503c.htm.

For more details on the controversy over

the hockey puck system when it was first

introduced, see http://www.pub.umich.edu/

daily/1996/apr/04-08-96/sports/hockey.

white.column.html and http://slam.canoe.ca/

HockeyNHL97Playoffs/jun2_strachan.html.