■ Recent thinking has it that AI, 25 years ago a uni-fied field with a shared vision of creating intelli-gent machines, has devolved into a loosely con-nected set of distinct specialty areas with littlecommunication or mutual interest between them.To the extent that this is true, it certainly appearsto lessen the value of a centralized AI organizationlike AAAI and of traditional grand-scale AI confer-ences. But, I argue, the consequences are actuallyfar worse: because of the very nature of intelli-gence, the centrifugal force on the field couldthwart the very mission that drives it by leaving noplace for the study of the interaction and synergyof the many coupled components that individual-ly in isolation are not intelligent but, when work-ing together, yield intelligent behavior. To raiseawareness of the need to reintegrate AI, I contem-plate the role of systems integration and the valueand challenge of architecture. Illustrating somereason for optimism, I briefly outline some prom-ising developments in large projects that are help-ing to increase the centripetal force on AI. I con-clude by discussing how it is critical that the fieldfocus its attention back on its original mission, ledby a heavy dose of integrated systems thinking andgrand challenges, and why after its first quartercentury, AAAI is more essential than ever.

One of the privileges afforded the Amer-ican Association for Artificial Intelli-gence (AAAI) president is the chance to

stand in front of the entire membership of theorganization and speak in an unfiltered wayabout whatever he or she has on his or hermind. This is a wonderful opportunity, yet adaunting one. Since one is keenly aware of thecommitment to speak very far in advance, onecan muse about the speech at many odd

moments over a long stretch of time. Thisallows the jotting of notes and the collectionof meandering thoughts over quite a protract-ed period. But because of the sheer length ofadvance-warning time, it encourages one to beexpansive and to note virtually anything onewould like to opine about in a large forum. Inmy case, this freedom led to a great deal of ran-dom thinking and a fairly large pile of notes.But as the time drew near to speak, and Ilooked over what I had written, I found thatthere was almost no coherence to my manyminor brainstorms. There were numerous spe-cific things and a variety of independentresearch directions to consider, but no big pic-ture. Then it occurred to me that this mightactually be symptomatic of a fundamentalproblem that we are facing as a field and thatAAAI is facing as an organization, and that thelack of a strongly unifying force might itself bea worthy theme for the address.

Between 2002 and 2005, I had the privilegeof working at the Defense Advanced ResearchProjects Agency (DARPA) in a position that isuniquely important to the history of AI: I washonored to be able to serve as director of theInformation Processing Technology Office(IPTO). In that role I had the opportunity tomeet a very large variety of people with greatideas in all aspects of artificial intelligence and,more broadly, across all of computer science.While my ability to get into technical depthwas limited by the sheer volume of conversa-tions and visits, the breadth one sees in such aposition is very hard to match in any other.The global perspective accrued through suchextensive interactions with the communityalso afforded me the opportunity to contem-

Articles

WINTER 2006 19Copyright © 2006, American Association for Artificial Intelligence. All rights reserved. ISSN 0738-4602

AAAI Presidential Address

(AA)AIMore than the

Sum of Its Parts

Ronald J. Brachman

AI Magazine Volume 27 Number 4 (2006) (© AAAI)

plate the big picture and, perhaps more impor-tantly (and consonant with the nature of thejob at DARPA), to identify gaps in our nationalcomputing research agenda. It also occurred tome that that perspective was a very specialasset to use in drafting this presidentialaddress.

So, instead of addressing a technical topic indepth or picking on a single new direction—often the fodder for AAAI presidential address-es—I want to raise a broad issue and considersome larger questions regarding the nature ofthe field itself and the role that AAAI as anorganization plays in AI. My hope is to encour-age thinking about some things that I believeare very important to the future of the field asa whole and perhaps to start a dialogue aboutresearch directions and collaborations that inthe end might bring us all back together andallow us to take advantage of the opportunitythat AAAI affords all of us as AI practitioners.

A Wonderful Time to Mark Progress

The year 2005 was a momentous one in artifi-cial intelligence, at least in the United States. Itmarked the 25th anniversary of the foundingof the American Association for Artificial Intel-ligence (celebrated in the Winter 2005 issue ofAI Magazine), and AAAI-05 was our 20th con-ference. The organization was started in 1980in response to vibrant interest in the field,

which back then was served mainly by anInternational Joint Conferences on ArtificialIntelligence (IJCAI) conference held only everytwo years. The first AAAI conference was heldat Stanford University; it was very much aresearch conference, a scientific event that gen-erated a lot of excitement. The conference wassmall and intimate, with few parallel sessions.There were excellent opportunities for us totalk to one another. AAAI-80 gave real sub-stance to the organization, clearly getting AAAIoff on the right foot, and it gave new identityand cohesiveness to the field.

This year—2006—has also been a big year,celebrating the 50th anniversary of the originalmeeting at Dartmouth College, where thename “artificial intelligence” first came intocommon use. Numerous events around theworld, including a celebratory symposium atDartmouth and an AAAI Fellows Symposiumassociated with AAAI-05, have marked thisimportant milestone in the history of the field.

Progress since our first AAAI conference hasbeen substantial. While each year’s results mayhave seemed incremental, when we look backover the entire period we see some truly amaz-ing things. For example, a computer programfinally beat the world’s chess champion. Inhindsight this may no longer look so exciting(purists will say that it was not an “AI” systemthat beat Garry Kasparov but rather a highlyengineered special-purpose machine largelymade possible by Moore’s Law), but it is worthcontemplating from the point of view of1980—or, even more dramatically, from that of1956. Looking forward from back then, nomatter how Deep Blue actually worked, playingchess well was clearly an AI problem—in fact, aclassical one—and our success was historic.More recently, a robotic vehicle from StanfordUniversity, using machine learning technolo-gy, conquered the 2005 DARPA Grand Chal-lenge, successfully managing a course of morethan 140 miles over difficult terrain in lessthan 10 hours without any human interven-tion. By any measure this was an incrediblefeat.

Another notable aspect of AI life over the lastquarter-century was the broad rise of excite-ment and financial investment in the field.That ranged from a wave of startups that lastedinto the 1980s to significant diffusion of ourtechnology and practitioners throughoutworldwide industry. The influence of the AIcommunity in current large and critical com-mercial enterprises—through, for example, textprocessing, speech and language processing,robotics, machine learning, data mining,knowledge management, and a host of appli-

Articles

20 AI MAGAZINE

The First AAAI Conference was Held at Stanford University.

cations of the sort that the Innovative Applica-tions of Artificial Intelligence (IAAI) conferencehas highlighted for years—while accomplishedperhaps with little fanfare, has been undeni-able. Important systems with significant AIcontributions are deployed and operating dailyin virtually every industry. Small robots havesaved lives, both on the battlefield and in diffi-cult search and rescue situations. AI systems areflying in space. And through Internet searchengines, every day AI directly touches millionsand millions of people.

Perhaps the zenith of our field’s popularitywas in 1985, when we held our joint confer-ence with IJCAI in Los Angeles on the Univer-sity of California, Los Angeles (UCLA) campus.That conference had more than 5,000 atten-dees. The excitement in the community waspalpable, and the interest from the outside,both in the commercial sector and in the press,was extraordinary. There was a great deal ofnational and international attention. Forexample, Woody Bledsoe (then president ofAAAI) and I were asked to appear on a nationalradio talk show, where we debated variousaspects of AI. Hector Levesque gave the Com-puters and Thought lecture in a crowdedPauley Pavilion, the home of UCLA basket-ball—quite an exciting experience. We proba-bly haven’t seen anything like that since, but itwas a wonderful time.

Since then, we’ve had an “AI Winter,” with adramatic drop in funding for the field, fol-lowed by many years of reasonable growth andsignificant thaw. One small indicator of thecurrent improved situation, based on our workat DARPA: starting with the fiscal year 2006budget, there has appeared a line item (a “Pro-gram Element”) in the U.S. federal budget thatexplicitly calls out “Cognitive Computing Sys-tems”—very much an AI agenda. This ex pres -sed the government’s intent to fund a signifi-cant budget item directly focused on artificialintelligence research, as well as at least a mod-est suggestion of longevity for the item. Sincethen, several hundred million dollars of fund-ing were approved and spent for this area. Andthis represents money coming from one fund-ing agency—it does not include current andprospective funding from places like theNational Science Foundation (NSF) andNational Institutes of Health (NIH) and themilitary services’ research organizations. AI hasreached the level of explicit high-level lineitems in the U.S. budget.

Finally, there is the undeniable general infil-tration of AI-related ideas into the public con-sciousness. Prior to 1980 AI was an interestingtopic for science-fiction writers and media pro-

ducers, with the HAL 9000 Computer in 2001:A Space Odyssey, Star Trek, and the early StarWars movies, which together brought roboticsand the idea of intelligent androids to Holly-wood. But since then, we’ve seen substantialgrowth in the public vision of interesting pos-sible futures for artificial intelligence, includingthe recent film I, Robot, based loosely on theIsaac Asimov stories, and Steven Spielberg’sdirection of a provocative film that wasexpressly called Artificial Intelligence. Whoamong us, 25 years ago, would ever have imag-ined a Hollywood blockbuster, created by oneof the great directors of our time, with the verytitle, Artificial Intelligence? Quite remarkable.While it is unclear how much the specific storyhad to do with what we do every day in ourresearch labs, nevertheless, there it was, frontand center in international popular culture:what might the far future of AI be?

On the other end of the spectrum—and per-haps more importantly—we have the well-loved Roomba robot vacuum cleaner, from iRo-bot. Through a remarkably modest invention,in a way that we may never have foreseen, AIhas finally begun to enter the homes of tens ofthousands of regular people who have no ideawhat artificial intelligence is about, but whoknow they benefit from what it brings to theirlives.

In virtually every respect, our field has comea long way in the last 25 years.

Centrifugal Intellectual ForceOne of the natural consequences of the growthof interest in AI in the last 25 years has been anexplosive rise in the number of venues inwhich AI researchers can present their work.We have seen a large number of conferencesdevelop, as well as a large number of new pub-lications, in all of the specialized areas that atone point were all considered simply core AI.These include, among others, machine learn-ing, knowledge representation, automated rea-soning, agents, robotics, vision, planning,uncertainty in artificial intelligence, computa-tional linguistics, and data mining and knowl-edge discovery. In fact, as mentioned at AAAI-05, there may be as many as 30 “sisterconferences” that we can draw from for our sis-ter conference report sessions. For quite a longtime in the earlier days of this expansion, inter-est continued in the core conference—in AAAIitself—with people also spending time at theirown specialized area conferences. But overtime this seems to have changed. With strongspecialized program committees to review sub-missions, and a greater concentration of people

Articles

WINTER 2006 21

in the specialized disciplines attending theseconferences, the more narrowly defined meet-ings have become more attractive. If fundingbecomes tight or if people are doing too muchtraveling, AAAI and perhaps IJCAI conferencesare left off of the itinerary; by attending theKnowledge Representation (KR) conference orthe International Conference on MachineLearning (ICML), or the International Confer-ence on Autonomous Agents and MultiagentSystems (AAMAS), many discipline-focusedresearchers are still getting to see most of thecolleagues that they would like to talk to andhear about the latest technical developmentsin their areas.

As a consequence of this, what we hear oftenin AAAI organizational circles are concernsabout the future of our own organization:What is its role? Why are people not coming tothe AAAI conference? Should we bother tohave the conference at all? In fact, if you wereto look back over the last several years’ worthof statements written by candidates nominatedfor AAAI Executive Council positions, youwould probably find a majority who havefocused on the issue of specialization in thefield detracting from AAAI as an organizationand possibly decreasing clarity of the meaningof our central conference. This is a commontheme that has been discussed often in AAAIleadership circles over the last few years. Thereare good reasons for this evolution, and I thinkit is a natural phenomenon; it can be seen inother fields and in some ways it may not besomething to be worried about.

But my concern here—and the main themeof my talk—is around the fact that it may bethe case that in our field—in contrast withmany other fields—the specialization and thekind of centrifugal force pulling people away fromAAAI as an organization and a conference mightactually be having a much more challenging effecton our ability to do the very work of our field.While a natural consequence of the maturity ofthe field is the “loss” of some of our subfields,this specialization and spinning off of areas ofresearch presents us with a deep and funda-mental problem.

We have been seeing great progress alongspecialized lines in each of the areas of AI, andsome have made extraordinary gains over thelast 10 years (machine learning being one ofthe most obvious cases). But in general itappears that while we are getting greater tech-nical depth and greater achievements in ourspecialized fields, we do not seem to be gettingany closer to what people might consider “trueAI”—the goal that caused us to start AAAI in1980 and that gave birth to the field in the1950s and that, to be honest, got most of usvery excited about getting into the field whenwe were even back in high school—the grandvision of building a truly intelligent human-level AI system. As a whole, the field justdoesn’t seem to be making a lot of progress inthat direction, even while we make tremen-dous progress in our specialized areas. In myview, this is a consequence of too much focuson AI-related “components” and not on artifi-cial intelligence itself, which is not simply thesum of its component parts.

Another way to look at this (to pick some-what unfairly on one subfield) is to considerthe fact that, no matter how outstanding ourlearning algorithms get, it’s hard to call any ofthem “intelligent.” The world’s greatest sup-port vector machine is not an intelligent sys-tem that has the robustness, breadth, and com-mon sense that even a three-year old child has—nor will it ever be. In a way, the peculiarity ofour current situation with respect to AI as awhole is exemplified by a simple phrase that iscommonly used in the learning area: we seemto have made a lot of progress in “knowledge-free learning.” Think about that: imagine talk-ing to an average person on the street about AI,trying to explain that we want to build intelli-gent humanlike robots, and that we are makingtremendous progress in “knowledge-free”learning. I suspect that that would be at leastpuzzling for a lay person, if not downrightbizarre. Why would you handicap your systemby expressly discarding anything that it hadlearned in the past or knowledge it had gainedour could infer through other means? Howcould you even have the concept of knowl-edge-free learning in a fully integrated AI sys-tem? Yet this is common terminology, exem-plary of a common point of view.

Similarly, as much as I am personally fond ofwork on the semantic web, not to mention thatmuch of my own technical work was involvedin description logics that provide a foundationfor web ontology languages, no matter how farwe go down that road—no matter how won-derful our description logics become—theywon’t by themselves lead us to artificial intelli-

Articles

22 AI MAGAZINE

… the specialization and the kind of centrifugalforce pulling people away from AAAI as an

organization and a conference might actually behaving a much more challenging effect on our

ability to do the very work of our field.

gence. As with isolated learning algorithms,they may have fantastic applications, they mayhave great intellectual depth, and they mayeven make some people a lot of money, but Ido not believe that such specialized tracks willlead us to AI. The same can be said for virtuallyall of our subdisciplines, for example, machineperception and natural language processing.They may produce wonderful artifacts andgreat scientific breakthroughs, but they don’tseem individually to be leading us to an artifi-cial intelligence. So my concern goes waybeyond the fact that specialized conferencesare drawing attendees away from AAAI confer-ences. The centrifugal acceleration that wehave been seeing, with more and more peoplespending more and more of their concentratedtime in the subareas and at their own confer-ences and in their separate journals, may actu-ally be sabotaging the future of our own field atits very heart.

Intelligence Is a Many-Splendored Thing

It has been hypothesized that whatever intelli-gence is (and we obviously have not been ableto fully define it so far), it is a multidimension-al thing. Natural human intelligence appears tobe multifaceted. “Human-level intelligence”does not lie at the end of the road that focusesonly on machine learning algorithms, nor willit be the result of the world’s greatest naturallanguage processing machinery taken in isola-tion. It is somehow a combination of thesethings and more. Let’s think about this com-monsensically for a moment: imagine that youcould build a system that was fantastic at allthe kinds of reasoning that people do, butwithout the ability to learn. Such a system,even with all of the knowledge in the world putinto it right now, would no doubt appear smartin the first instant, but without the ability toadapt as the world changes and knowledgeneeds to evolve, it will simply become moreand more stupid over time. That is, systemsthat are knowledgeable at one moment, buthave no adaptive capability, ultimately becomestupid because they just stay set in their waysand hold onto beliefs that are increasingly outof touch with reality. Similarly, systems that“learn” without knowing anything—theknowledge-free learning idea—will ultimatelybecome very good test takers, being able to spitback just what their teachers have told them,but they will not be able to apply what theyhave learned to truly novel situations, which infact is the essence of the versatility thathumans exhibit every moment of their lives.

Even an average human can learn some-thing in one context and then later use it in anew one; as a result, the average person is ableto cope with many things that he or she neverspecifically anticipated and is not directly pre-pared for. But you can’t to do that if you justhave a learner that doesn’t know how to store,represent, and reason with the knowledge thatit learns. In a similar fashion, perceptual sys-tems that may appear to be excellent at dealingwith pure data-driven, bottom-up data collec-tion tasks, but that are not guided by cogni-tion, will not work very well in many complexsituations. When the input is multidimension-al and full of interesting tidbits, they will getdistracted by things that are not important orare not material to the task at hand. They maynot recognize that something is an anomaly, oron the flipside, they may end up paying atten-tion to something that is a completely unim-portant anomaly. When you look at humanand animal systems, perception is guided verystrongly by mechanisms in the higher parts ofthe brain in some magical way that we current-ly don’t understand, but that guidance is verycritical. Despite the fact that our sensors areindependently quite competent, they are notjust data-driven transducers doing blind pat-tern recognition—they are much more effec-tive when working together and, in fact, whenguided by common underlying cognitivemechanisms that can synthesize their variousoutputs and use expectations to cut throughnoisy, distracting data.

Note that success or failure is not a matter ofthe quality of the individual components. Weare able to use flawed parts—our vision andhearing systems are less than ideal, our naturallanguage generation systems are imperfect, ourjudgment is flawed—and yet, nevertheless, as awhole, as something that in a way is more thanthe sum of its parts, we do very well gettingaround in an unpredictable world that throwsthings at us that we can never anticipate. So,even the ultimate end products of each of ourspecialized disciplines done perfectly well arenot likely to yield an intelligent system with-out us doing something fundamentally differ-ent. This is perhaps the key point of this trea-tise: AI work done in separate subdisciplines—even that resulting in practically flawless com-ponents—is not itself “AI.” We must considerthe integration and synergies of componentsin an overall system to really approach someform of artificial intelligence.

Now the idea that intelligence is actuallybuilt out of many parts and is not a single,monolithic thing is not a new idea. Many peo-ple have talked about this over the years. One

Articles

WINTER 2006 23

of the things that we did during my recenttenure at DARPA was to look at ways to testintelligent systems that have multifacetedcapabilities. It occurred to us—and I want tothank both Dave Gunning and Paul Cohen forthis idea—that we may want to test intelli-gence skills and capabilities in machines likewe do very broad athletic abilities in humans.Someone might be the world’s greatest shot-putter, but world-leading performance in theshot put alone doesn’t seem to be indicative ofwhether or not the person is, in fact, a fantasticathlete. They might be the physical equivalentof an idiot savant and have only a single skillthat doesn’t translate well into any other formof athletic performance. To determine who isthe best overall athlete—the most “athletic”—we typically use the decathlon. We run a set ofevents that themselves aren’t connected in anyobvious way (throwing the javelin may havenothing to do on the surface with running hur-dles), but nevertheless, there are muscles andcoordination and general athletic conditioningthat somehow support all of these. Therefore,by testing a handful of different skills, we cometo the belief that someone who may not evenbe best in any individual event, but somehowis best in the aggregate, is truly the world’sgreatest athlete, exemplifying what we consid-er to be athleticism. It may be that at themoment, given our crude understanding ofnatural intelligence, this is perhaps the best (ormaybe the only) way to test intelligence. Being“intelligent” may be the same kind of amor-phous mix that being “athletic” is, or alterna-tively, given our poor understanding of it, amultifaceted test may be the only way toapproach the measurement of intelligence.

Flying in FormationMany others, largely from the psychology com-munity, have talked about intelligence as amultifaceted thing, and there are “multipleintelligence” theories. Similarly, some in AI (forexample, Marvin Minsky in Society of Mind(New York: Simon & Schuster, 1985) and subse-quent writings) have talked about intelligencebeing multifaceted. Let us grant for a momentthat indeed intelligence is not a single, mono-lithic capability. There is one more criticalpoint: intelligence is not created by just mixingtogether the individual facets. Indeed, intelli-gence takes many different individual, specialcapabilities—like reading and thinking andlearning and seeing and hearing and talking—but, even if we have all of those pieces eachdone extraordinarily well, if we just toss themtogether we are not likely to get an intelligentsystem. My sense is that intelligent systems arenot simply the mere compositional sum oftheir parts. Individual senses and cognitiveabilities don’t exist separate from one anotherand don’t simply drop their independent con-clusions into a centralized data store that theycan all read from independently (the appeal ofblackboard architectures notwithstanding). Ifyou start to examine what people actually do,you see very deep and complex connectionsbetween these different capabilities.

Unfortunately, these relationships are thethings that are often missed in our specializedconferences because in order to analyze andunderstand them, specialists in one area wouldneed to reach out to communities that havetheir own specialized conferences that areintellectually quite far away. For example, nat-ural language understanding in humans,which is often addressed in computational lin-guistics conferences, really does seem toinvolve understanding—as tautologous as thatsounds, the focus of most natural languageprocessing work these days is not on under-standing, but on doing as much as one canwithout positing an understanding component.Understanding involves world knowledge, rea-soning, and the ability to do inference and seethe consequences of things that you learn andapply them to new situations, which soon getsquite out of the realm of what you typically seeat a computational linguistics conference. As Ihave said several times now, learning producesknowledge as an output and that knowledgeneeds to be used in reasoning components;without this, reasoning systems can chug alongand do work with what they’ve got, but theyend up being pretty stupid unless they can takeadvantage of newly generated knowledge fromlearning mechanisms. Additionally, real-world

Articles

24 AI MAGAZINE

learning in the kind of normal everyday situa-tions in which we find ourselves is heavilydependent on what we already know, orexpressly seek to learn because of other thingswe are thinking about. Further, as I have alsosaid, perception doesn’t really operate inde-pendent of cognitive state.

As we examine more and more of the piecesof an intelligent system—the separate events inthe cognitive decathlon if you will—we findthat we still need to account for the magicalway in which they all have to fit together inorder to create something that is more than thesum of its parts. Perhaps the best way to getthis point across is to quote someone I used towork with (Dado Vrsalovic), who, in a differentbut perhaps comparable context, was fond ofsaying the following: “100,000 spare parts fly-ing in close formation is not the same as an air-plane.” I am not sure how you launch 100,000spare parts to fly in close formation, but imag-ine taking all the parts in a Boeing 777 and put-ting them really, really close to one another,but not totally integrated in the way we like tosee our airplanes integrated when we fly onthem. Now when you launch this thing, it isn’tgoing to stay up in the air very well and, in fact,it isn’t really an airplane. There is no architec-ture—an integration that needs to be taken intoaccount to take the parts and turn the wholeinto something that really serves a broaderfunction.

So somehow, even in the decathlon, whilewe test skills separately, they are all being doneby the same person with much overlap in thesubsystems that are used in the various events.Running hurdles and doing the long jumpclearly use related leg muscles; throwing thejavelin and pole-vaulting both use a runningstart and significant arm action. Imaginethough, a field that had emerging specializedjournals and conferences that focused totallyon the mechanics of javelin-throwing. Mem-bers of the Hurdle Society and attendees at theInternational Pole-Vaulting Conference wouldno doubt be in the dark about what came to bediscussed at the javelin meetings—and in factmight declare those fields to be so far fromtheir interests that they were no longer inter-ested in attending general athletics conferenceswhere there would be boring and irrelevantpapers on javelin-throwing, sprinting, andhigh-jumping. If the experts in those special-ized areas were interested only in creatingmachines that could star in their individualevents, they might be able to get away withthis separatism. Even then, of course, thejavelin folks might miss an important develop-ment in the discus-throwing literature that

could provide critical insight and might evenhelp their subfield take a quantum leap ahead.But much more importantly, if the genesis ofthe separate subareas was a once-exciting fieldthat had originally aspired to create a generalartificial athlete, able to compete competentlyin many different events—including novel ath-letic competitions that had not previouslybeen on the program—then the subareas goingtheir own separate ways would fly directly inthe face of the original overarching goal. Itseems to me that the way to build a competi-tive artificial decathlete is not to lose the focuson the athlete as a whole by working inde-pendently on the subsystems needed for spe-cialized events. Certainly the leg and arm sys-tems would need to be studied in detail andemulated in machinery; but their interactionsto produce the rhythm of a long-distance run-ner or the coordination necessary to achieve asuccessful pole vault would need to be a main-stream concern of the field as a whole. Withoutattempting to study and build a whole bodywith integrated musculature, where would bal-ance, strength, endurance, and coordinationcome from?

Now this is admittedly a naïve view, and Iam no physiologist, but I would suspect that ifyou looked closely at how athletic skills areimplemented in natural systems, you wouldreally need to pay attention to the cross-cou-pling of different muscle and skeletal compo-nents, not to mention the overall cardiovascu-lar infrastructure—and the key point here isthat it is the same thing with intelligent sys-tems. “Learning” isn’t really learning in intelli-gent systems until it is tightly connected withreasoning. Perception doesn’t mean muchunless it is coordinated with cognition. Lan-guage essentially involves thought and under-standing.

The implications for AI should be obvious. Ifwe simply keep our separate components fly-ing in close formation, we won’t ever really getan AI airplane. And if we build machines thatwin races in specialized areas, we won’t get adecathlon winner or an athlete capable of suc-ceeding at a newly invented event. We need tothink about how all the parts of an artificialintelligence should work together, and howthey need to be connected, or we simply won’tget one. So to my mind, what we really need todo as a field is to spend a lot more time focus-ing on comprehensive, totally integrated sys-tems. It doesn’t look like we are going to getthere from here if we keep going in the direc-tion we are going, especially given the centrifu-gal force that is drawing people away from oneanother into their separate subfields.

Articles

WINTER 2006 25

Mutual InfluencesNow, why hasn’t there been a lot of work onintegrated systems? In fact there have beensome wonderful role model projects, but byand large, if you look at AI, most of it isexplored in specialized venues or in narrow,specialized sessions at conferences. As a resultwe don’t spend a lot of time talking aboutpulling the pieces together and creating themagic glue that turns them into intelligent sys-tems rather than just sets of pieces. First andforemost, it is clear that building full, integrat-ed AI systems is extremely difficult. It takes alot of talent with a lot of breadth. It requiressizable teams, software engineering skills, andeven project management skills—the kinds ofskills that are not deciding factors in hiringuniversity professors. Graduate students arereally not taught this, and it is often the casethat many engineering and architecture con-siderations are deemed “below” some scien-tists. It is notoriously difficult to get AI systemspapers published in key places.

Nevertheless, these things are absolutely crit-ical to our ultimate success. As a field we don’tnecessarily have a lot of these skills around.Building large-scale integrated systems is really,really difficult. It is also very messy just becauseof the essence of what’s involved. Everything

in the system can ultimately affect everythingelse. And that makes building and debuggingthe system very hard.

Three Focal PointsBefore I get into specifics about some role mod-el efforts, I want to talk briefly about three rel-atively underdiscussed issues that I believe areimportant. One is the notion of architecture.Another is what we might call knowledge-richlearning—somewhat the opposite of the knowl-edge-free learning that I mentioned earlier. Thethird is the nasty, unpopular topic of metricsand evaluation.

ArchitectureWe are all familiar with architectures in AI sys-tems. We often see block-and-arrow diagramsin papers, and in fact, there is a pretty substan-tial amount of work largely driven from thepsychological side of things around what arecalled “cognitive architectures.” In my view,however, there is still significant room for workin this area as applied to integrated AI systems.The cognitive architecture literature has tendedto focus strongly on psychological validity,matching what goes on in the human brain.Since our concern in AI is not just the replica-tion or understanding of humans, but in fact abroader space of possible minds (thanks toAaron Sloman, a philosopher and long-timeresearcher in the field, for that description), Ithink we need to expand our thinking aboutarchitectures that are adequate to support inte-grated AI systems.

A potentially fruitful question to ask iswhether we can learn from architecture workin other disciplines. If you have ever beeninvolved with a large-scale software systemeffort, you’ll know that architecture and sys-tems engineering are critical parts of the equa-tion. These arts are practiced by specialists withsubstantial technical education in the area, andhard-won experience is crucial to success.Architectures are not just boxes and arrows puttogether by the seat of the pants. System archi-tecture as a discipline has its own principles,and we may be able to learn a lot by taking abroad look at the literature on software systemsarchitecture, hardware architecture, and evenbuilding architecture. It would be interesting tosee how well-known principles of good archi-tecture might apply to AI.

For example, here are a few principles (cour-tesy of Gregg Vesonder) that you might learnabout good architecture if you were taking aclass in software engineering:

A good architecture is one that is amenable to

Articles

26 AI MAGAZINE

implementation of a skeletal system with min-imum functionality overall in order to test thecommunication paths between the compo-nents. One way to get started is to build stubsfor all of the parts of the system without mak-ing them rich and fully functional; this allowsyou to figure out whether the APIs and thecommunication paths are right before youinvest in full implementation of the parts.

A good architecture would feature well-definedmodules with functional responsibilities allo-cated on the principles of information-hidingand separation of concerns with well-definedinterfaces.

A good architecture should feature a very smallnumber of interaction patterns. Modules thatproduce data should remain separate frommodules that consume data. Good architec-tures avoid doing the same thing in differentways and in different places.

Principles like these are familiar to computerscientists, but the kind of formal architecturereviews necessary to successful large softwareprojects are not common in AI work. I think wehave some important things to learn from theway architecture discipline is applied in moremainstream software projects.

On the other hand, we also need to considerhow building an intelligent system might bedifferent from building a more conventionalsoftware system. It certainly appears to be thecase in the human brain that the architecturedoesn’t look like it does when we construct sys-tems with simple components and do informa-tion-hiding and separation of tasks and onlydo one thing each in one place in a system.Neural architectures hypothesized by brain sci-entists look quite different from those of con-ventional software systems. One critical ele-ment here is the fact that the one role modelthat exists in the world that we’ve been lookingto emulate was not a designed artifact; thebrain evolved over millennia, and in fact, whatyou get with an evolutionary approach is verydifferent and potentially much messier thansomething you would do with a very goodengineering design. So this leaves us with aninteresting conundrum: as we build large inte-grated AI systems, it would make sense to fol-low conventional architecture discipline, butwe generally look for inspiration to somethingthat evolved in an organic way and whose cur-rently discernible architecture is very differentfrom that of the systems we typically design.

Knowledge and Learning Humans learn, it is clear, in many differentways. We learn by being taught, we learn by tri-al and error, we learn by mental simulation andexplanation, we learn by reading, and we learn

in a host of other ways that may or may not allbe implemented differently. When we thinkabout learning, though, the goal is to produceknowledge or resources that we can use in deal-ing with novel situations later in life. So whatwe tried to do when I was at DARPA, and whatI think is important for the field in general, wasto move back to the more traditional AI view oflearning. Here I’m not talking about MachineLearning with a capital “M” and “L” as a sub-field, but good old learning as done in core AIsystems, where, for example, we might havesystems as capable as humans are of learningfrom very few examples (including just a singleone). Learning also is profoundly important ingetting around in life when we use it in whatwe might call a “transfer” way—for example,absorbing things we’re taught in a classroomand actually finding out many years later thatthey apply in unanticipated ways to real life.We learn things in one context that work outto be very useful in a different time and con-text.

Another phenomenon in learning is “lad-dered learning” or what Dan Oblinger hascalled “bootstrap learning,” where learningworks best when principles are presented in astructured approach, so you learn one “rung”at a time. We know that when things are pre-sented to us in school in the right order, we canactually learn incredibly complex concepts,whereas if we just jumped to them right away,we could never learn them. So we know peda-gogically it is very important to ladder things,that is, to master one rung before we jump tothe next rung, yet that’s not how we are cur-rently approaching research in machine learn-ing.

Consider also the difference between bot-tom-up, or data-driven knowledge acquisition,and top-down, or goal-driven simulation andacquisition of knowledge. These two approach-es come together in learning. For example,when reading you don’t just read the words;rather, you read with expectations about whatwill come next. You can complete a sentenceeven if you hide the last word in the sentenceand your interpretation of the words you see isinfluenced by what you’ve just read. Your for-ward-looking expectations influence how youinterpret the next sentence, and what topicsmight be presented in the next paragraph. Thisimplies a potentially complicated architectureunderlying reading that we ought to include inour learning research.

A final issue on the subject of learning sys-tems is how to tell whether a system haslearned and whether its learning can beimproved.

Articles

WINTER 2006 27

All in all, these notions of learning are quitedifferent from the main current thrust ofmachine learning research. This is ironic in away, since as far back as the first AAAI confer-ence these concepts were posited as the basicfoundations for learning.

A Word on EvaluationBefore working in the government, I, like manyof us in the field, really hated the idea of met-rics. It was too difficult to figure out quantita-tive metrics for knowledge representation andreasoning or concept learning, and when youdid, they tended to distract from the main lineof work. But since I started working with otherson larger integrated AI projects, and securingfunding for the field, I have found that evalua-tion is an absolutely critical ingredient of ourresearch program. I believe metrics and evalua-

tions are requirements for the future of thefield, both for our own sake and also for thesake of those who oversee our funding. In par-ticular, in the United States it is very importantto show Congress that we are actually makingforward progress—even if our goals are 20, 50,or 100 years in the future. It no longer matterswhat our subjective description is of how muchbetter things are than they were five years ago.Rather, it is important to understand in a meas-urable and repeatable way where we are head-ing and how we will get there.

Now this is very challenging. It’s hardenough to apply meaningful evaluation de -signs to AI component technologies, just giventhe nature of research. But the difficulty is mag-nified when you take to heart the message thatwe need more work on integrated, comprehen-sive AI systems. How will we evaluate intelli-gence in a machine when we don’t even knowhow to measure intelligence in people? But thischallenge alone should prod us as a scientificcommunity to come up with new ideas andapproaches to the issue of evaluation and met-rics.

Some Role ModelsI am encouraged to note that there are somegood examples of the integrated, systemsapproach to AI. I want to highlight briefly acouple of projects that I think could becomerole models. One involves work in the area ofspace exploration at the National Aeronauticsand Space Administration (NASA) and at theJet Propulsion Laboratory (JPL). The other is aDARPA project called the Personalized Assis-tant That Learns.

One of the most exciting premises of NASA-style space exploration projects is the fact thatthe environmental constraints and missiongoals are such that AI must be a fundamentalingredient. Simply put, you can’t teleoperate aremote vehicle in dangerous or rapidly chang-ing circumstances when there is a 30-second (orworse) radio transmission lag. In these situa-tions the vehicle has to care for itself in a fun-damental way. Dan Clancy presented an invit-ed talk at AAAI-04 in which he talked about thevehicles we are sending to Mars and deep space.Dan described how these machines monitorand maintain their health on a constant basis.They may get hit by meteorites, they may expe-rience power failures, they may pick up dust inmoving parts, and they may have faulty elec-tronics—a host of things that require real-timeobservation and response by the vehicle itself,because there may not be time to get a signalback to earth before it’s too late.

Articles

28 AI MAGAZINE

Photo courtesy, NASA.Deep Space One Encounters an Asteroid.

In addition to keeping track of their ownhealth and other prosaic things, these vehiclesneed to assure the success of their science mis-sions, following plans for collecting data andgetting them back to earth—and changingthose plans on the fly. Here we have a set ofinteresting constraints from a nasty set of real-world problems where AI—integrated AI—isfundamentally necessary. There are some reallyfantastic projects completed or imagined, rang-ing from the remote agent mission launchedon the Deep Space One probe a few years agoto something called the “Autonomous Science-craft Experiment,” which has been running ona continuous basis for some time now. It isclear in this kind of application that planning,execution, perception, and action are integrat-ed and multilayered. Further, because subsys-tems can override each other and opportunitiesfor conflict abound, the architecture of theintegrated system and how it carries out itsintelligent activity are of absolutely fundamen-tal concern. You can’t send a great camera, agreat pattern recognition system, a great set ofwheels and motor controls, and a host of otherseparate parts into space and expect those“spare parts” flying in formation to carry outthe mission. They must be deeply integrated.One has to think in advance about all criticaland dependent interactions between the partsof the system. In other words, the architectureis the overriding concern. And it is fantastic tosee deployed AI systems of the sort that we allknow and love actually running in real time,live on $100,000,000 NASA space missions.

At JPL, Steve Chien and others have pointedout how these issues come into play in thedesign and implementation of their systems.Steve notes that having multilayered systems,coupled with the fact that there are differentpriorities at different times, means that weneed to worry about redundancy, safety checks,real-time considerations, priorities, and con-stant change and adaptation of plans based onthe real world’s effects on the mission. And asif that were not hard enough, systems likethese can’t really be fully tested until they arelaunched. To run a realistic test on earth to sim-ulate what the mission would be like in spacefor four months or five years is totally imprac-tical, if even possible to design. And so we havesome very interesting challenges to face whenwe build a true AI system that’s going to bedeployed in a real-world, real-time, mission-critical environment.

The second example involves some majorefforts under the Personalized Assistant thatLearns Program at DARPA. One of them is SRIInternational’s Cognitive Agent that Learns

and Organizes (CALO)—an artificial secretary,one that aspires to do virtually all of the kindsof things that a good human secretary can doin the office. What CALO is trying to emulateis not a Ph.D-level physicist, but rather, theseemingly mundane tracking, learning, andreminding aspects of a good secretary who canadapt to real-world circumstances, improveover time, become personalized to the personhe or she is supporting, and take into accountthe many small things that really make every-day life challenging. This is of course a very dif-ferent context than putting a rover on Mars,but equally challenging and exciting from acore AI perspective. The CALO team has iden-tified six very high-level areas of competencyin which the system must learn and succeed:organizing and managing information, prepar-ing information products, observing and man-aging interactions, scheduling and organizingin time, monitoring and managing tasks, andacquiring and allocating resources. The multi-institutional team has been working for severalyears now on a very wide variety of AI subjects,trying to put this all together into an integratedsystem.

To build a CALO assistant, SRI Internationalhas served as the systems integrator for a num-ber of technology specialists. Importantly,from the very beginning SRI instituted a sys-tems approach that required, among otherthings, software engineering discipline, a chiefarchitect, and a release schedule for softwarethat will be tested as a system and used as a pro-totype in DARPA’s evaluations. Of course, agreat deal of fundamental research work forthis project is being tackled in many wonderfuluniversities. But CALO is more than the sum ofthese individual research efforts. It starts withan architecture and a design approach thatbrings all of these pieces together in an inte-grated AI system. The architecture allows spe-cialized learning methods, created by individ-ual scientists at different institutions, to beinserted by using common APIs and built fromcommon core learning algorithms. These mod-ules interact in a well thought out, well-engi-neered way using a knowledge base, reasoningmethods, perceptual apparatus, and otherthings that go into a complete integrated AIsystem. It is the architecture for bringing thesethings together where they can learn from oneanother, interoperate, and deal with inconsis-tencies that is the essence of this project, andone of its greatest challenges.

The results so far are very promising. Afterthe first year of the project, the SRI team devel-oped a flexible, true AI architecture with manycomponents that could interact in at least a

Articles

WINTER 2006 29

minimal way to accomplish a very wide varietyof things—ranging from moderating meetingactivity and taking notes, to preparing briefingpackets that you can take in advance to a meet-ing. There are also obvious things such as help-ing with putting mail in folders and dealingwith calendars, and so on—the kinds of thingsyou would expect from an office assistant. Butthe exciting thing to me about this project isthe deep integration between the many com-ponents and the different types of capabilitiesthat CALO brings to the table. Tom Dietterichrecently noted some specific ways in which theCALO model has driven new types of research,outlining to me some really interesting novelchallenges to people doing work in machinelearning. In fact, one of the more excitingthings about this project is the way thatresearch in machine learning has been affectedby the requirement of building this integratedsystem. One of Dietterich’s observations is thefact that learning and reasoning must be com-bined in a tight fashion in CALO. This meansthat, especially when you’ve got multiplelearning systems, when a prediction is madeand the world either verifies or refutes that pre-diction, or the user provides additional feed-back (“No, that was the wrong document togive me for that meeting”), the system mustkeep track of the new knowledge and assigncredit to the component or set of componentsthat collaborated to make that recommenda-tion. But there may not be an obvious pathback, and in fact, different components cancollaborate to come up with a recommenda-

tion, making it very difficult to assign credit. Sohere is an important, somewhat novel problemin machine learning that might not have sur-faced if the focus were simply on a boostingalgorithm or support vector machine. Similar-ly, this kind of integrated, long-lived systemneeds to formulate its own learning problems.Because it is running autonomously and is try-ing to be helpful, and because it knows it needsto improve, it needs a plan for acquiring, in anunobtrusive fashion, a set of labeled trainingexamples and to evoke feedback from the userwithout being too annoying. These are thingsthat people just do naturally; CALO has to fig-ure out how to do this using deep interactionbetween a planning component, perceptionand action, learning, and reasoning technolo-gy that are all part of a complex, always-run-ning, always-evolving system.

Not too long ago, Dietterich observed to methat in working on CALO, he was now talkingwith people he hasn’t spoken with in 25 years.In 1980 at the first AAAI conference, we wereall working on the same thing—it was AI.While we were individually focused on differ-ent pieces, we were at the same time workingtogether and communicating. Twenty yearsago, learning people didn’t totally divorcethemselves from knowledge representationpeople, and natural language people didn’t gooff and spend their time at their own confer-ences. Today, projects like CALO and theNASA-related projects have rekindled the kindof collaboration and communication that wehad at the beginning. I hope this is a harbingerof the future—a way to bring the field backtogether and help us develop true artificialintelligence systems.

Versatility, Purposeful Perception, and Doing Something Reasonable

One of the historic frustrations in our field isthat when we have built something that webelieve is a successful example of AI, as it getspress and gets analyzed, it seems no longer tobe AI. Instead, it becomes some form of engi-neering or just a tour de force. I believe thatthis is not really a great mystery, but that weusually bring it on ourselves by solving special-ized problems with components of intelligence,not complete, integrated intelligent systems.Deep Blue is a classic example: what succeededin the very specialized domain of chess wasn’tan AI system embodying general intelligencewith perception, knowledge, reasoning andlearning, but a set of very special purpose algo-rithms that allowed the system to succeed at a

Articles

30 AI MAGAZINE

Illustration courtesy, SRI International.

CALO: Cognitive Assistant that Learns and Organizes.

specialized task. We see this all the time withthe specialized tasks that AI addresses. (It’s fairto say that chess is such a specialized problemthat in the end we discovered it didn’t reallyneed general intelligence. But if so, we certain-ly need to find some challenge problems forthe field that do need true intelligence.) So weneed to ask the question: “what really is AI andwhat is intelligence about?”

To my mind the key word may be versatility.(I’d like to thank Rich Doyle for first getting meto think about versatile systems when I was atDARPA.) Humans are versatile: they can takeon new missions all the time; they can learn todo new things. Nils Nilsson, in a paper talkingabout human-level AI and grand challenges forthe field, talks about educable, or what he calls“habile,” systems. Versatile or habile systemsembody the real value that true intelligencebrings to the table—that is, the ability to doalmost anything, maybe not brilliantly or won-derfully, but at least adequately well, and to betrainable to do new, previously unanticipatedthings. I think we need to pay more attentionto this notion of versatility in our field. Whenthe Rover completes its planned mission onMars and yet is still functioning, it would bewonderful if it could adapt to a new mission—be “re-purposed,” if you will—merely by usexplaining to it what was needed or educatingit didactically on a different class of missions. Ahuman could easily do that. However, this isalmost impossible to do with a classically struc-tured NASA spacecraft because the mission isbuilt into the architecture of the system.

General intelligence, or real AI, would allowus to describe, in a declarative way, a brandnew mission and a recommended approach tothat mission that the system would understandand could then take on. In fact, one of the ear-liest thoughts on a true artificially intelligentsystem was McCarthy’s proposal of an “advicetaker,” back in the late 1950s.1 While we’venever really built such a system, McCarthy’soriginal idea remains much more about true AIthan most of the “specialized AI” systems wehave built since then.

Another area that I believe is well worthy ofmore attention is what Manuela Veloso andothers have called “purposeful perception.”This is not passive absorption and purely bot-tom-up processing of sensor inputs, but ratherperception guided by and influenced by high-er-level cognitive processes and current needs.What in the perceptual field needs payingattention to, and how sensor inputs should beinterpreted, is clearly driven at least partially ina top-down way in natural systems. This kindof architecture is worth understanding and

emulating. In the CALO project, for example,the perception subsystem should be influencedin an appropriate way by the overall currentpriorities, upcoming tasks, things that havebeen learned, and similar cognitive aspects ofbeing intelligent.

Finally, a great mystery of naturally intelli-gent systems is that when presented with newsituations, we are almost always capable ofdoing something reasonable. Our responsesmay not be ideal, but we don’t simply go cata-tonic when we are faced with a new situation.We, generally speaking, muddle our waythrough and most often arrive at an adequateoutcome. Afterwards we can reflect on whatwe’ve done and learn from our experience anddo things better the second time. This is a con-cept we can all understand, at least informally,but reasonableness hasn’t been thought abouta lot in AI. Why is this? As with the other issuesjust discussed, I believe this escapes seriousstudy because it is another virtue of an inte-grated, comprehensive intelligent system, notjust a collection of spare parts hanging aroundtogether. And we’re still paying almost totalattention to the spare parts. Versatility, pur-poseful perception, and reasonableness are allexemplary characteristics of real, honest-to-goodness intelligent systems, and all seem torequire the kind of integration that I have beentalking about.

Measuring ProgressOnce we build an intelligent system, how do weknow if it’s any good? How do we tell if it isdoing well, especially if its behavior is kind ofreasonable, it is not great at any one thing, andevery piece of the system influences every otherpiece? How do we actually measure progress?Again, the CALO experience points in a prom-ising direction. For the project, we hired a full-fledged, independent evaluation team to iden-tify the task challenges for the system and thendecide on the right measurements to be appliedyear after year—thus giving essentially the sametest each year, but without the system beingable to cheat because it saw the test the yearbefore. In a way, the project has developed anSAT-like test for the system by creating a set ofparameterized questions, or question templates,that can be varied from year to year, but thatrequire precisely the same capabilities and intel-ligence to answer each year. For CALO, we canask about a new set of meetings or tasks for theoffice assistant each year but use the same test-ing to end up with a quantitative score of per-formance and track progress from one year tothe next. What really matters is whether we can

Articles

WINTER 2006 31

show that a system can learn, that it will ulti-mately do much better than an engineered sys-tem because its knowledge growth and itsimprovement in skills just outstrips what thedesign team can do as programmers.

AI Grand ChallengesIf our goal is to build a machine that has someform of true, versatile intelligence, and yet thefield is not progressing much in the right direc-tion, how can we stimulate more of the rightkind of work? Certainly having a system “situ-ated” and interacting with the real world is animportant element, partially because whateverAI system we build will need to work in a realcontext. But there is more. The real world ismessy and unpredictable, so I think we need tospend more time building systems that weexperiment with in the wild, not just in thelab—we need our AI systems out there dealingwith the messiness of the world, dealing withnoise and misdirection and things that we wishwe generally didn’t have to deal with.

Another important element of a refocusingof the field is to move AI to be thought of moreas a system science—not a handful of discon-nected subdisciplines, but an overarching viewof how those disciplines and the resulting tech-nologies they create interrelate. In fact, I thinkwe need to find room in our conferences for allaspects of computer science that can con-tribute to and benefit from the needs of com-plex AI systems. How can we do this, given ourcurrent commitments to our universities andcompanies, our students, and our careers,which tend to demand simple allegiance tofields defined by existing conferences and jour-nals? What kind of game-changing eventmight create the motivation we need toleapfrog to a new paradigm? I suggest that welaunch a new series of “grand challenges.”Such challenges, like President Kennedy’s boldpromise in 1961 to put a man on the moon(and return him safely) by 1969, can grab theattention of a broad group of people. Theypresent hard problems to be solved—hardenough to require lots of work by lots of peo-ple, and when defined right, they move fieldsin significant leaps in the right direction. Thereally good ones over the years are interestingbecause they have really demanded the kind ofintegration that I have been advocating. In myview, with appropriate sets of challenges, wemight actually generate a tremendous amountof new excitement in the field of AI systems,which doesn’t really have its own separate sub-area or conference.

Criteria for Successful Grand ChallengesThere are aspects of creating such challengesthat require some serious thought. First andforemost, we need a problem that demands aclear and compelling demonstration of AI. Suchproblems, as I’ve intimated above, will need tostress versatility, adaptation, learning, reason-ing, and the interaction of many aspects ofintelligence. Next, a successful challenge can-not be gameable in a way that somebody couldwin the prize with very specialized engineeringthat doesn’t get us closer to an AI system. Thisrequires extensive thinking about the groundrules for the challenge. Another thing that isreally critical to a successful grand challenge isthat the measurements—the success criteria—are well-defined, easy to articulate, and under-standable to any lay person who might partici-pate. For example, the success criterion for theDARPA Grand Challenge autonomous vehiclecompetition in the desert was simple: travel onthe ground along a prespecified route frompoint A to point B in less than 10 hours, withno human intervention—period. You don’thave to say much more—people know whatyou want and what it takes to succeed.

Another key aspect of a good grand chal-lenge is to make the problem hard enoughsuch that it is not clear that it can be solvedand to allow partial successes to lift the generalstate of the art each year. Here, failures alongthe way can point the direction we should goto improve things—a critical aspect of goodgrand challenges. (Thanks to Paul Cohen forthe notion of failure being “diagnostic” as anaspect of a good grand challenge.)

Of course, the challenge has to be ambitiousand visionary, like putting a man on the moonor cracking the human genome or beating theworld’s chess champion, but it can’t be unreal-istic. As much as I would like to put out thechallenge of building a Star Trek–like trans-porter, I don’t believe such a thing would bepossible in my lifetime (if ever). People will getfrustrated if they don’t see their way clear topossibly solving the problem and winning theprize. We can debate what the right time hori-zon is, but we certainly want to have solid faiththat someone will succeed in somewherebetween, say, 5 and 15 years, with reasonabledoubt about solving it in the first 5 years.That’s what makes it really fun.

A successful AI grand challenge would needto be compelling to the general public, gener-ating good press and building general excite-ment. The public has to find it compelling andexciting enough to follow the stories in thepress and be rooting for the AI researchers to

Articles

32 AI MAGAZINE

succeed. Finally, a successful challenge has tobe strongly motivating for the research com-munity, one that gets specialists talking witheach other and exploring new collaborativepaths to success, asking new questions andquestioning past beliefs.

The aforementioned criteria should beapplied rigorously to grand challenge sugges-tions. Hopefully we will end up with not just alaundry list of possibilities but rather a com-pelling, exciting quest that will attractresearchers and lay people to find a solution. Ibelieve AAAI could play a very important parthere.

Some ExamplesWhat specific problems might qualify as high-quality AI Grand Challenges? Back when I wasat DARPA we held a workshop to identify,debate, and refine some ideas. Here, briefly, aresome of the ideas that were still on the board atthe end of the workshop.

Truly read and understand a book. Here, thechallenge would be to build a system that canread a book (a textbook, say) and successfullyanswer the questions at the end of a chapter or,better yet, sequential chapters. To make such achallenge compelling, it probably has to be theequivalent of what a student does in a highschool science course, with some details andground rules appropriate to computer pro-grams. But pretty much, that’s it. We wouldhave to think through the ground rules andvery carefully plan a nongameable way toadminister the test. But the overall problem isintuitive and challenging. We could easilyimagine a poster announcing the challenge.

Take the Scholastic Aptitude Test (SAT). Hereeach system would take the SAT on the sameSaturday morning that human students do invirtually the same context and would berequired to end up in the 90th percentile ofhigh school student performance. Again in thiscase, it is very simple to describe the challengeand it would be very clear to even the generalpublic how monumental an achievement thiswould be. Many people complain that the SATis not a “good” or adequate test for humans ofhigh school age. But no matter how flawed weall think the SAT is, we use it, our kids take it,and it matters for getting into college. It is fair-ly broad and involves a substantial amount ofbackground knowledge and the ability to applylearned skills in novel settings. Wouldn’t it beexciting if a computer program could, underthe same ground rules as the students, actuallydo well?

Write an essay that is graded well by a high-school teacher. Picture your typical school

research paper assignment: write a short essayon an assigned topic, first doing some researchusing defined information resources, and thenproducing a coherent paper in a finite periodof time. For example, I may not know muchabout the French revolution, but if I were giventwo hours and access to the world wide web, Icould probably put together at least a plausibletwo-page essay on the causes of the French Rev-olution, or what happened at the Bastille, orthe origins of “Let them eat cake!” Whyshouldn’t a machine be able to do this? Whydon’t we have computers try to write essaysjust the way we have kids do it, have the essaysgraded by human teachers in exactly the sameway, and then as soon as one of them gets an“A,” celebrate a monumental step in the histo-ry of AI?

Articles

WINTER 2006 33

Be a “search engine” in the real world. Whydon’t we build search engines for the physicalworld? In fact, why don’t we create a grandchallenge treasure-hunt adventure? Imaginewe are going to take mobile robots entered intothe competition to a previously undisclosedphysical location. It could be a house, but onethat the entrants have never seen before. Wetell them to find a $1,000,000 treasure in achest in one of the rooms (or some other chal-lenging physical and mental quest). Make thecircumstances difficult enough such that itwould test a human. In the end, the treasurewould be much more than watching the robotbring out the chest—the real treasure would bethe advances in AI systems and science that arebrought to the world.

Of course, the sky is the limit here. You canimagine a very wide variety of challenges,which would be successively harder and insome cases would ultimately appear to take tru-ly an integrated AI system to actually solve aproblem like this and win the grand prize. For-get a vehicle that can drive itself across thedesert to Las Vegas—let’s challenge our col-leagues to build intelligent machines to driveto Las Vegas from anywhere in North America,find a reasonable hotel for a night, check in,and then win some of its designers’ moneyback in a casino, walking to the cashier with itschips, and depositing money in the right bankaccount….

Conclusion: The AAAI ChallengeThis somewhat meandering discussion leadsme to a very simple and obvious conclusion.Who will lead the way to these new, integratedsystems? Who will bring together our appar-ently divergent research specialties? And whowill lead the charge in creating, motivating,and running a set of AI Grand Challenges?Clearly, if we are going to focus more on inte-grated systems work, what you might call trueAI, we still need a venue for exploring and shar-ing our results, talking to one another, and alsodiscussing and running these grand challenges.I don’t particularly think that this should be agovernment activity, although if we can con-vince DARPA to sponsor another grand chal-lenge that demands more integrated intelli-gence, that would be great. I think these thingsshould be the province of the researchers andscientists and leaders in the field.

The obvious place to do this is, of course,AAAI. In some ways, this is exactly what theorganization was founded to do more than 25years ago: to be the central place where all of uscome together, in a conference setting, in

print, and as an organization. It should sponsoractivities to bring together people who ulti-mately need to collaborate, and create anatmosphere where those researchers couldwork together closely to build truly integratedAI systems. This movement back to the roots ofAAAI has to start with the AAAI leadership, andwith each of you.2 We all need to do what TomDietterich observed that he was forced intodoing: all of us should in fact start talkingmuch more to colleagues we haven’t spoken toin 5, 10, or 25 years. Only through such per-sonal, substantive collaboration, facilitatedthrough AAAI’s operations and meetings, willwe be able to reach our goal of building thekind of AI system that was the heart and soulof the field when AAAI was founded in 1980,and was the dream of the founders of the fielda half-century ago.

Note1. See John McCarthy’s “Programs with CommonSense,” Teddington Conference on the Mechaniza-tion of Though Processes, December, 1950 (www-for-mal.stanford.edu/jmc/mcc59/mcc59.html).

2. I am pleased to note that AI Magazine (27[2]:2006)recently featured a special issue with the theme,“Achieving Human-Level Intelligence through Inte-grated Systems and Research.”

Ron Brachman is the vice presi-dent of worldwide research opera-tions at Yahoo! Research, theadvanced research arm of theworldwide leader in Internet serv-ices. He earned his B.S.E.E. degreefrom Princeton University (1971),and his S.M. (1972) and Ph.D.(1977) degrees from Harvard Uni-

versity. Brachman has made numerous importantcontributions to the area of knowledge representa-tion, including Knowledge Representation and Reason-ing, a recent textbook written with Hector Levesque.Brachman started his career at Bolt Beranek & New-man, spent several years at Fairchild/Schlumberger’sLab for AI Research, and, having developed a world-class AI group at AT&T Bell Laboratories, moved intosenior research management jobs at Bell Labs andAT&T Labs. He served as the director of DARPA’sInformation Processing Technology Office from 2002to 2005, and there developed IPTO’s Cognitive Sys-tems initiative, which brought hundreds of millionsof dollars to the national research community. Brach-man was program chair for AAAI-84, and was thepresident of AAAI from 2003–2005. He is a foundingAAAI Fellow and a Fellow of the Association for Com-puting Machinery (ACM). At the International JointConference on Artificial Intelligence in January of2007 he will be awarded the Donald E. Walker Distin-guished Service Award.

Articles

34 AI MAGAZINE