How Society Can Maintain Human-Centric Artiﬁcial Intelligencejjb/ftp/BrysonTheodorou19.pdf · How Society Can Maintain Human-Centric Artiﬁcial Intelligence 3 Given that these

How Society Can Maintain Human-CentricArtificial Intelligence

Joanna J. Bryson and Andreas Theodorou

Abstract Although not a goal universally held, maintaining human-centric artifi-cial intelligence is necessary for society’s long-term stability. Fortunately, the legaland technological problems of maintaining control are actually fairly well under-stood and amenable to engineering. The real problem is establishing the social andpolitical will for assigning and maintaining accountability for artifacts when theseartefacts are generated or used. In this chapter we review the necessity and tractabil-ity of maintaining human control, and the mechanisms by which such control canbe achieved. What makes the problem both most interesting and most threatening isthat achieving consensus around any human-centred approach requires at least somemeasure of agreement on broad existential concerns.

1 Introduction: Remit and Definitions

The greatest challenges of appropriately regulating artificial intelligence (AI) aresocial rather than technical. First, we cannot agree on a definition of the term, eventhough there are perfectly well established definitions of both artificial and intel-ligence. The primary problem is that we as humans identify as intelligent, whichcertainly is one of our characteristics, but that does not imply that intelligent means‘human-like’. We are not only intelligent but tall, long-lived, and terrestrial, at leastcompared to other vertebrates (animals with spines). So from the outset it shouldbe clear that this chapter is not—or not principally—about artificial humans, butabout all artefacts that are intelligent. This includes not only humanoid robots buta wide range of intelligent tools and services, including social media platforms,

Joanna BrysonUniversity of Bath, BA2 7AY, United Kingdom, e-mail: [email protected]

Andreas TheodorouUniversity of Bath, BA2 7AY, United Kingdom, e-mail: [email protected]

1

2 Joanna J. Bryson and Andreas Theodorou

driverless and AI-ehanced conventional automobiles, smartphones, spellcheckers,and thermostats.

The term human in this chapter will be reserved to mean members of the speciesHomo sapiens as a species is ordinarily recognised in biology. While it is at a mini-mum generous and possibly highly moral to concern ourselves about the well beingof anything that could share phenomenological sensations such as pain and loneli-ness that members of our species do, it is essentially impossible that we will everbuild something from metal and silicon that will be as phenomenologically similarto us as rats or cows are. So again, it is worth being clear from the outset that thischapter is not about humans that have been created via cloning, or other forms ofintentional but slight alterations of what is fundamentally our evolved biological de-sign. Rather, this chapter concerns artefacts built from the ground up, though we domean to include systems with non-deterministic elements of design such as machinelearning or random number generators. We will however leave discussions of prob-lems concerning the phenomenological experiences of such artefacts until humanityhas agreed to avoid the suffering of rats and cows.

Having said how we do not define ‘intelligence,’ it is now appropriate to discusshow we will. For the purpose of this chapter:

• An agent is anything capable of altering the world. This includes chemicalagents.

• Computation is the systematic transformation of information from one state toanother. Computation is a physical process, requiring time, space, and energy.

• Intelligence is a special case of computation, that generates a special form ofagency where actions — alterations of the world — are generated from percep-tion — sensing of the world. Intelligence is a property of an agent that allows thatagent to change its world in response to contexts, to opportunities and challenges.This recognition and addressing of the environment is achieved via computation.This definition is widely used in both natural and artificial intelligence, and datesto at least the nineteenth century (Romanes, 1883).

Artificial intelligence (AI) is simply intelligence expressed by an artefact, which forsimplicity we will define as something built intentionally by a human, or multiplehumans working together.

We also define two more terms that are the real sources of societal concern thatare often misdirected towards the term intelligent.

• A moral agent is an agent that a society holds responsible for its own actions.• A moral patient is any entity that a society considers it to be the responsibility of

moral agents to protect.

Whilst we may often think that such concepts must be universal—and certainly his-torical ethical systems such as religions will often lead us to believe this is so—infact there is tremendous variation by society on these details. Only recently havemany humans come to recognise climate as a moral patient. Different nations andeven states within nations have different ages at which they consider a human tobe old enough to vote, fight in a war, choose a marriage partner, or consent to sex.

How Society Can Maintain Human-Centric Artificial Intelligence 3

Given that these are some of the most momentous decisions an individual can make,it is striking that there is no universal agreement on when moral agency is achieved.From this it becomes evident that ethics itself is a social construction. In fact ar-guably, ethics may be definitionally the means by which a society constructs itself,an idea explored at more length by Bryson (2018).

Finally, the title of this chapter implies that we already have Human-Centric AI.This is largely true, though arguably not entirely. We certainly do already have AIby the straight-forward definitions given here. First, we have technology like Websearch, spell and grammar checking, and Global Positioning System (GPS) navi-gation systems—all AI that billions of people interact with daily. These are AI asservice; intelligent systems that transform data into recommendations that we actupon, or not. But secondly, some would argue that our existing corporations andgovernments are excellent examples of AI (List and Pettit, 2011). True, these arte-facts include humans as part of their systems, but they are also already exactly thesort of phenomena some describe when they use words or phrases like ‘superintel-ligence’ or ‘artificial general intelligence.’ Human society as a whole is increasingits capacity to learn exponentially, by extending ourselves through our artefacts,and also just by extending our own sheer numbers. Many of the artefacts benefitingthis system are not AI, but simply communication, education, and nutritional tech-nologies which make us as individuals smarter, and give us access to each other’scapacities for intelligence. But the identified challenges of superintelligence such asrun-away processes over consuming available resources (Bostrom, 2012) are a gooddescription of humanity’s present challenges with respect to sustainability.

The extent to which governments, corporations, and their technological tools arehuman-centric can be debated, but more often the debate concerns who among hu-manity benefits, not whether something other than a subpopulation of humans istruly benefiting. This chapter does not seek directly to solve this question, but doesassume that governments and corporations at least are focused on and controlled byat least some set of humans. Our purpose is to show that similar or greater levels ofcontrol can and should be expressed over the AI products humans produce. At thehighest level, the means by which this objective may be achieved is by maintain-ing ordinary levels of human accountability for the devices we produce. We will gointo greater detail about how this can be achieved below, but first we discuss why itshould be.

2 Why Maintain Human-Centric AI

As just admitted, ‘maintaining’ human-centric AI isn’t exactly the situation we findourselves in. To the extent that corporations or governments function to serve theirown persistence even where that does not benefit humanity, then AI may already beseen as not human-centric. The extent to which this is the current situation is muchdebated. This will not be the focus of this chapter, but we will return to this questionbriefly at the end of this section. For the purpose of the present chapter, we will


assume that these institutions largely serve humanity, and that what we really meanby our title is that we wish AI to make the situation no worse than it is, and perhapseven to improve it.

There are many possible humanist reasons to maintain human-centred control.First, we should say that there are two possible alternatives, which actually amountto much the same thing. The first is that we lose control absolutely, and the secondis that control is handed from humanity to our artefacts. Whilst there will alwaysbe anarchists and nihilists arguing for the former, we will neglect that option heresince people holding such positions are unlikely to become organised enough todismantle control globally. The latter though is seen by many as desirable or evennecessary. Aware of their own mortality and that of civilisations and species as well(cf. Scranton, 2015), they put their hope in artefactual progeny. Perhaps this is be-cause (ironically) they can exercise more control over artefacts than over biologicalprogeny, or perhaps they mistakenly believe that machines (unlike humans) can beimmortal or omniscient. The fact that the average working ‘life’ of an artefact is far,far shorter than the average lifespan of a human (or even a chimpanzee) is apparentlyregarded as irrelevant. Perhaps they think machines can be made self-repairing, butin this sense so are biological lineages (Taylor and Bryson, 2014). Again, that anypurely-mechanical technology lineage we produce will exceed the lifespan of ourbiological lineage is phenomenally unlikely.

It seems that the problem is that AI is viewed not as a type of computation—aphysical process, but as a type of math—an abstraction. Mathematics may be eternaland perfect, but that is because it is not real. Computation being a physical processrequires time, space, an energy. Even if we are able to achieve long-term energyindependence (at least relative to our level of our demand) at some stage, we willalways be constrained by space and time.

The above are only reasons not to argue against human-centred AI, but here wegive two reasons to argue for it. First, every aspect of our values—not only ourethics, and our human drives and desires, but also our sense of aesthetics—all ofthese have coevolved with our species and societies in order to maintain our speciesand societies (Bryson and Kime, 2011). There is no coherent sense in having ma-chines enjoy hedonism for us, although we can use machines to capture resourcesthat we could not ourselves exploit, preventing them from being exploited by others.While some openly find pleasure in such an expression of power, it is not somethingwe choose to openly condone here, and we doubt it would be condoned by the ma-jority of any stable society were they to recognise this as being the impulse for theirsupport of ‘artificial life.’

Second, all social order is based on concepts and institutions of justice that un-fortunately have human suffering at their core as a means of dissuasion (Brysonet al., 2017). Law may seem to create compensation, and we could imagine a ma-chine (for example) financially compensating for its wrongful actions. But in fact,law is mostly about dissuasion. Laws and treaties are a means by which we set outagreed behaviour, and agreed costs of violating that behaviour. We have coevolvedwith these institutions for so long that we really do feel like we’ve received someform of compensation when in fact we have only received justice. For example, if


someone kills your lover, and that person goes to jail, you have received nothingremotely like what you have lost, but you perceive victory. In fact, perhaps part ofwhat you lost is social status and faith in the system, and perhaps justice returnsthese to you. But these abstractions exist in order to maintain social order, and restupon our biological architecture that makes stress and pain pervasively dysphoric,and isolation and loss painful and stressful.

We cannot build machines that can so systemically experience such pervasivedysphoria. Probably we cannot build such a machine at all, but certainly we cannotbuild one for which we can guarantee its safety. In fact here we return to the ideathat AI is already somewhat out of our control, if we accept the List and Pettit(2011) account of corporations as AI. Corporations are extended legal personhoodas a legal convenience, but it’s a convenience allowable only because real humansare dissuaded from doing wrong by human justice. And we should not have said‘because,’ we should have said ‘to the extent which.’ A shell company dissociatesthe humans who would suffer if the company does wrong from the humans whodecide what the company does (cf. Bryson et al., 2017). Weapons such as guns,airplanes and bombs, and also chains of command (military or corporate) similarlyremove individuals at least some ways from the consequences of their decisions,which makes decisions with deeply aversive consequences easier to take.

The primary reason to maintain or even increase the extent to which AI is human-centric is that to do otherwise would be far more likely allow a greater dismantlingof justice, resulting in greater human suffering, than it would be to produce a newform of social or somehow universal good.

3 Maintaining Human Control Through Design

There are two means by which human control may be maintained over AI. First,good design of AI systems allows us to ensure that intelligent systems operate withinthe parameters we expect. Contrary to some contemporary horror stories, machinelearning (even DNN) doesn’t make this difficult. It is not hard to ring fence whataspects of an AI system are subject to (can altered by) its own learning or planning.In fact, constraining processes like learning and planning allows them to operatemore efficiently and effectively, as well as more safely. This is because the amountof computation (time, space, and energy) required is directly related to the amountof possibilities that need to be explored. Thus appropriate constraint is one of themain means for making any system, including humans, smarter. We teach studentsthe sets of tools, facts, and approaches that have been shown to date most likely toproduce useful outcomes.

The second means of maintaining human control is by holding those who build,own, or operate AI accountable for their systems through law and regulation. Thisapproach will be described in the following section, but requires first understandingthat the first approach is both possible and desirable. That is the focus of this section.


To begin with, it has long been established that the easiest way to tackle verylarge engineering projects is to decompose the problem wherever possible into sub-projects, or modules (Bryson, 2000). For example, one component of a driverless caris the GPS navigation system, which has been so completely modularised that it isroutinely used by enormous numbers of human drivers daily. There is no reason thata single automobile’s ‘mind’ should alter the algorithm by which new routes arechosen, although the observations of an automobile may contribute to the crowd-sourced data on the current traffic on a road, or even the nuances of controlling aparticular make of car. Here again, even if such a crowd-sourced learning strategy isused to recognise and avoid congestion, the constantly-updating models of the cur-rent traffic conditions will not alter the independent model of the underlying roads.Neither model will have any direct access to control over where or whether the carmoves, which is another module still, or for the time being, more likely a humandriver.

More generally, one method for designing modular decomposition for an AI sys-tems is to assess what the system needs to know, and for each aspect of that knowl-edge, the best way to maintain that knowledge, as well as to exploit it. Here wedescribe one such approach to systems engineering real-time AI. We use this as acase to demonstrate what is possible, and then to illustrate the more general claimsabout accountability, transparency, regulation, and social control of AI made in thesection following.

3.1 Behaviour Oriented Design

The above observation—that an ontology of required knowledge and its most con-venient representation for expressing timely action should be used as the basisfor modular decomposition for intelligent systems—is a core contribution of Be-haviour Oriented Design (BOD), which is one methodology for systems engineer-ing real-time AI systems (Bryson, 2001, 2003). BOD takes inspiration both fromthe well-established programming paradigm of object-oriented design (ODD) andits associated agile design (Cockburn, 2006; Gaudl et al., 2013), and an older butstill very well-known AI systems-engineering strategy, called Behaviour-Base AI(BBAI, Brooks, 1991). Behaviour-based design lead to the first AI systems capableof moving at animal-like speeds, and many of its innovations are still extremelyinfluential in realtime systems today. Its primary contribution was to emphasisedesign—specifically, modular design. Previous AI researchers, inspired by their in-terpretation of their own conscious experience, had expected to express the entireworld in a system of logical perfection and then to take only the provably optimalaction (Chapman, 1987). BBAI instead focuses on

1. the actions the system is intended to produce, and2. the minimum, maximally-specialised perception required to generate each ac-

tion.


In BBAI, each module derives action from its own dedicated perception.While based in real-world experience of building robots, and as mentioned be-

ing the first approach that really succeeded in animal-like navigation at animal-likespeeds, there were problems with BBAI as Brooks originally construed it. The firstproblem with this approach is coordinating the modules. Decomposing for simplic-ity is of little use if the subsequent coordination proves intractable. Second, Brooks’experience with traditional robot planning and the complexities of dealing with theworld lead him to dismiss any real-time extension of intelligent control whatsoever.BBAI in its original form has no onboard planning (at least, no revision of the pri-orities encoded in the AI) nor learning whatsoever. Brooks initially claimed (likeLorenz before him) that embedding intelligence in its ecological niche was too del-icate a problem to be open to risky processes like onboard learning, and that whatappeared to be thought and learning were epiphenomenal suppositions imposed byus as observers as the organism interacted with a complex environment. “The worldis its own best model” (Brooks, 1991). While this emphasis revolutionised AI by re-focussing it on proper systems design, it cannot really account for all of human-likeor even insect-like behaviour (Papaj and Lewis, 2012).

BOD connects AI properly back to systems engineering via OOD, affordingsafety in AI by exploiting BBAI-like modular architectures to limit the scope ofmachine learning, planning, or other real-time plasticity to the actions or skills re-quiring the capacity to accommodate change. Such architectural design is essentialnot only for safety, but also simply for computational tractability. As mentioned ear-lier, learning systems are faster and more likely to succeed if they are conductingtheir search over relevant possible capacities. Brains do the same thing. Contrary toSkinner (1935), pigeons can learn to peck for food or to flap their wings to escapeshock, but not to flap their wings for food or to peck to avoid shock (Gallistel et al.,1991). Biological evolution also provides architecture as scaffolding for viable sys-tems. Again in contrast to some sensationalist contemporary horror stories, there arein fact zero AI systems for learning chess that represent power switches, or have ac-cess to guns. No AI system built to learn chess will ever shoot someone that movesto turn it off at night1.

BOD makes such common sense architectural decisions an explicit part of itsdevelopment process. In general BOD is one means of using systems engineeringto overcome problems of complexity for intelligence, by introducing an ontology,methodology, and architecture that promotes iterative design and testing. BOD in-cludes common-sense heuristics for modular decomposition, documentation, refac-toring, and code reuse. By using well-established OOD techniques, BOD allowsdecomposition of intelligent behaviour into multiple modules forming what we calla behaviour library. Behaviour library modules may wrap machine learning sys-tems, or indeed commercial AI services providing specific capacities such as facerecognition or navigation.

Stringing these modules together into a coherent agent requires then only specify-ing the priorities of that agent. Notice that multiple agents with completely different

1 Another stupidity of the gun-toting, chess-learning murderous AI fairytale propagated by theFuture of Life Institute is that real AI developers prefer our systems to do work while we sleep.


goals can be constructed from the same behaviour library, providing only that theyeither exploit the same type of hardware platform, or that the modules have beenconstructed to be platform-independent. Aspects of intelligence can also be hostedon servers or in clouds and accessed over an API, but of course for a real-time sys-tem much critical intelligent infrastructure needs to be hosted in a way such thatthe communication rate between modules and their embedded hardware substratecan be guaranteed. Further, any system learning proprietary information e.g. aboutits owner’s household should probably better host such information securely andsolely on site (Kalofonos et al., 2008).

3.2 Specifying a System’s Priorities

One of the innovations of BOD compared to both BBAI and OOD is to simplifythe problem of arbitrating between different modules that might otherwise producecontradictory actions away from a highly distributed, difficult to conceptualise ordesign network of dependencies, and back towards a more traditional hierarchicalrepresentation of priorities. Of course, there were good reasons for Brooks’ origi-nal avoidance of these hierarchies, concerning efficiency. As Blumberg (1996) ob-served, action selection is only necessitated where there is competition for resources.If no such competition exists, the behaviour modules are able to work in parallel.However, many things are in this sense a resource, including a robot’s physical lo-cation, direction of gaze, and what it can hold on to.

Bryson (2001) introduces POSH (Parallel-rooted, Ordered, Slip-Stack Hierar-chical) action selection. These ideas were taken up also by the far better-named Be-haviour Trees (BT) (Isla, 2015; Rabin, 2014) which function just as well for BODsystems engineering of real-time AI, but here we focus on our original nomencla-ture. For historic reasons, the data structure built from POSH (or BT) components,describing an agent’s priorities, is termed a plan, and the part of the AI systemthat checks these priorities is called a planner. This is true even though the plannertypically will not alter the POSH plans in the system, but the planner and the planstogether determine the sequence of steps the agent takes in pursuing its goals, whichmight be more conventionally seen as a plan.

POSH plans combines faster response times similar to the fully reactive ap-proaches for BBAI with a greater ease of developing goal-directed plans. A POSHplan consists of the following elements:

1. Drive Collection (DC): This is the root or apex of the plan’s hierarchy. It con-tains a set of Drives and is responsible for giving attention to the highest priorityDrive that presently could use that attention. The POSH planning cycle alternatesbetween checking for what is currently the highest level priority that should beactive, and then progressing work on that priority. This check is made hundredsor thousands of times a second, to ensure the system’s highest priority goals(which should ensure its safety) are constantly monitored.


2. Drive (D): Allows for the design of behaviour in pursuit of a specific goal. Eachdrive maintains its execution state even when it is not the focus of planner at-tention, allowing the system to express coarse-grained parallelism even withinprioritised actions, as well as independently by modules not requiring arbitra-tion. Each drive specifies its own perceptual context which is suitable to or re-quires its deployment, while the Drive Collection as a whole maintains track ofthe multiple Drives’ relative priorities.

3. Competence (C): A simpler hierarchical plan element for representing the prior-ities within a particular component of a plan (also known as a subplan). Com-petences are similar to the drive collection but lack the extra checks and mecha-nisms for concurrency, which are handled entirely at the top level or root D. Eachcompetence contains one or more Competence Elements (CE), which also are as-sociated with both a priority relative to the other CEs, and a context which canperceive and report when that element can execute. The highest-priority actionthat can execute will execute when the Competence receives attention.

4. Action Pattern (AP): These are simple sequences of actions and perceptions usedto reduce the design complexity of a plan when such a sequence of actions canbe determined in advance.

5. Action (A): A call to code in the behaviour library that sets a skilled act in mo-tion. To maintain agility in the planner, actions should not block (wait for a fi-nal response in the world) but simply return immediately with a code indicatingwhether or not the action was successfully initiated. Other plan elements can bedesigned to watch for a context in which this action has succeeded or failed ifthat knowledge is essential. However, in both biology and AI quite often actionsare just run ‘open loop’, without checks, and action selection is simply repeatedin the next instant in the new context produced by the agent’s actions or inactionsas time has progressed.

6. Sense (S): Senses are very much like actions, and also depend on the behaviourlibrary for their implementation. The difference is that they return a value indi-cating context, which may be used to determine for example whether a Drive orCompetence should be released to execute, or an Action Pattern aborted.

Taken together, these plan elements are sufficient for expressing the goals ofmany systems. Of course, for complex systems with multiple, potentially conflict-ing goals (e.g. maintaining a job and maintaining a relationship, or hoovering thehouse and entertaining the dog) it may be useful for the order of priorities to shiftover time. For this we have developed several systems of synthetic emotions ormoods. Essentially, a mood or emotion is a special type of behaviour module thatdetermines its own current priority. Drives linked to these emotions have from thedrive collection’s perspective the same level of priority, and a separate system en-sures that only one of these at a time receives the focus of attention (Bryson andTanguy, 2010; Gaudl and Bryson, 2018; Wortham et al., 2018).


3.3 Real-Time Debugging of Priorities

Another myth of AI is that systems should become as intelligent as humans andtherefore not require any more training than a human. In reality, very few will wantto put as much energy into training an AI system as is required to raise a child, oreven to train an intern, apprentice, or graduate student. Programming is generallya far more direct and efficient way to communicate what is known and knowableabout generating appropriate behaviour. However, debugging a complex, modular,real-time system requires more insight than ordinary programming. Further, we maywell want to allow non-programmers to set priorities and choose between capacitiesfor their agents once reliable behaviour libraries have been defined. Both of theseactivities requires an element of transparency to a system. Here we use transparencyto mean that the direct workings of the system should be made apparent — visibleand understandable (Bryson and Winfield, 2017; Theodorou et al., 2017).

Hierarchical definitions of priorities like POSH plans or Behaviour Trees offera sensible means of transparency for either of these two applications: expert de-bugging or ordinary user understanding. Here we again describe novel work in ourown group, but the basic concept behind this may be generalised to other forms ofsystems engineering. At Bath, we have developed a real-time visualisation systemand debugger for POSH plans. The system, ABOD3, is based on, but a substan-tial revision and extension of, ABODE (A BOD Environment, originally built bySteve Gray and Simon Jones, Bryson et al., 2005; Brom et al., 2006). ABOD3, firstdescribed by Theodorou (2017) and shown here in Figure 1, allows the graphicalvisualisation of POSH-like plans. The editor, as seen in its architecture diagram inFigure 2, is implemented in such a way as to provide for expandability and customi-sation, allowing the accommodation of a wide variety of applications and potentialusers.

ABOD3 is designed to allow not only the development of reactive plans, but alsothe debugging of such plans in real time. The editor provides a user-customisableuser interface (UI). Plan elements, their subtrees, and debugging-related informa-tion can be hidden, to allow different levels of abstraction and present only relevantinformation to the present development or debugging task. The graphical representa-tion of the plan can be generated automatically, and the user can override its defaultlayout by moving elements to suit needs and preferences. The simple UI and cus-tomisation allows the editor to be employed not only as a developer’s tool, but alsohas been demontrated to present transparency-related information to naive users thathelps them develop more accurate mental models of a mobile robot (Wortham et al.,2017a).

Plan elements flash as they are called by the planner and glow based on thenumber of recent invocations of that element. Plan elements without any recent in-vocations start dimming down, over a user-defined interval, until they return backto their initial state. This offers abstracted backtracing of the calls, and the debug-ging of a common problem in distributed systems: race conditions where two ormore subcomponents are constantly triggering each other then interfering with oreven cancelling each other’s effects. Finally, ABOD3 can also support integration


Fig. 1 The ABOD3 Graphical Transparency Tool displaying a POSH plan for a mobile robot,in debugging mode. The highlighted elements are the ones recently called by the planner. Theintensity of the glow indicates the number of recent calls.

with videos of the agents in action, allowing for non-real-time debugging based onlogged performance. Logging of actions taken and contexts encountered is a sub-stantial aspect of AI accountability and transparency, which we will return to in thenext section.

4 Maintaining Human Control Through Accountability andTransparency

To reiterate, although we have here described the systems-engineering approach andtools we have been developing together at the University of Bath, we are not claim-ing that these are the only, best, or most essential means for maintaining human con-trol of AI. We are rather communicating that such control is perfectly possible, andillustrating examples of some of the technological mechanisms by which such con-trol can be maintained. It is also perfectly possible to build AI for which accountingis not possible, indeed this too has already been done and is indeed too prevalent inour society (Pasquale, 2015). In this section, we summarise what is essential abouttechnological mechanisms for human control, then close with a discussion about thesocial, legal, and political mechanisms for maintaining that control, which are actu-


Fig. 2 System architecture diagram of ABOD3, showing its modular design. All of ABOD3 iswritten in Java to ensure cross-platform compatibility. APIs allow the support of additional BODplanners for real-time debugging or even multiple file formats for the plans. The editor is intended,through personalisation, to support roboticists, software AI developers, and ordinary users inter-ested in AI systems.

ally far more important. Technology serves and extends human society, but ethics iswhat forms and defines human society.

4.1 Technological Mechanisms for Ensuring Transparency andAccountability

What is important to realise is that every aspect of an artefact is a consequence ofdesign decisions. We are not saying that it is trivial to know what any AI systemis doing. We are saying that it is possible to provide the tools and keep the recordssuch that we know at the level sufficient to maintain human accountability what goeswrong with a system, if it goes wrong, and how and why it was constructed suchthat it did go wrong. There are social requirements underlying these technologicalfeatures: can a person or a company show that they followed due diligence whenthey created an artefact? If not, they should be held liable for what that artefactdoes.


Fig. 3 ABOD3 implemented as part of a serious game (the Sustainability Game) so that gameplayers can understand the interaction between agent motivations and the viability of an artificialcommunity (Theodorou et al., 2019).

This does not mean that AI has to be deterministic or formally provably correct.Due diligence can be demonstrated by corporations despite the fact they employpeople. People learn, have free will, and have incomprehensible systems of synapsesmaking up their action selection mechanisms. Many humans are dishonest, careless,or sometimes just upset or incompetent. Nevertheless, we can construct systems thatensure that humans working together tend to succeed. These systems generally in-clude records, such as financial accounts, access permissions, and minuted meetingswhere executive decisions are agreed. They also include external regulatory bodiesand law enforcement.

Exactly the same kinds of procedures can be used for retaining control of AI,and indeed already are at least in well-regulated sectors like the automotive industry(Doll et al., 2015). In every single case so far concerning a human fatality involv-ing a driverless car, newspapers have within a week shown us exactly what the carperceived, how that perception had been categorised, and what actions the car wastaking at the point of fatality, and even why. Keeping records of this sort of in-formation is not difficult, but it is a design decision. That decision is enforced inthe automotive industry by its high levels of regulatory accountability mandatedby the incredible amount of human suffering and death generated as its by-product(Williams, 1991). The design decision to provide adequate logging is one we canand should also enforce for other AI systems in socially critical roles.

As we described in the previous discussion of moduarlity and safety, the equiv-alents of ‘access permissions’ are also a completely standard part of design thatanyone with any practical experience of creating an intelligent systems takes forgranted. Every sensor or actuator a system has is an expense for its manufacturing,so these will naturally be restricted to those required by a system’s task, but fur-ther within the system, access to information can and should be restricted to thatinformation likely to be useful, not only for safety but simply for efficiency.


In addition to logging what a system perceives and performs, we can also logevery aspect of how we designed that system. Standard practice in software de-velopment is to use a software revision control system that documents the exactauthor and timing of any change to the system’s software. Unfortunately, not everydevelopment team will exercise best practice in terms of ensuring that each indi-vidual developer has their own individual login, or documents the reasons for theirchanges, or documents the versions of software libraries used to support their pro-gramming, or data libraries used to train their machine learning. In fact, there hasbeen a well-documented, scandalous disregard for the provenance of both softwarelibraries (Gurses and van Hoboken, 2018) and data libraries (Pasquale, 2015). How-ever, there is no technological reason that a better standard of practice couldn’t begenerated, and even required.

All of the mechanisms described above, and also the architectural concepts andsoftware tools described in the previous section, are mechanisms of transparency.To be clear, when we talk about transparency here, we mean neither invisibility (asis sometimes advocated by human-computer interaction specialists), nor (necessar-ily) mandatory open-sourced code or formal, symbolic programming. The former—invisibility—actually increases the hazard of AI as ordinary users fail to realise theirdata is being gathered or to consider the consequences of compromising the securityof the system. The latter can produce more information than humans can accom-modate without resulting in clarity about responsibility or good practice. What iseffectively transparent therefore varies by who the observer is, and what their goalsand obligations are (Bryson and Winfield, 2017; Theodorou et al., 2017).

The goal of transparency is never complete comprehension. That would severelylimit the scope of human achievement. Rather, the goal of transparency is providingsufficient information to ensure that at least human accountability, and thereforecontrol, can be maintained.

Our position about transparency is supported by Dzindolet et al. (2003), whoconducted a study where the participants had to decide whether they trust a particu-lar piece of pattern recognition software. The users were given only the percentageof how accurate the prediction of their probabilistic algorithm was in each image.Yet, by having access to this easy-to-implement transparency feature, they were ableto calibrate their trust in real time. Our own studies (discussed in Wortham et al.,2017a,b), demonstrate how users of various demographic backgrounds had inac-curate mental models about a mobile robot running a POSH planner. They wereascribing unrealistic functionalities, potentially raising their expectations for its in-telligence and safety. When the same robot was presented with ABOD3 providingan end-user transparency visualisation, the users were able to calibrate their mentalmodels. This lead to more realistic expectations concerning the system’s capabili-ties, though interestingly also a higher assessment of its intelligence.


4.2 Maintaining Human Control Through Governance andRegulation

There have at various periods, including the present, been a worrying tendency toblame individual scientists or programmers for the consequences of their work.While it is true that individuals are accountable for their actions—including lifechoices concerning their employers—successful regulation requires looking at theentire context of that action. If we know there will or at least can be individualswho are dishonest, sloppy, suicidal, corrupted, or simply prone to occasional errors(that is, human), then we should expect systems containing such individuals shouldhave some means for ensuring and promoting the quality of their work. For AI, thescale of this task may sound insurmountable—do we really think we should checkthe work of every individual programmer, globally? Who would do such a thing?Yet this is exactly what Apple does for individual programmers who want to writesoftware applications for Apple’s smart phone, the iPhone. Smart phones are themost fantastic information-gathering devices ever created, so it makes sense to havethis level of security and scrutiny enforced by the maker and owner of this platform.Note also that despite the cost of such an operation, Apple has a perfectly successfulbusiness model for producing wealth as well as products.

We mentioned just above that car manufacturers already are developing vastamounts of AI in a highly regulated environment. At least some of them have alsobeen able to successfully demonstrate that they practice due diligence when they areinvestigated by state prosecutors (Doll et al., 2015). But what about organisationsthat changed the world in unanticipated ways by introducing entirely new platformsand therefore capacities into societies and economies. Can they also be held ac-countable for damage done with the tools they’ve provided?

This is a question being addressed in courts and legislatures globally as we write,but we believe that the short answer is ‘yes, to a point.’ That point is demonstratedcommunity standards of good practice. So if for example damage results from theobviously poor (and often illegal) standards of conduct documented by Pasquale(2015) or Gurses and van Hoboken (2018), then governments and other collectivesshould hold organisations that profit from this conduct accountable for the damagethey cause. Similarly, if most organisations refuse to sell access to the data they col-lect from their users because doing so would seem a clear ethical violation, but someorganisations do sell such data, then these latter organisations can be held account-able for violating the known ethical standards of their sector. This is particularlytrue for organisations of scale, who are routinely held to higher standards by the lawbecause of their position of leadership. With great power (or even just money) doesindeed come great responsibility.

In discussions we’ve held in the United Kingdom (UK) at least, it appears thatthere is not really a call for changes in legislation (House of Lords, 2018). Rather,what is needed is only to get through the fog of confusion caused by the smokeand mirrors associated with ‘intelligence.’ This is why we started this chapter as wedid, to make it clear that AI and indeed I are ordinary properties amenable to both


science and law. Once this is clear, then with a little education and some good hiring,ordinary legal enforcement of liability standards should be sufficient to maintainhuman control.

AI does present two special problems however. One we’ve already mentioned butwill come back to again here. There is a mistaken belief that the capacity to expresshuman-like behaviour is in any way indicative of commonality of pheneomenolog-ical experience between machines and humans. As Caliskan et al. (2017) demon-strate, a glorified spread sheet that has just counted words on the Internet can reportphenomenological commonality with humans, e.g. that flowers are more pleasantthan insects, or even stereotyped beliefs such as that women’s names are more as-sociated with the domestic sphere. Such a system barely even qualifies as AI bythe definition we’ve given since the only ‘action’ from its perception of the Web isthe numerical report of what words are associated with what others. Further, thesecounts are replicated globally in standard AI tools, so there is no hazard of loss ofa unique perspective if we destroy one of these spreadsheets, as there is if we losea single human life, or even a unique copy of an old book or fossil. Humans actdifferently around robots that look human to them, but then humans act differentlyaround statues that look human. Public spaces that had felt and been dangerous feeland become safer when ordinary human statues are introduced at ordinary humanscale (Johnson, 2017). Thus reports of phenomenological similarity generated eitherby AI or by human observers cannot be seen as valid demonstrations of AI moralpatiency.

Unfortunately, many people argue that empathy is core to ethics. Empathy is aterrible metric of moral patiency; it is extended more to those more like us (Bloom,2017). Also, people are moved to self-deception by their fear of mortality and desirefor powerful progeny and partners. There are many proposals to extend the mecha-nisms that sadly often fail to protect humans to protect robots or AI (Gunkel, 2018).We share the goal of not wanting any entity to suffer unnecessarily, but we takethis to imply we should design AI so that it will not suffer, and further to ensurethat damage to AI would not incur human suffering. Again, it is a design decisionwhether we make AI that is robust, can be backed up and thus protected by standardmeans for protecting and preserving digital data. This is the only ethical decision forAI that anyone cares about, and eliminates the necessity of the sorts of protectionsextended to unique human lives.

Another problem with mistakenly thinking AI is human-like is believing thathuman punishments such as social shunning, fines, prison, and the other tools ofhuman law could be extend to it. Again, if we accept the List and Pettit (2011) def-inition of corporations as AI, we can already see that where the humans who makethe decisions are not the humans who will be held to account, corruption follows. Ifwe make artefacts to be legal persons, those artefacts will be used like a shell com-pany, to evade justice and corrupt economies and power structures (Bryson et al.,2017), leaving ordinary citizens disempowered with less protection from powerfulinstitutions (Elish, 2016).

The second special problem of AI is not actually unique to it but rather a char-acteristic of Information Communication Technology (ICT) more generally. ICT


thanks to the Internet and other networking systems operates transnationally, andtherefore affords the accumulation of great wealth and power, while simultaneouslyevading the jurisdiction of any particular nation. This means that appropriate reg-ulation of AI requires transnational cooperation. Again, the process to establishtransnational agreements, treaties, and enforcement mechanisms is nontrivial, butalready known, and already under way.

Conclusion

In conclusion, societies both can and should maintain control over artificial intel-ligence. Fortunately, significant progress is being made in achieving this goal—progress made by technology companies, regulatory bodies, governments, profes-sional organisations, and individual citizens including software developers who aretaking the time to understand the social consequences of technology. We welcomethe opportunity to describe these efforts here, and encourage our readers to jointhe perpetually ongoing project of creating a richer, fairer, and more just society inwhich we may all flourish with dignity.

Acknowledgements We would like to acknowledge our collaborators, particularly Rob Worthamfor his work on architecture and accountability, Swen Gaudl for his work on the architecture, andAlan Winfield, Karine Perset, and Karen Croxson for their work on regulation and accountability.Thanks also to the helpful reviewers for this volume. We thank AXA Research Fund for part-funding Bryson and EPSRC grant [EP/L016540/1] for funding Theodorou. Some of the descriptionof ABOD3 previously appeared in Theodorou (2017).

References

Bloom, P. (2017). Against empathy: The case for rational compassion. Random House.Blumberg, B. M. (1996). Old Tricks, New Dogs: Ethology and Interactive Creatures. PhD thesis,

MIT. Media Laboratory, Learning and Common Sense Section.Bostrom, N. (2012). The superintelligent will: Motivation and instrumental rationality in advanced

artificial agents. Minds and Machines, 22(2):71–85.Brom, C., Gemrot, J., Bıda, M., Burkert, O., Partington, S. J., and Bryson, J. J. (2006). POSH tools

for game agent development by students and non-programmers. In Mehdi, Q., Mtenzi, F., Dug-gan, B., and McAtamney, H., editors, The Ninth International Computer Games Conference:AI, Mobile, Educational and Serious Games, pages 126–133. University of Wolverhampton.

Brooks, R. A. (1991). New Approaches to Robotics. Science, 253(5025):1227–1232.Bryson, J. J. (2000). Cross-paradigm analysis of autonomous agent architecture. Journal of Ex-

perimental and Theoretical Artificial Intelligence, 12(2):165–190.Bryson, J. J. (2001). Intelligence by Design: Principles of Modularity and Coordination for Engi-

neering Complex Adaptive Agents. PhD thesis, MIT, Department of EECS, Cambridge, MA.AI Technical Report 2001-003.

Bryson, J. J. (2003). Action selection and individuation in agent based modelling. In Sallach,D. L. and Macal, C., editors, Proceedings of Agent 2003: Challenges in Social Simulation,pages 317–330, Argonne, IL. Argonne National Laboratory.


Bryson, J. J. (2018). Patiency is not a virtue: the design of intelligent systems and systems ofethics. Ethics and Information Technology, 20(1):15–26.

Bryson, J. J., Caulfield, T., and Drugowitsch, J. (2005). Integrating life-like action selection intocycle-based agent simulation environments. Proceedings of Agent, pages 1–14.

Bryson, J. J., Diamantis, M. E., and Grant, T. D. (2017). Of, for, and by the people: the legal lacunaof synthetic persons. Artificial Intelligence and Law, 25(3):273–291.

Bryson, J. J. and Kime, P. P. (2011). Just an artifact: Why machines are perceived as moralagents. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence,pages 1641–1646, Barcelona. Morgan Kaufmann.

Bryson, J. J. and Tanguy, E. A. R. (2010). Simplifying the design of human-like behaviour: Emo-tions as durative dynamic state for action selection. International Journal of Synthetic Emo-tions, 1(1):30–50.

Bryson, J. J. and Winfield, A. F. T. (2017). Standardizing ethical design for artificial intelligenceand autonomous systems. Computer, 50(5):116–119.

Caliskan, A., Bryson, J. J., and Narayanan, A. (2017). Semantics derived automatically fromlanguage corpora contain human-like biases. Science, 356(6334):183–186.

Chapman, D. (1987). Planning for conjunctive goals. Artificial Intelligence, 32:333–378.Cockburn, A. (2006). Agile Software Development: The Cooperative Game. Addison-Wesley

Professional, Boston, second edition.Doll, N., Vetter, P., and Tauber, A. (2015). Wen soll das autonome auto lieber uberfahren? (whom

should the autonomous car drive over?). Welt.Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., and Beck, H. P. (2003). The role

of trust in automation reliance. International Journal of Human Computer Studies, 58(6):697–718.

Elish, M. (2016). Moral crumple zones: Cautionary tales in human–robot interaction. In WeRobot2016. working paper.

Gallistel, C., Brown, A. L., Carey, S., Gelman, R., and Keil, F. C. (1991). Lessons from animallearning for the study of cognitive development. In Carey, S. and Gelman, R., editors, TheEpigenesis of Mind, pages 3–36. Lawrence Erlbaum, Hillsdale, NJ.

Gaudl, S., Davies, S., and Bryson, J. J. (2013). Behaviour oriented design for real-time-strategygames: An approach on iterative development for STARCRAFT AI. Foundations of DigitalGames Conference, pages 198–205.

Gaudl, S. E. and Bryson, J. J. (2018). The extended ramp model: A biomimetic model of behaviourarbitration for lightweight cognitive architectures. Cognitive Systems Research, 50:1–9.

Gunkel, D. J. (2018). Robot Rights. MIT Press, Cambridge, MA.Gurses, S. and van Hoboken, J. (2018). Privacy after the agile turn. In Selinger, E., Polonetsky,

J., and Tene, O., editors, The Cambridge Handbook of Consumer Privacy, pages 579–601.Cambridge University Press.

House of Lords (2018). AI in the UK: ready, willing and able? Technical Report HL 2017–2019(100), United Kingdom.

Isla, D. (2015). Handling Complexity in the Halo 2 AI. In GDC 2005.Johnson, S. (2017). Celebrating the familiar. claim made in books and on web pages,

am trying to track down data. http://www.groundsforsculpture.org/Seward-Johnson-The-Retrospective/Celebrating-the-Familiar.

Kalofonos, D. N., Antoniou, Z., Reynolds, F. D., Van-Kleek, M., Strauss, J., and Wisner, P. (2008).Mynet: A platform for secure p2p personal and social networking services. In Pervasive Com-puting and Communications, 2008. PerCom 2008. Sixth Annual IEEE International Conferenceon, pages 135–146. IEEE.

List, C. and Pettit, P. (2011). Group agency: The possibility, design, and status of corporate agents.Oxford University Press.

Papaj, D. R. and Lewis, A. C. (2012). Insect Learning: Ecology and Evolutinary Perspectives.Springer Science & Business Media.

Pasquale, F. (2015). The black box society: The secret algorithms that control money and informa-tion. Harvard University Press.


Rabin, S. (2014). Game AI PRO 1.Romanes, G. J. (1883). Animal intelligence. D. Appleton.Scranton, R. (2015). Learning to Die in the Anthropocene: Reflections on the End of a Civilization.

City Lights Publishers, San Francisco, CA, USA.Skinner, B. (1935). The generic nature of the concepts of stimulus and response. The Journal of

General Psychology, 12(1):40–65.Taylor, D. J. and Bryson, J. J. (2014). Replicators, lineages, and interactors. Behavioral and Brain

Sciences, 37(3):276–277.Theodorou, A. (2017). A graphical visualisation and real-time debugging tool for BOD agents. In

CEUR Workshop Proceedings, volume 1855, pages 60–61.Theodorou, A., Bandt-Law, B., and Bryson, J. (2019). Game technology as an intervention for

public understanding of social investment. in preperation.Theodorou, A., Wortham, R. H., and Bryson, J. J. (2017). Designing and implementing trans-

parency for real time inspection of autonomous robots. Connection Science, 29(3):230–241.Williams, H. (1991). Autogeddon. Jonathan Cape Ltd., London.Wortham, R. H., Gaudl, S. E., and Bryson, J. J. (2018). Instinct: A biologically inspired reactive

planner for intelligent embedded systems. Cognitive Systems Research.Wortham, R. H., Theodorou, A., and Bryson, J. J. (2017a). Improving robot transparency: Real-

time visualisation of robot AI substantially improves understanding in naive observers. In 26thIEEE International Symposium on Robot and Human Interactive Communication (RO-MAN),Lisbon. IEEE.

Wortham, R. H., Theodorou, A., and Bryson, J. J. (2017b). Robot transparency: Improving under-standing of intelligent behaviour for designers and users. In Gao, Y., Fallah, S., Jin, Y., andLekakou, C., editors, Towards Autonomous Robotic Systems, pages 274–289, Cham. SpringerInternational Publishing.

Documents

How Society Can Maintain Human-Centric Artiﬁcial Intelligencejjb/ftp/BrysonTheodorou19.pdf · How Society Can Maintain Human-Centric Artiﬁcial Intelligence 3 Given that these