4/8/01 CSC309 Miller 1

Helen Nissenbaum

Communications of the ACM

January 1994/Vol.37, No.1

Computing and Accountability

4/8/01 CSC309 Miller 2

Computing and AccountabilityFull agreement that computers are playing an ever increasing role in our lives.

A community that insists on accountability, in which agents are expected to answer for their work, signals esteem for high-quality work, and encourages diligent, responsible work.

When the community does not insist on accountability no one answers for harms and risks. They are regretted but seen simply as unfortunate accidents ---

Accountability in the computing field is becoming increasingly elusive.

4/8/01 CSC309 Miller 3

Why Accountability Needed

Those who are held accountable for failures, safety, risk, and harm are most driven to prevent them (and are in the best position to prevent them).

We want the protection that accountability gives not only to reduce our risk of major disasters (life threatening, large monetary loss, etc.) but to reduce the occurrences of malfunctions that cause individual losses of time, convenience, and contentment which negatively impact quality of life.

4/8/01 CSC309 Miller 4

Two Conditions Needed forResponsibility

(1) Causal Condition: The person's actions (oromissions) must have caused the harm.

Write and release a virus that is designed to destroy files.

(2) Mental Condition: The person must have intended, or willed the harm.

Write and release a virus that is designed to destroy files.

5/16/09 CSC309 Miller 5

Causal Condition

Don't need to be "the" cause but can be merelya significant causal factor among many.

Includes actions done while acting under the orders of someone else.

When someone is accountable for a harm then they are "liable" to punishment (pay a fine, get censored, see income reduced, get fired)or is "liable" to compensate the victim.

4/8/01 CSC309 Miller 6

Liability

It is not always the individual who did the harm that gets to pay. Parents pay for damages done by their children and insurance companies pay for accidents caused by their policy holders.

5/16/09 CSC309 Miller 7

Strictly Liable

To be strictly liable means you pay for harm even if the fault condition is not met. For example,owners of dangerous animals are strictly liable for the action of their animals (even when there is no proof of negligence.) Strict liability is applicable in product liability cases against manufacturers, who are legally responsible for injuries caused by defects in their products, again, even if they were not negligent.

5/16/09 CSC309 Miller 8

Why Strict Liability?

Strict liability serves the public interest by providing sellers of consumer products and owners of potentially dangerous property to take extraordinary care. Strict liability also assures compensation for victims and establishes who is at fault.

10/22/01 CSC309 Miller 9

Mental Condition (intent)"the Fault Condition"

This condition is often weakened to include unintended harm if the harm was the result of negligence, carelessness, or recklessness.

In general, if a person fails to take precautions of which he is capable, and that any reasonableperson with normal capacities would have taken in those circumstances, then he is not excused from blame merely because he did not intend the outcome.

The Internet Worm was not designed to do harm but Morris is still responsible for the harm created.

4/7/01 CSC309 Miller 10

Four Barriers to Accountability

1. The Problem of Many Hands

2. Bugs

3. It's the Computers Fault

4. Ownership Without Liability

5/16/09 CSC309 Miller 11

The Problem of Many Hands

Modern software is usually not produced by a single programmer working in isolation. Usually production is the result of a team effort of diverse individuals that might include designers, engineers, writers, programmers, psychologist, graphic artist, managers, and sales people.

4/8/01 CSC309 Miller 12

The Problem of Many Hands (Cont.)

Don't confuse the obscuring of accountability due to collective action, with the absence of blameworthiness.

If we consistently respond to complex cases by not pursuing blame and responsibility, we are effectively accepting agentless mishaps and a general erosion of accountability.

4/8/01 CSC309 Miller 13

Bugs

We accept bugs as a fact of life. Even when we factor out sheer incompetence, bugs in significant numbers are endemic to program- ming. They are the natural hazards of any substantial system.

But blind acceptance of bugs as inevitablehazards of programming leads to the conclusion that it is unreasonable to hold anyone accountable for flaws in a system.

4/8/01 CSC309 Miller 14

It's the Computers Fault

Easy to blame the computer because it is often the action of the computer that is the more direct causal antecedent. (We don't, however, pick on the bullet but focus on the person who fired the gun).

The computer is a different inanimate object in that we attribute to them mental properties, such as intentions, thought, preference, desires, all of which are factors that help make people responsible.

4/8/01 CSC309 Miller 15

Ownership Without Liability

We spent a lot of time talking about copyright (or patent) protection for software as a means to provide the benefits of ownership to its producer.

For software, the owners (usually the producers) are in the best position to directly affect the quality of the product released to the public but the trend is to demand maximal property protection while denying accountability.

4/8/01 CSC309 Miller 16

Ownership Without Liability (Cont.)

Read a software license. "Will not be liable for any direct, indirect, special, incidental, or consequential damages resulting from any defect in the software or its documentation, even if advised of the possibility of such damages."

4/8/01 CSC309 Miller 17

Keep Accountability Distinct from Liability to Compensate

Accountability and liability are two different things.

Liability deals with who will compensate the victim. In the case of "Many Hands" does the entity with deep pockets take everyone else off the hook?

Accountability is based upon causal and fault (intent) actions.

4/8/01 CSC309 Miller 18

Accountability/LiabilityExample

Great example of this difference is the decision by Ford executives to look at liability projections in deciding not to improve the design of the Pinto where considerations of accountability to society would have produced a different set of actions.

4/27/09 CSC309 Miller 19

Another Accountability/LiabilityExample

7/8/99 A jury ordered that General Motors Corp. pay $107 million in compensatory damages and $4.8 billion in punitive damages to six people injured in a 1993 accident involving their Malibu.

The problem was that the gas tank had been placed dangerously close to the rear bumper.

GM studies had estimated cost of fixing the problem $8.59 per car while the cost of dealing with lawsuits involving deaths at only $2.40.

4/8/01 CSC309 Miller 20

Clarify and Vigorously Promote a Substantive Standard of Care

Things that we know that promote quality are: simpler design, a modular approach, user involvement and review, meaningful quality assurance, independent auditing and testing, built-in redundancy, and excellent documentation (training). If these good practices could become the standard, then you have a basis for distinguishing between a bug due to inadequate practices on the part of a programmer and those that occur in spite of all reasonable efforts to eliminate them.

4/8/01 CSC309 Miller 21

Impose Strict Liability for Defective Consumer-oriented Software

Making the producers of software strictly liable is a way of assuring that the public is protected against the potential harms of risky artifacts and property.

Software seems, therefore, to be precisely the type of artifact for which strict liability is necessary --- assuring compensation for victims, and sending an emphatic message to producers to of software to take extra-ordinary care to produce safe and reliable systems.

5/16/09 CSC309 Miller 22

Nissenbaum Conclusion“In the twentieth century B.C. the Code of Hammurabi declared that if a house collapsed and killed its owner, the builder of the house was to be put to death. In the twentieth century A.D. many builders of computer software would deny responsibility and pass the "entire risk" to the user. While the centuries have placed a distance between the harsh punishments meted out by Hammurabi’s Code and contemporary legal codes, the call for accountability remains a standard worth restoring and one whose achievement would be a source of professional pride.”

4/8/01 CSC309 Miller 23

Impose Strict Liability for Defective Consumer-oriented Software Might

Happen

4/8/01 CSC309 Miller 24

EU Software Company Accountability

3/22/09 CSC309 Miller 25

Next Slide from Chapter 6Accountability was part of the discussion in Chapter 6 where we were looking at the topic of anonymity.

6/25/02 CSC309 Miller 26

Another Viewpoint on AnonymityDavenport, David, “Viewpoint”, CACM, April 2002/Vol45.,

No. 4, pp33-35.

Anonymity, so the argument goes, ensures governments cannot spy on citizens and thus guarantees privacy and free speech.

The view is fundamentally mistaken; by allowing anonymous communication we actually risk an incremental breakdown ofthe fabric of our society. The price of freedom is not anonymity, but accountability.Without accountability we loose all recourse to the law and hence risk our very freedom.

6/25/02 CSC309 Miller 27

Another Viewpoint on AnonymityDavenport, David, “Viewpoint”, CACM, April 2002/Vol45,

No. 4, pp. 33-35.

In circumstances where people can be largely anonymous, and the threat of punishment is thus minimal, they find it easy to justify to themselves actions againstthose they perceive as outsiders or enemies.

Experience suggests that a society that relies solely on the goodwill and conscience of its citizens would be unlikely to succeed in assuring justice.

12/25/08 CSC309 Miller 28

Two Conditions Needed forResponsibility (Aristotle)

To receive praise or blame for an action

(1) The action must not have been done under compulsion (action must be voluntary)

(2) The actor must have had relevant knowledge about the circumstances

Joseph Koterski, “The Ethics of Aristotle”,The Teaching Company, 2001

6/20/02 CSC309 Miller 29

An example of concern6/17/02  Push for Software Manufacturing Liability Support is growing for software companies to be held to the same liability standards as other manufacturing businesses.  Microsoft, with its plethora of software holes and "deep pocket[s]" is a likely target for a liability suit.  Air Force CIO John Gilligan says patches and fixes for the Microsoft products they use have cost more thanthe software itself. http://www.usatoday.com/life/cyber/tech/2002/06/17/microsoft-security.