Inferential and Expressive Capacities of Graphical Representations

Inferential and Expressive Capacities of Graphical Representations

Tutorial

Diagrams 2004University of Cambridge March 23, 2004

Survey and Some Generalizations

School of Knowledge ScienceJapan Advanced Institute of Science & Technology

ATR Media Information Science Labs

Atsushi Shimojima

Purpose

• To understand three concepts useful to capture the inferential-expressive capacities of many graphical systems.

• In my personal terminology:– Free ride– Over-specificity– Derived meaning

Why Important?

These concepts are very often alluded to in the

literature

Their ranges of application never explicated in full

Their exact contents seldom defined

but

Different people used different terms to refer to them, sometimes missing

important connections of their ideas and findings

thus

Plan for the hour

Free ride

Over-specificity

Derived meaning

ExamplesAnalysis & definition Connections

Outstanding questions

1. Free Ride

A toy example:

• Jon, Ken, Gil, Bob, and Ron run races• of the kind with no “ties” in arrival

Suppose:

• Different ways of expressing the information that Jon defeated Bob :

Compare:

Defeated(Jon,Bob)

Jon defeated Bob.

Jon Bob

Defeated(Jon,Bob)

Jon defeated Bob.

Jon Bob

Atomic sentence of a first-order language (FOL) with:

• two-place predicate Defeated • its arguments Jon and Bob

Defeated(Jon,Bob)

Jon defeated Bob.

Jon Bob

Representation of PD system (position diagrams) where:

Horizontal relation of names indicate arrival order of people.

Defeated(Jon,Bob)

Jon defeated Bob.

Jon Bob

Sentence of English describing the arrival order of two people.

PD system (a bit more precisely)

Syntactic rules:– Two or more of the

names “Jon”, “Ken”, “Gil”, “Bob”, and “Ron” appear in a horizontal row.

– The same name appears at most once.

Semantic rules:– If the name X

appears to the left of the name Y, the bearer of X defeated the bearer of Y.

Defeated(Jon,Bob)FOL

Jon defeated Bob.English

Jon BobPD

Look Similar

But behave quite differently when more information expressed

Difference 1

Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL

Jon defeated Bob and Ken lost to Bob.English

Jon Bob KenPD

Express information:Jon defeated Bob.Ken lost to Bob.

Jon Bob KenPD

The PD system expresses an additional piece of information…


Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL

Jon defeated Bob and Ken lost to Bob.English

…while FOL and English don’t.


• In PD, expressing certain sets of information results in the expression of additional, consequential information.

Free rides=

Another example: Venn diagrams

Express information:All As are Bs.No Bs are Cs.

As

Bs Cs

As

Bs Cs

As

Bs Cs

Expressing certain sets of information results in the expression of additional, consequential

information

Another example: Euler diagrams

Express information:A ⊂ BC ∩ B = φ

A

B

CA

B

Expressing certain sets of information results in the expression of additional, consequential

information

Another example: Maps

Express information:B’s house is in front of F’s house across the river.

SCD1PEFLA 7MW SCD1PEFLA 7MWBExpressing certain sets of information results in the expression of additional, consequential

information

Of course we cannot always do the manipulations in our heads: we may have to draw a diagram on paper, or re-arrange parts of a scale model, in order to see the effects.…The main point is that the ability to apply such subroutines to parts of analogical configurations enables us to generate, and systematically inspect, ranges of related possibilities, and then…to make valid inferences, for instance about the consequences of such possibilities. (p. 220.)

Sloman (1971)

Diagrams are physical situations….As such, they obey their own set of constraints…By choosing a representational scheme appropriately, so that the constraints on the diagrams have a good match with the constraints on the described situation, the diagram can generate a lot of information that the user never need infer. Rather, the user can simply read off facts from the diagram as needed.

Barwise & Etchemendy (1990)

We have seen that formally producing perceptual elements does most of the work of solving the geometry problem. But we have a mechanism---the eye and the diagram---that produces exactly these “perceptual” results with little effort. We believe the right assumption is that diagrams and the human visual system provide, at essentially zero cost, all of the inferences we have called “perceptual.” As shown above, this is a huge benefit. (p. 99.)

Larkin & Simon (1987)

Other conceptions

• Non-deductive representation systems where the operation of the construction process entails the “making” of the inferences (Lindsay 1988, p. 112)

• Inference by recognition (Novak 1995)

• Inference by inspection and transformation (Olivier 2002, p. 72--74)

• Emergence effect (Kulpa 2003, p. 90)

• Emergent properties (Koedinger 1992, as cited by Olivier 2001)

• Emergent relations (Chandrasekaran, Kurup, and Banerjee 2004)

But

What, more exactly, is the “free-ride” capacity?What is the general condition---semantic mechanism---for a system to have that property?

Basic Assumption

indicates

a property

a property Represented object Y

Representation X

A representation X expresses information about the represented object Y by having a property that indicates the corresponding property of Y.

Example: PDs

indicates

[the name “Jon” appears to the left of the name “Bob”]

[Jon defeated Bob] a particular running race

a particular position diagram

Jon Bob

Example: Euler diagrams

indicates

[the circle “A” appears inside the circle “B”]

[A B]⊂a particular groupof objects

a particular Euler diagram

AB

Condition for Free Ride: PD system

The name “Ken” appears to the right of the name “Bob”

The name “Jon” appears to the left of the name “Ken”.

The name “Jon” appears to the left of the name “Bob”

Jon defeated Ken.Jon defeated Bob. Ken lost to Bob.

indicates indicates indicates

constraint

constraint

Condition for Free Ride: Euler Diagram

A circle “C” and the circle “B” has no overlap.

The circle “C” and the circle “A” has no overlap.

A circle “A” appears inside a circle “B”.

C ∩ A = φA B⊂ C ∩ B = φ


constraint

constraint

Condition for Free Ride: General


constraint

constraint………

………

Constraints on representations themselves track constraints in the represented domain.

(Shimojima 1996a, 1996b)

Thus:A system with a free-ride property supports deductive inference through physical manipulation of representations on an external display, not in the head.

A (paradigm) case of distributed cognition

Connection: AI systems

• WHISPER for the prediction of the collapsing of objects (Funt 1980)

• REDRAW I & II for the deflection shape problem (Tessler, Iwasaki, and Law 1995a, 1995b)

• KAP for the prediction of the movements of cam-follower pairs and meshing gears (Olivier, Ormsby, and Nakata 1996)

• DRS component (Chandrasekaran et al. 2004)

Some AI systems utilize the free-ride capacities of graphical systems by installing some manipulation-inspection abilities on diagrams.

Connection: graphical simulations

• Dynamic behaviors of strings, flexible rods, and rings, falling in free space, etc. (Gardin and Meltzer 1995)

• Liquid behaviors in and out of containers with complex shapes (Decuyper, Keymeullen, and Steels 1995)

Some graphical simulations can be considered free rides in an extended sense, combining computer-controlled dynamic constraints with geometrical-topological constraints on graphics.

Connection: studies of sketching in designFree-ride capacities may be an essential factor of the utility of pictorial sketches in design process.

Each move is a local experiment which contributes to the global experiment of reframing the problem. Some moves are resisted (the shape cannot be made to fit the contours), while others generate new phenomena. As Quist reflects on the unexpected consequences and implications of his moves, he listens to the situation’s back talk, forming new appreciations which guide his further moves. (p. 94.)

Schoen (1982)

Thus the drawing represents a sort of hypothesis or “what if” tool. With a plan, for example, the architect can say, what if the kitchen were here, the dining-room next to it and the living-room there? How could I then organize the entrance and the stairs? (p. 242 , colored emphasis by me.)

Lawson (1997)

• Goldschmidt (1994)– “One reads off the sketch more information than was invested in i

ts making” (p. 164)– Such reading-off of unexpected reading-off as an essential step in

“interactive imagery” in design

• Suwa, Gero, and Purcell (2000)– Relationship between unexpected discoveries in sketches and inv

ention of new design requirements

Also:

Warning: Recognition problem

• “Cheap rides” (Gurr, Lee, and Stenning 1998, Gurr 1999)• Expertise in diagram construction to facilitate the recogniti

on of useful consequences (Novak 1995)

Free rides only guarantee the expression of consequential information in the representation, not its recognition by the user.

2. Over-Specificity

Difference 2

Express information:Jon defeated Bob.Ken defeated Bob.

Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL

Jon defeated Bob and Ken defeated Bob.English

Jon Ken BobPD Ken Jon Bob? ?

Jon Ken BobLD Ken Jon Bob? ?

The PD system can’t express the info without additional info…


…while FOL and English can.

Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL

Jon defeated Bob and Ken defeated Bob.English


• In PD, certain sets of information cannot be expressed without expressing additional, non-warranted information.

Over-specificity=

Another example: Euler diagrams

Express information:A ⊂ BC ∩ B ≠ φ

A

B

A

B

C

Certain sets of information cannot be expressed without expressing additional, non-warranted information.

How do you placeThis circle?

CC

C

Another example: Maps

Express information:K’s house is between A’s house and B’s house.

SCD1PEFLA 7MWBCertain sets of information cannot be expressed without

expressing additional, non-warranted information.

K

Where do you placethis icon?

``the demand by a system of representation that information in some class be specified in any interpretable representation'' (p. 98)

Stenning and Oberlander (1995)

Specificity

=

However, there are some ways in which a picture can often carry too much information or indicate a degree of precision which may be inappropriate….It would be difficult to construct a drawing which did not suggest other features of the form of the finished product which might restrict a future designer. (p. 242.)

Lawson (1997)

[In geometrical proofs,] though we do not for the purpose of the proof make any use of the fact that the quantity in the triangle (for example, which we have drawn) is determinate, we nevertheless draw it determinate in quantity (As cited by Kulpa 2003, p. 101.)

Aristotle (350 B. C. E.)

Other conceptions

• Analog property of representation systems as opposed to digital property (Dretske 1981)

• Smaller degree of discretion (Norman 2000, p. 110)

• Particularity feature (Kulpa 2003, p. 96, p. 101)

Not in the sense of Goodman

(1982)!

Analysis of over-specificity: Euler Diagram

A circle “C” and the circle “B” has some overlap.

A circle “A” appears inside a circle “B”.

A B⊂ C ∩ B ≠ φ


constraint

C ∩ A ≠ φC ∩ A =φ

indicates

The circle “C” and the circle “A” has some overlap.

The circle “C” and the circle “A” has no overlap.

constraint

Analysis of over-specificity: Position Diagram


constraint

indicates

(1’)

(1’) The name “Jon” appears to the left of the name “Bob”.

(2’)

(2’) The name “Ken” appears to the left of the name “Bob”.

(2)

(2) Ken defeated Bob(1) Jon defeated Bob

(1)

(4’) The name “Ken” appears to the left of the name “Jon”.

(4’)

(3’) The name “Jon” appears to the left of the name “Ken”.

(3’)

constraint

(3)

(3) Jon defeated Ken

(4)

(4) Ken defeated Jon

Analysis of over-specificity: General


constraint

constraint

indicates

………

………

………

………

(Shimojima 1996b)

Thus:

The expressive efficacy of a system depends on “what it allows you to leave unsaid” as well as “what it allows you to say” (Levesque 1988, p.370).

A system with an over-specificity property prohibits the exclusive expression of certain (small) sets of information even when they are in the system’s expressive coverage.

“Knowledge expressed in propositional format can determine part of the state of the world while conveniently leaving other parts undetermined.” (Ioerger 1992, as cited by Kulpa 2003, p. 101.)

In contrast:

“[Descriptions] need only express their intended information; they can leave unsaid and indeterminate some aspect of what is described; they need not convey more than is required.” (Norman 2000, p. 110.)

Also:Expressing such a set of information in the system produces a representation with semantic contents beyond that set.

Accidental features

=

“Having demonstrated that the three angles of an isosceles rectangular triangle are equal to two right ones, I cannot therefore conclude this affection agrees to all other triangles which have neither a right angle nor two equal sides” (Introduction, paragraph 16).

Berkeley (1710)

More recent discussion:Problem of “unintended exclusion” (Giaquinto 1993) Problem of “over-looked divergence” (Kulpa 2003)

• Hyperproof (Barwise and Etchemendy 1994)• System for geometry proof (Luengo 1995, Winterstein et

al. 2000)

An over-specificity property of a representation system poses a challenge to some (all?) attempts to build a formal deductive system based on that system.

Thus:

3. Derived Meaning

Difference 3

Express information:

Jon defeated Bob.Gil defeated Jon.

Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL

Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English

Gil Jon Bob Ken RonPD

Bob defeated Ken.Ken defeated Ron.

Gil Jon Bob Ken RonPD

You can count the number of names to find out the count of people satisfying a certain condition…






…while you can’t, in FOL and English.






The count of names don’t mean the count of people.




• In PD, some additional meaning relation holds that does not hold in FOL and English.

• Moreover, that relation is derivative in that it is not written in basic semantic rules.

Derivative Meaning=

Another example: Tables

Gil Jon Bob Ken Ron

Count of circles in a row means the count of people to the right

Ron ÅõÅõÅõÅõGil ÅõKen ÅõÅõBob ÅõÅõÅõJon RonGilKenBobJonCount of circles in a column means the count of people to the left

Additional meaningrelation

Another example: Scatter plots

151050 20100151050 20100

I IIThe shape formed by dots means a general fact about the distribution:

• Existence of correlation, • Its strength, • Existence of an exceptional instance, etc.


From Tufte (1983)

Scatter plots...employ point symbols (such as dots, small triangles, or squares) as content elements. The height of each point symbol indicates an amount. These displays typically include so many points that they form a cloud; information is conveyed by the shape and the density of the cloud. (p. 46.)

Kosslyn (1994)

Another example: Data mapsFrom Tufte (1983)


The concentration of dots along the Broad Street band means a concentration of deaths along Broad Street.

Another example: Node-edge graphs


Concentration of lines on one node means that the corresponding station is a “hub” of the subway system.

Another example: Node-edge graphs

K

C

E

H

G

H

D

A

F

JAdditional meaningrelation

“H” node’s being lower in the same group as “K” node means that K is an ancestor of H.

Olivier (2001)

Analysis of derived meaning: PDs

(1) At least two people defeated Jon

indicates

constraint

indicates

(1*) At least two names appear to the left of the name “Jon”

constraint

…

…

indicates indicates

A particular way (1*) holds

A particular way (1) holds

Enlarged view

The name “Gil” is to the left of the name “Jon”

The name “Bob” is to the left of the name “Jon”

Gil defeated Jon Bob defeated Jon

indicates indicates

A particular way (1*) holds

A particular way (1) holds

(1) At least two people defeated Jon

constraint

(1*) At least two names appear to the left of the name “Jon”

constraint

Analysis of Derived Meaning: General

indicates

constraint

indicates

constraint

… … …

… … …

indicates indicates

A particular way holds

A particular way holds

(Shimojima 1999, 2002)

Thus:A representation system with a meaning derivation property allows the simultaneous presentation of local information and global information implied by the local information.

Concerning the original numbers, they can be easier and more accurately read off from a list of numbers, without the expense of producing that graph. What such graphs are really for is something different---namely, a possibility to see at a glance some general conclusion, i.e., a result of some reasoning that follows from the interaction of these numbers. (p. 111)

Kulpa (2003)

aa

'96'97'98'99-100102030mln $

Indeed, the central purpose of many scientific diagrams is to depict relationships and interactions….If students are to understand such diagrams, they need to be able to do more than just decode the symbols used. They must also be able to uncover and assimilate salient relationships between the symbols that constitute a diagram and appreciate how these relationships map onto the real-world situation being represented. (p. 28.)

Lowe (1989)

Petre (1995): Generally critical about the insensitive use of graphics in programming environment, but mentions “gestalt response” as an “informative impression of the whole that provides insight into the structure” (p. 42) and admits it as a potential benefit of graphics.

Also:

Ratwani, Trafton, and Boehm-Davis (2003): Assumes the difference of global/trend reading and local reading, and goes on to demonstrate that different mental operations are involved in them.

Connection: expert reading of graphicsCorrectly assessing and recognizing derived meaning may be a major component of expertise in reading graphical representations.

For example:

• Lowe (1989): Compared the professional meteorologists’ way of inspecting (incomplete) meteorological charts with non-meteorologists’ and find their greater appreciation of large-scale patterns of organization.

• Winn (1991): Discusses expertise of reading topographical maps as perceptual chunking of contours to form larger features, such as valleys.

Summary

Derived meaning

Expertise in graphics readingInformation graphics/

visualization

Graphics semantics

Free ride

Distributed cognition

Graphical simulation

AI systems with diagrams

Design studies

Over-specificity

Formal deduction systemsJustificatory status of

diagrammatic proofs

Design studies

Useful and central set of concepts

Free ride

Over-specificity

Derived meaning

Mental transformation of graphicsNarayanan, Suwa & Motoda (1995), Schwartz (1995), Trafton & Trickett (2001), Shimojima (2003)

Perceptual chunking of graphical elementsLowe (1989), Anderson & Koedinger (1992), etc.

Spatial indexing of informationLarkin & Simon (1989), etc.

Low-encoding diagramsCheng (2003) etc.

Auto-consistencyBarwise & Etchemendy (1995), Stenning & Inder (1995), Lemon & Pratt (1997)

Concepts not covered today

Documents

Inferential and Expressive Capacities of Graphical Representations