Visual Communication: theoretical issues

Research Seminar Master Informatiekunde13/09/06

Leonie Bosveld-de SmetHumanities Computing, RuG

04/22/23 Visual Communication 2

Question Pictures can be worth ten thousand words.

Visual representations can be superior to verbal ones in thinking (problem solving), teaching, and communication.

Can benefits observed in other areas (esp. science) be exploited in other fields?

04/22/23 Visual Communication 3

Outline talk

Characteristics of graphical versus sentential languages

Facilitating effects of visual representations

Visual communication: a semantic view

Conclusion and discussion

04/22/23 Visual Communication 4

Theoretical concepts Key words:

Representation system Interpretation function Form of representation Perception of representation Medium modality

Piece of information

Structured information

Representation system

Form of representationlink

04/22/23 Visual Communication 5

Media and modality contrasts Stenning and Inder(1995) “Medium” how is a representation perceived?

“Modality” how is a representation interpreted?

04/22/23 Visual Communication 6

Graphical vs Linguistic Representations

Stenning and Inder (1995) the more constrained, the less expressive, the more

tractable

Stenning and Oberlander (1995) limited abstraction, aided “processibility”

Shimojima (2004) Free ride properties Auto-consistency Specificity Meaning derivation properties

04/22/23 Visual Communication 7

Free Ride Property

Defeated(Jon,Bob) & Lost_to(Ken,Bob)FOL

Jon defeated Bob and Ken lost to Bob.English

Jon Bob KenPD

Express information:Jon defeated Bob.Ken lost to Bob.

04/22/23 Visual Communication 8

SpecificityExpress information:Jon defeated Bob.Ken defeated Bob.

Defeated(Jon,Bob) & Defeated(Ken,Bob)FOL

Jon defeated Bob and Ken defeated Bob.English

Jon Ken BobPD Ken Jon Bob? ?

04/22/23 Visual Communication 9

Meaning Derivation PropertyExpress information:Jon defeated Bob Bob defeated KenGil defeated Jon Ken defeated Ron

Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron)FOL

Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron.English

Gil Jon Bob Ken RonPD

04/22/23 Visual Communication 10

Benefits of graphical systems Automaticity of inference Ease of consistency inferences Difficulty to express “abstract” information Richness of semantic content

“Transparent” problem solving Computational offloading Amplification of cognition

04/22/23 Visual Communication 11

Monk Puzzle (Winn, 1987)“A monk went to the temple at the top of a holy mountain to meditate and pray. He started out early one morning along the path that led up to the temple. Because he was an old man, and the way was steep and arduous, he frequently slowed his pace, and even sat and rested a while beside the path. Toward evening, he came to the temple at the top.After several days of meditation and prayer, it was time for him to leave. Early in the morning, he set off back down the path. Again, he frequently changed his pace and rested by the way. He arrived back at the bottom in the evening.Show that there is one single point on the path up the mountain where the monk will be at precisely the same time both when he goes up and when he comes down.”

04/22/23 Visual Communication 12

“See” the solution

04/22/23 Visual Communication 13

Larkin and Simon (1987) “Why a Diagram is (Sometimes) Worth Ten Thousand

Words”

Diagrammatic vs. Sentential representations Informational equivalence, computational offloading

Pulley problem Geometry problem

Grouping of information Exploitation of space Easy perceptual inferences

04/22/23 Visual Communication 14

Using vision to think Kosslyn (1994)

Scatter plots...employ point symbols (such as dots, small triangles, or squares) as content elements. The height of each point symbol indicates an amount. These displays typically include so many points that they form a cloud; information is conveyed by the shape and the density of the cloud

Simultaneous presentation of local information and global information implied by the local information.

04/22/23 Visual Communication 15

Visual Communication

Raw Data

Data Tables

Visual Structures

Views

Data Transformations

Visual Mappings

View Transformations

04/22/23 Visual Communication 16

Semantics of pictures:Wang’s view

Graphical Domain Application Domain

12

3

1: geometrical characterization of pictures

2: interpretations: graphical entities – application domain objects

3: specifications: application domain concepts – picture classes

04/22/23 Visual Communication 17

Natural vs. Unnatural Link

a

b

c

ab

c

04/22/23 Visual Communication 18

Misleading Link

Bill

Charles

Gregory

04/22/23 Visual Communication 19

Kinds of visual communication Via interpretations:

Illustration Demonstration Reasoning

Via picture specifications: Geometric constraint maintenance Design Computational art

04/22/23 Visual Communication 20

RequirementsMeaningful and Easily Tractable diagrams

Clear delimitation of critical information

Description of information by well-defined notions and properties (elements of application domain)

Graphical devices suitable for representation of information (elements of graphical domain)

Explicit, natural, not misleading link between graphical domain and application domain

04/22/23 Visual Communication 21

Graphical domain Research in information visualization and diagrams and

diagrammatic reasoning

Bertin Tufte Shneiderman

Blackwell Hegarty Scaife & Rogers Novick

04/22/23 Visual Communication 22

Possible application areas Language instruction

Linguistic research

History instruction

Historical research

04/22/23 Visual Communication 23

Towards an automated system for diagram research

“When we saw the spaceship, we stopped the car.”

e1: see

e2: stop

t t’ now

e2: ring

s: making lunch

e1: make lunch

t t’ now

t now

“Two men delivered the sofa. I had already paid for it.”

“David was making lunch when the phone rang.”

e1: deliver

e2: pay s: have paid

04/22/23 Visual Communication 24

3-D Syntactic trees