Of Gods and Goats Elliot J. Crowley Andrew Zisserman

Visual Geometry Group, University of Oxford

Problem and Contributions

0. Objective

• Given a classical dataset with no visual annotation, automatically learn who the gods and animals are and provide bounding box annotation for them.

The Beazley Archive of Classical Art

Example Entry 1 - Zeus

A. Theseus and minotaur, with youths and women; B. Zeus Seated, between women and onlookers;

Example Entry 2 - Dionysos

A. Dionysos with vine and drinking horn between satyr and maenad with oinochoe;

B. Apollo, playing kithara, between Artemis and leto, deer;

Method

A weakly supervised learning approach proceeding in stages:

• a) Text mining to obtain visually consistent clusters.

• b) Multiple instance learning (MIL) to find god regions.

• c) DPM training to locate god across database.

• Goal: To find a way of using text to find visually consistent clusters. • Observation: Keywords describes pose or distinguishing object. • Approach: Mine pairs of gods and keywords.

Cluster Example 1 – Zeus Seated 1.zeus seated on stool between winged god and hermes with bag between draped men cock 2.zeus seated with spear on chair with panther hermes draped men with spears 3.birth of athena zeus seated on chair with bird head and draped youth apollo with kithara dionysos with ivy wreath eileithyiai ares device star 4.birth of athena zeus seated on chair with owl and staff draped youth hermes poseidon ares device tripod 5.man zeus seated between youths with spears and draped youths

b) Multiple Instance Learning

Results

• Windows obtained for each cluster are used to train a DPM. • The DPM is then used in a standard sliding window approach on all

images associated with vase entries that contain the god’s name to detect instances.

Quantitative results Qualitative results

ASIRRA Challenge III.

Which image contains Zeus? Where is he located in the image?

How do we distinguish between Dionysos and Artemis?

a) Text Mining

Vase 1 2 3 4 5

Side A B

Cluster Example 2 – Apollo Kithara 1.apollo playing kithara between muses 2.apollo playing kithara between hermes and athena seated 3.athena mounting chariot apollo playing kithara hermes all named 4.wedded pair in chariot woman goddess apollo with kithara hermes 5.wedded pair in chariot dionysos apollo with kithara goddesses one with torches hermes

1 2 3 4 5

A B

Cluster Example 3 – Herakles Lion 1. herakles and the lion warrior iolaos man and youth both draped woman 2. herakles and the lion athena iolaos 3. herakles and the lion between draped men some with spears and women with wreaths 4. herakles and the lion between iolaos with club and athena device leaf 5.herakles and the lion between woman and athena bow and quiver suspended

1 2 3 4 5

A B

• Goal: To find the region within visually consistent clusters containing the god.

• Observation: The god is large and often near the centre of an image. • Approach: Sample windows around image centres and perform steps:

MIL for mining • Observation: Correct windows will find other correct windows. • Train LDA detector using HOG features for each window. • Rank windows by the number of times the LDA makes a detection on

the positive bag before a detection is made on the negative bag. • Retain top windows. Positive bags consists of vase sets from the cluster. Negative bags consists of vase sets from the whole database.

Example sample windows for Zeus Seated (crosses indicate negatives):

Retained windows (most negatives are now gone):

Reconciliation • Observation: The LDAs for the top windows should agree with each

other. • Run LDA for each top window on the images for other top window. • Re-rank top windows by counting the number of overlaps. • Retain top windows and re-align using overlaps.

Obtain additional windows • Observation: The supervision is strong enough to find other instances. • Average re-aligned top windows to train a new LDA. • Run LDA over the cluster to obtain new windows.

Example windows:

An outlier. Perhaps there were horses in the positive bag.

The outlier does not overlap with the other windows and is re-ranked and removed.

Apollo Kithara Athena Device

Dionysos Kantharos Herakles Lion

Zeus Seated Mules

And of course GOATS

• 50,000 vase entries. • Each entry consists of images with text

annotations (see below). • 120,000 images across entries. • The number of vases associated with each god is

given to the right.

Weakly Supervised Learning of Figurative Art

c) Deformable Parts Models