Teaching Machines to Learn by Metaphors

Omer Levy & Shaul MarkovitchTechnion – Israel Institute of Technology

Concept Learning by Induction

Few Examples

Transfer Learning

Target (New)

Source (Original)

Define: Related Concept

Transfer Learning Approaches

• Common Inductive Bias

• Common Instances

• Common Features

Different Feature Space

Example

0 2 3-3 -2

Example

0 2 3-3 -2

0 4 9

Example

0 2 3-3 -2

0 4 9

𝑥𝑠=𝑥𝑡2

Common Inductive Bias

0 2 3-3 -2

0 4 9

Common Inductive Bias

0 2 3-3 -2

0 4 9

Common Instances

0 2 3-3 -2

0 4 9

Common Features

23

-3-2

4 9

𝑥𝑠=𝑦𝑡2

New Approach to Transfer Learning

Our Solution: Metaphors

Metaphors

Target (New)

Source (Original)

Concept Learner

Metaphor Learner

𝜇 h𝑠

Source

Target

+/-𝑥𝑡 𝑥𝑠

h𝑡 (𝑥𝑡 )=h𝑠 (𝜇 (𝑥𝑡 ))

is a perfect metaphor if:

1. is label preserving

2. is distribution preserving

Theorem

If is a perfect metaphor- and -

is a source hypothesis with error- then -

is a target hypothesis with error

The Metaphor Theorem

If is an -perfect metaphor- and -

is a source hypothesis with error- then -

is a target hypothesis with error

Redefine Transfer Learning

Given source and target datasets, find a target hypothesis such that is as small as possible.

Redefine Transfer Learning

Given source and target datasets, find an -perfect metaphor such that is as small as possible.

Metaphor Learning Framework

h

Concept Learning Framework

Search Algorithm

Hypothesis Space

Evaluation Function

Data

𝜇

Source

Target

Metaphor Learning Framework

Search Algorithm

Metaphor Space

Evaluation Function

Metaphor Evaluation

Metaphor Evaluation

1. is label preserving

2. is distribution preserving

Metaphor Evaluation

1. is label preservingEmpirical error over target dataset

2. is distribution preservingStatistical distance between and

Metaphor Evaluation

𝑆𝐷 (𝜇 (𝑥𝑡 ) ,𝑥𝑠 )

Metaphor Evaluation

𝑆𝐷 ¿

Metaphor Evaluation

𝑆𝐷 (𝜇 (𝑥𝑡− ) , 𝑥𝑠−)

Metaphor Evaluation

𝑆𝐷 ¿

Metaphor Spaces

Metaphor Spaces

• General

• Few Degrees of Freedom

• Representation-Specific Bias

Geometric Transformations

Я R

Dictionary-Based Metaphors

cheese queso

Linear Transformations

Which metaphor space should I use?

Automatic Selection of Metaphor Spaces

Which metaphor space should I use?

Occam’s Razor

Automatic Selection of Metaphor Spaces

Which metaphor space should I use?

Structural Risk Minimization

Occam’s Razor

Automatic Selection of Metaphor Spaces

Which metaphor space should I use?

Automatic Selection of Metaphor Spaces

ℳ1

ℳ2

ℳ3

ℳ 4

Automatic Selection of Metaphor Spaces

ℳ1

ℳ2

ℳ3

ℳ 4

𝜇1

𝜇2

𝜇3

𝜇4

Automatic Selection of Metaphor Spaces

ℳ1

ℳ2

ℳ3

ℳ 4

𝜇1

𝜇2

𝜇3

𝜇4

60 %

9 0 %

91 %

7 0 %

Empirical Evaluation

Reference Methods

Baseline• Target Only• Identity Metaphor• Merge

State-of-the-Art• Frustratingly Easy Domain Adaptation

– Daumé, 2007

• MultiTask Learning– Caruana, 1997; Silver et al, 2010

• TrAdaBoost– Dai et al, 2007

Digits: Negative Image

Digits: Negative Image

𝜇 (𝑥𝑡 )=1−𝑥𝑡

Digits: Negative Image

Digits: Higher Resolution

Digits: Higher Resolution

→

Digits: Higher Resolution

Wine

Wine

Qualitative ResultsTransfer Learning

TaskTarget

InstanceTarget Sample Size

1 2 5 10

Digits: Negative Image

Digits: Higher Resolution

Discussion

Recap

• Problem: Concept learning with few examples• Solution: Metaphors

Recap

• Problem: Concept learning with few examples• Solution: Metaphors

• Target Source

Recap

• Problem: Concept learning with few examples• Solution: Metaphors

• Target Source• Generic framework

Recap

• Problem: Concept learning with few examples• Solution: Metaphors

• Target Source• Generic framework• Wide range of relations

Recap

• Problem: Concept learning with few examples• Solution: Metaphors

• Target Source• Generic framework• Wide range of relations• Learn the difference

What if the concepts are not related?

What if the concepts are not related?

Metaphors are not a measure of relatedness

Metaphors are not a measure of relatedness

Metaphors explain how concepts are related

Vision

Explaining how concepts are related since 2012.M E T A P H O R S