Hyperdimensional Cognitive Computing: A New Approach to ......Hyperdimensional Cognitive Computing: A New Approach to Some Very Old Problems Simon D. Levy Washington & Lee University

Hyperdimensional Cognitive Computing: A New Approach to

Some Very Old Problems

Simon D. LevyWashington & Lee University

Lexington, Virginia, USA

Tuesday, April 14, 2009

Inspiration(s)

[I]t turns out that we don’t think the way we think we think! ... The scientific evidence coming in all around us is clear: Symbolic conscious reasoning, which is extracted through protocol analysis from serial verbal introspection, is a myth.

− J. Pollack (2005)

[W]hat kinds of things suggested by the architecture of the brain, if we modeled them mathematically, could give some properties that we associate with mind?

− P. Kanerva (to appear)


What is Mind?


What is Mind?




The Need for New Representational Principles• Ecological affordances (Gibson

1979); exploiting the environment (Clark 1998)

• Distributed/Connectionist Representations (PDP 1986)

• Holographic Representations (Gabor 1971; Plate 2003)

• Fractals / Attractors / Dynamical Systems (Tabor 2000; Levy & Pollack 2001)


The Need for New Representational Principles• Ecological affordances (Gibson

1979); exploiting the environment (Clark 1998)

• Distributed/Connectionist Representations (PDP 1986)

• Holographic Representations (Gabor 1971; Plate 2003)

• Fractals / Attractors / Dynamical Systems (Tabor 2000; Levy & Pollack 2001)


Pitfalls to Avoid

1. The “Short Circuit” (Localist Connectionist) Approachi) Traditional models of phenomenon X (language) use entities A,

B, C, ... (Noun Phrase, Phoneme, ...)

ii) We wish to model X in a more biologically realistic way.

iii) Therefore our model of X will have a neuron (pool) for A, one for B, one for C, etc.


a.k.a. The Reese’s Peanut Butter Cup Model


E.g. Neural Blackboard Model (van der Velde & de Kamps 2006)


Benefits of Localism (Page 2000)

• Transparent (one node, one concept)

• Supports lateral inhibition / winner-takes all


Lateral Inhibition (WTA)

A B C

L1

L2


Problems with Localism

• Philosophical problem: “fresh coat of paint on old rotting theories” (MacLennan 1991): what new insights does “neuro-X” provide?

• Engineering problem: need to recruit new hardware for each new concept/combination leads to combinatorial explosion (Stewart & Eliasmith 2008)


The Appeal of Distributed

Representations(Rumelhart &

McClelland 1986)


WALK

WALKED


ROAR

ROARED


SPEAK

SPOKE


GO

WENT


ignores(mary, john)

Mary won’t give John the time of day.


Challenges (Jackendoff 2002)


I. The Binding Problem

+

? ? ? ?Tuesday, April 14, 2009

II. The Problem of Two

+

? ? ?Tuesday, April 14, 2009

III. The Problem of Variables

ignores(X, Y)

X won’t give Y the time of day.


IV. Binding in Working Memory vs. Long-Term Memory

ignores(X, Y) X won’t give Y the time of day.

Long-term memory (idiosyncratic):

transitive-action(X, Y, Z) X Verb Y Z

Working memory (“rule-based”):

Mary gave John an apple.


Vector Symbolic Architectures

(Plate 1991; Kanerva 1994; Gayler 1998)


Tensor Product Binding(Smolensky 1990)


Binding


Bundling

+ =


Unbinding (query)


Lossy


Lossy


Cleanup

Hebbian / Hopfield /

Attractor Net


Reduction(Holographic; Plate 2003)


Reduction(Binary;

Kanerva 1994,Gayler 1998)


Composition / Recursion


Variables

X

john


Scaling Up

• With many (> 10K) dimensions, get

• Astronomically large # of mutually orthogonal vectors (symbols)

• Surprising robustness to noise


Pitfalls to Avoid

2. The Homunculus problem, a.k.a. Ghost in the Machine(Ryle 1949)

In cognitive modeling, the homunculus is the researcher: supervises learning, hand-builds representations, etc.


Banishing the Homunculus


Step I: Automatic Variable Substitution

• If A is a vector over {+1,-1}, then A*A = vector of 1’s (multiplicative identity)

• Supports substitution of anything for anything: everything (names, individuals, structures, propositions) can be a variable!


“What is the Dollar of Mexico?” (Kanerva, to appear)

• Let X = <country>, Y = <currency>, A = <USA>, B = <Mexico>

• Then A = X*U + Y*D, B = X*M + Y*PD*A*B =

D*(X*U + Y*D) * (X*M + Y*P) =

(D*X*U + D*Y*D) * (X*M + Y*P) =

(D*X*U + Y) * (X*M + Y*P) =

D*X*U*X*M + D*X*U*Y*P + Y*X*M + Y*Y*P =

P + noise


Learning Grammatical Constructions from a Single Example (Levy, to appear)

• Given

• Meaning: KISS(MARY, JOHN)

• Form: Mary kissed John

• Lexicon: KISS/kiss, MARY/Mary, ...

• What is the form for HIT(BILL, FRED) ?


Learning Grammatical Constructions from a Single Example (Levy, to appear)

(ACTION*KISS + AGENT*MARY + PATIENT*JOHN) *

(P1*Mary + P2*kissed + P3*John) *

(KISS*kissed + MAY*Mary + JOHN*John + BILL*Bill + FRED*Fred + HIT*hit) *

(ACTION*HIT + AGENT*BILL + PATIENT*FRED) =

....

= (P1*Bill + P2*hit + P3*Fred) + noise


Step II: Distributed “Lateral Inhibition”

• Analogical mapping as holistic graph isomorphsm (Gayler & Levy, in progress)

cf. Pelillo (1999)

A

B

C D

P

Q

R S


A

B

C D

P

Q

R S

Possibilities x: A*P + A*Q + A*R + A*S + ... + D*S

Evidence w: A*B*P*Q + A*B*P*R +...+ B*C*Q*R + .. + C*D*R*S

X*W = A*Q + B*R + ... + A*P + ... + D*S


∧

c∧

c

w * cleanup /∑

xt

xt+1πt




Future Work: Automatic Decomposition

MSC (Arathorn 2002)Tuesday, April 14, 2009

MSC (Arathorn 2002)Tuesday, April 14, 2009

Documents

Hyperdimensional Cognitive Computing: A New Approach to ......Hyperdimensional Cognitive Computing: A New Approach to Some Very Old Problems Simon D. Levy Washington & Lee University