Von Neumann to Bloom: Useful algorithmic techniques with vector spaces.

W P Cockshott

Topics

• Vector spaces and image representation    • Vector spaces and Bloom filtering• Vector spaces in economics• Metrics used when comparing image

patches• Vector spaces and Noethers principle --

what space should we use when modelling a system?

    Vector spaces and image representation

• use of basis spaces  – Cosine Transform, Wavelet Decomposition

• use of approximation vectors  – - Shannons coding theorem

• clustering, voronoi regions, analytic decomposition, hierarchical decomposition

Patches are vectors

A patch on an image can be unrolled to make a vector. Thus local areas of images can be treated as vectors

In this example we have an 8 x 6 patch which amounts to a 48 dimensional vector

use of basis spaces

• Systems of image representation such as DCT or wavelets work by changing the basis of the vector space to one that is more amenable to certain operations such as compression.

• The key to these transforms is the conceptof an orthogonal basis or coordinatesystem in which areas of a picture can berepresented.

Orthogonal basis

• An orthogonal pair of vectors is a pair of vectors at right angles.For example the vectors along the X and Y axes [1,0], [0,1] are at right angles and thusorthogonal. Similarly the 45degree vectors [1,1], [1,-1] are at right angles and thus orthogonal.

[1,0]

[0,1]

[1,0]

[0,1][1,1]

[1,-1]

Vector Lengths

v

v2=(vx)2+(vy)2

Orthonormal vectors

If in addition to being at right angles the vectors are of length 1, then they are orthonormal.

For example the pairs of vectors ([0,1],[1,0]) and

[1,0]

[0,1]

Are both orthonormal pairs

Inner product operations

• We can measure how far along an axis provided by one of the basis vectors a given v is by using the inner product operator.

• Let v be defined in the x,y basis and let p,q be and alternative orthonormal basis. The point v in terms of p,q is given by the vector [p.v,q.v]

x

yp

q

Extension to n dimensions

•

Transforming to and from a Basis

• Given v of dimension n and orthonormal basis vectors b1,b2,..,bn we can express v in this new basis as w=v [b1,b2,..,bn ] = [v.b1,v.b2,..,v.bn ] without loss of information

• Thus we can change the basis by multiplying by a matrix of the new basis vectors

• we can then reconstruct v from w as follows

This last stage involves forming v by a superposition of the basis vectors

Linear transform

These transforms are rotation operations in an appropriate vector space.

• effect of rotations by 45 on the unit vectors x = [1, 0], y = [0, 1]. The result is that

• x → [a, b] = [ 1/√2, 1/√2] and

• y → [−a, b] = [−1/√2, 1/√2].

Reasons for doing this

• We typically want to transform image vectors onto a different basis space in which the image energy is unevenly spread.

• If we take the basis formed by cosine waves of different frequencies for example, we find that the bases representing high frequencies tend to have little energy and can be discarded.

• This is the basis of the MPEG system used in digital TV

Metrics used when comparing image patches

• Suppose we have a binocular pair of cameras and we want to compute the disparity field.

• We can take a patch on the left image and search for a patch on the right image that looks most similar to it.

• The inner product operator turns out to be useful for detecting such similarity in appearance, provided that we do two things first.

Compensate for brightness and contrast

• Original images may differ in brightness and contrast

• First perform difference of Gaussian filter to remove local mean brightness

• Then take local patches and normalise them as vectors ( set |v|=1)

• Then compare patches using inner product

Inner product and unit hyper-sphere

• By normalising our vectors we cause them to lie on the unit hyper-sphere

• The inner product then computes the cosine of the angle between vectors.

• This turns out to be a good visual similarity metric.

Vector spaces and Bloom filtering – an IR application

• The original use of Bloom filtering in Content Addressable Filestore      

• Recast this into vector space representation      

• Compare with Dominic Widdows’ technique of random projections

CAFS was an intelligent disk controller from ICL

•It could do content access of the data on disk, and using a clever vector space technique invented by Bloom it could do relational join.

•It had hardware match units that could recognise patterns on a track.•It also had a number of bitmap rams for doing filtering.

CAFS BOARD3.5’’ disk

Performing relational join

Department Table

DepartmentID DepartmentName

31 Sales

33 Engineering

34 Clerical

35 Marketing

Employee Table

LastName DepartmentID

Rafferty 31

Jones 33

Steinberg 33

Robinson 34

Smith 34

Jasper 36

LastName Department ID Department Name

Smith 34 Clerical

Jones 33 Engineering

Robinson 34 Clerical

Steinberg 33 Engineering

Rafferty 31 Sales

Result table

Filter buffers

Suppose we had 8 filter buffers on board, each in the form of 64K bit ram chips.(1980s remember)

• Pass through the Department table and for each department no generate 8 hash address, one to each buffer chip.– Set the corresponding bit

• Pass through the Employee table, and for each dept no, generate 8 hash address, one to each buffer chip.– If all 8 bits are set then pull out the employee as a

joining record.

Analysis

• Suppose that we have 128 distinct field values occuring in the first relation.

• Then each RAM will have approx 128 bits set, and the probability of any one bit being set is 128/64k=1/512

• Probability of all 8 RAMS having a bit set is 1/4096

• Thus after two passes of the data we will have found the matching entries to very high probability

Relation to vector spaces

• Consider concatenating the 8 buffers, we get a boolean vector of length 512K call this m

• In this we have set 8 random bits for each field that we have hashed, call this k.

• Each of these these operations thus creates us an m element pseudo basis vector.

• Why pseudo-basis, because the vectors created by distinct keys are almost certainly orthogonal.

• After a pass through the first relation we have in our buffers a superposition of these basis vectors

Iverson’s Generalised Inner Product

• Iverson included four operators to modify the action of functions.

• An operator has one or two functions as its arguments and produces a modified function as a result; the modified function then acts on the argument or arguments to the right (and maybe also to the left) of the operator expression. The operators are – inner product (.), – outer product ( ∘ .),– reduction (/) and – scan (\). – For example the plus dot times inner product (written + .

×) is conventional matrix multiplication. • ( OR . AND) is then one possible Boolean inner

product, but if values are 0,1 this is equivalent to (+. ×)

Projection space

• In relational databases, the fields are typically strings, say up to 40 chars long.

• The cardinality of the set of all possible 40 character strings is vast.

• The Bloom filters project n keys from this huge set into the space spanned by pseudo basis vectors of length m each of which has k non-zero elements.

• Since these are almost orthogonal we can superpose them using Boolean 1 bit arithmetic and the inner product of this superposition using Iversons generalised inner product can then be used for set membership.

• Let b be our superposed filter buffer, and vk be the pseudo basis of key k, then using Iversons notation the approximate set membership test is b(×. ×)vk

• Algebraically this is just what Dominic Widdows described, earlier this month except that Bloom used Iverson’s generalised inner product and the values 0,1 rather than -1,0,1 as used by Widdows

Vector spaces in economics

• Work of von Neumann  - limits to growth      

• Work of Sraffa - modeling prices      

• Metrics to use when comparing theories of prices          – cosine, mean absolute deviation, correlation

Work of von Neumann 

• 1933 Johann von Neumann, Mathematische Grundlagen der Quantenmechanik

• 1932 First presentation by von Neumann of the lecture eventually published as 1945 J Neumann A Model of General Economic Equilibrium - Review of Economic Studies, – The latter work brings to bear many of the techniques

he develops in the former.

• 1945 "First Draft of a report to the EDVAC," lays the grounds for the current architecture of computers, makes large scale linear algebra calculations practical.

Starts field of matrix economics

Important results• System of equations to determine equilibrium

prices• Determination of profit rate• Determination of maximal growth rate• Turnpike theorem, composition of industrial

output required for maximal economic growth.

Later impact of matrix economics

• Theory of prices Pierro Sraffa and neo-Ricardian School– empirical tests of Sraffian price theories are one of my

interests

• Input output tables – Leontief• Linear Programming – Kantorovich, Koopmans and Danzig

– Theory of economic planning, this is a long lasting interest of mine

Leontief, Kantorovich, Koopmans get Nobel Prizes for this workIt was argued by Kaldor who knew von Neumann (and more

recently by Kurz and Salvadori) that the origins of von Neumanns model should be seen in the 18th-19th century classical economists (Quesnay,Ricardo, Marx) rather than the later ‘neo-classical’ school which is now dominant.

Explanation of von Neumann model

• I will present a simple von Neumann economic model as a Pascal program to give an idea of how it works.

This is a very simple economy, with only 3 products, but the principles remain the same however many products there are.

Technology Matrix A

• The matrix A encodes the technology of the economy

• element Ai,j of the matrix specifies how much of the jth commodity is required to produce a unit of the ith commodity , e.g.:

corn coal iron

corn 0.2 0.1 0.02

coal 0.0 0.2 0.1

iron 0.0 0.7 0.1

Labour and wage vectors

• We next introduce a labour vector L which specifies the labour needed to produce one unit of each output

• L= corn coal iron 0.2 0.1 0.02

To survive labour consumes a real wage vectorreal wage per unit of labour

corn coal iron

0.50000 0.20000 0.00000

Other variables used in the example model

Iterate to stability

We then iterate to a stable state to derive the prices, profit rate and wage. As is conventional in such models we express prices as relative prices, taking the relative price of the first commodity to be 1.

Set new prices

Total labour

y = output

fix real wage and converge on prices and profit rate

corn coal iron

n 8.00000 3.20000 0.90000

y 10.00000 7.00000 2.00000

x 2.00000 3.80000 1.10000

u 9.00000 6.60000 1.10000

s 1.00000 0.40000 0.90000

p 1.00000 0.54880 0.68783

r 0.13742

w 0.60835

What is important here

Technology and the real wage will, on the assumption of profit rate equalisation fix :

1. All prices

2. The rate of profit

Kurz says that some of this was developed a year or two earlier by a colleague of von N called Remak who called these prices superposition prices, but Remaks model was less general. Kurz says von N was replying to Remak.

Maximal growth path

• Von Neumann was concerned with the maximal rate of growth of the economy on the assumption that capitalists reinvested all their profits.

• If this was to occur the mix of surplus outputs produced had to match the inputs needed for steady growth the next year. We can derive this by adding an additional rule, which will cause the system to converge on a maximal growth path.

Max Growth path

y 10.57631 6.99289 1.16904

u 9.29777 6.14754 1.02772

s 1.27854 0.84535 0.14132

r 0.13751

w 0.61019

Note that for all industries the ratio of si/ui = r

U is now an eigenvector of the technology:labour matrix and (1+r) is the corresponding eigenvalue

Applications

Matrix economics of this sort is obviously very relevant to the theory of economic planning, and to understand the processes going on in a rapidly industrialising economy like modern China or Russia in the 1930s.

In China today, national output is hugely directed towards the production of capital goods ( around 50% of national output) which is necessary on a maximal growth path.

   Meta-theory: Vector spaces and Noether’s principle

Put very loosely, Emmy Noether’s principle says that in a physical system, a conserved quantity at one level of abstraction corresponds to a symmetry property of the system at another level of abstraction.– Translational symmetry implies conservation of

momentum– Temporal symmetry implies conservation of

energy

Example

• Top shows points in phase space traversed by a projectile thrown upward in a gravitational field

• Bottom, points in the space of (altitude, velocity squared) traversed by the particle

• Bottom diagram shows conservation of energy

Find the symmetry

• If we transform to yet another space we find the path is a circle.

• Here we can see the symmetry associated with the conservation of energy.

• The new representation means that we can treat the path as the result of rotational symmetry in a vector space.

v

Another example is in quantum mechanics.

Why do we use amplitudes whose square gives us probabilities?

• Because this vector space, which allows the application of unitary rotation operators, projects onto the space of probabilities which are a scalar conserved property.

• This is the same as the change of representation we had to illustrate conservation of energy.

Conclusion

• Look for a change of representation that enables you to detect symmetry in your system.

• Vector space representations allow us to model rotational and translational symmetries.

• The best representation may not be the obvious representation at first sight.