I. Overview of some basic image processing Here is a quick story about an image on the internet to set the stage for our discussion. A galaxy sits out on edge of our galaxy. It emits a wide spectrum of electromagnetic waves. A satellite is designed and sent into space to image the wide range of electromagnetic waves emitted from things like this galaxy. On board it has different equipment including a camera that is capable of capturing visible light. To simplify things let’s pretend this lens is a lot like modern digital camera lenses and consists of a matrix of photocells that are sensitive to red, green, and blue light. These build an array of data based on the intensity of a particular band of light received. After capturing an image our satellite encodes the data and transmits it using radio frequencies. These are received on earth by satellite receivers, transmitted to computers, decoded, and analyzed by scientists. These scientists are pretty clever and create a color image by combing the data received; they decide to share it as .jpg on the Internet. In this way the visible information emitted from the galaxy is turned into electronic information that is then encoded and retransmitted as yet another type of electromagnetic information, received, decoded and retranslated yet again into different electronic signals and recombined only to be sent again through a series of routers and servers to re-emerge through your monitor as visible light. Let’s take a minute to look at just what a .jpg image is in terms of linear algebra. You can think of any image as an nXm matrix where each entry in matrix consisting of pixel values. In an 8-bit .jpg these values range for 0-255. A typical color .jpg consists of 3 such matrices, each one describing a discreet color space. When these spaces combine they form what we typically see as a color image in the visible spectrum of light. Each pixel can in turn be thought of as a “binary” vector in the form [b1 b2 b3 b4 b5 b6 b7 b8] where each bit is a member of the finite vector field F2 |{0,1}. Basically that means they can only be 1’s or 0’s and have special “mod 2” addition rules. They can also be easily turned back into their integers. It’s to the delivery of this single pixel vector our discussion now turns.

II. Overview Of Error Correction One of the many of technical challenges involved with our little satellite taking its picture that will eventually end up on Wikipedia is getting that information, and our little pixel, all the way back to earth in one piece. Space is pretty big and there are lots of things that could mess that signal up. You add in the probability of the on board electronic hardware introducing errors, along with the whole information chain once it gets TO earth and its pretty amazing we can send any digitized information anywhere. To top off our challenge, our satellite only has so much energy with which to communicate, so each and every image, and each and every pixel should be accounted for! So how do we make sure every bit transmitted gets received with perfect parity? Sending duplicate copies would work, but it means that each transmission wastes energy because some amount of your signal is going to be received just fine. Imagine having power for 100 images, but duplicating the data 4 times means you can only send 25 images worth of information. We could just wait to get a transmission, check it out, if its messed up ask the satellite to resend it. Again that uses more energy, and there isn’t any guarantee you would get any better messages the second, third, fourth, nth time you asked. This could cause us to use up all the energy for just a single image. There has to be a better way that we can have our satellite send us data... There is! And the foundation for the techniques employed is called Code Theory. It heavily leverages linear algebra, and provides methods by which data is encoded in such a way that even if it gets messed up through transmission or reception it can be “fixed up” once someone gets their hands on it. This means that every message our little satellite sends is consuming less power than if we relied on redundancy or resending. Along with its long storied history, it also has the added benefit of allowing us to automatically correct the errors that get introduced.

III. History Of Error Correcting Codes And Introduction To Code Theory Error correction codes, and code theory in general, has its start with our need to send information through channels that mangle it to machines that can’t reason between messed up data and good data. The individual attributed to the start of modern error correction is Richard Hamming. A mathematician who among other things, was responsible for the machine level calculations to determine if the atomic bomb would ignite the atmosphere. During this early time of computing he had a lot of automated tests to run over weekends. If everything went well he would have a pile of answers. If everything didn’t go so well he would have a pile of garbage. After multiple weekends of things not going so well he began thinking about ways the machine itself could just fix the errors as they occurred saving everyone, including himself, a lot of time and headache. The basic premise of code theory is as follows: A message is encoded by a generator matrix. Once encoded, the message becomes a code-word and consists of two parts. One is the message bits, and the other is a set of parity bits. Once received, the code word is ran through a parity-check matrix that produces a result called a syndrome. If the syndrome is the zero vector then the message was received without errors and the original message can be processed. If the syndrome is not the zero vector, it will be a vector corresponding to the location of the error introduced through the communication channel, and thus automatically fixable. This is because parity bits are constructed using the message bits themselves, thus becoming linearly dependent on them. Each message bit is tied to some unique combination of parity bits. This means that if a message bit is changed, you can simply look to see if the parity bits still make sense, and if they don’t you know where the error is coming from. Because all you work with is 1’s and 0’s you can then fix it. For a traditional “Hamming Code” The amount of errors your can detect and correct is based on the relationship between how many data bits you have, and how many parity bits you use. A message contains K bits of data, and N total bits. The number of parity bits needed is N-K, and is referred to as D. D is also the “distance” or the maximum number of ways in which each message can differ from one another. It is also written as [n,k,d] code. N/K produces a ratio called the rate that informs you as to how effective an [n,k,d] code is versus redundantly sending a message , so you know much power per bit you are saving.

IV. Mechanical Explanation Of Error Correcting Codes Below is the mechanical “bit by bit” process of encoding and decoding a signal using an error correction code. This example uses the standard set up of a [7,4,3] hamming code with an extra 8th bit. Start with an empty array of 8 blank bits: {[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]} Assign the 8 Bits to be either Parity bits or Message bits. The actual parity bits chosen are based on a pattern. Bit 1 checks 1 bit skips one bit, Bit 2 checks 2 bits skips 2 bits, Bit 3 having been checked is out of the rotation. Bit 4 checks 4 bits skips 4 bits. Bits 5, 6, and 7 have already been checked by other bits so they become our message bits with the eighth bit reserved for later. Our new, empty, bit array looks like this: {[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]} Now assign a binary message to be sent: {[ ] [ ] [1] [ ] [1] [0] [1] [ ]} Now generate the parity bits using the message bits. Bit 1 add (mod 2) bits 3, 5, and 7. 1+1+1 = 1. {[1] [ ] [1] [ ] [1] [0] [1] [ ]} Bit 2 add (mod 2) bits 3, 6, and 7. 1+0+1 = 0. {[1] [0] [1] [ ] [1] [0] [1] [ ]} Bit 4 add (mod 2) bits 5, 6, and 7. 1+0+1 = 0. {[1] [0] [1] [0] [1] [0] [1] [ ]} Bit 8 add (mod 2) the first seven bits. 1+0+1+0+1+0+1 = 0. {[1] [0] [1] [0] [1] [0] [1] [0]} So our final coded message is: {[1] [0] [1] [0] [1] [0] [1] [0]} Say we send this message but it gets received as: {[1] [0] [1] [0] [1] [1] [1] [0]} How do we detect the error? We just recalculate all the parity bits to see which ones don’t match the received message. In this case Bit 1 is still 1, Bit 2 is 1, Bit 4 is 1. Comparing to our received message we have a discrepancy with bits 2 and 4. To find the message bit with the error just add the parity bit locations with discrepancies together 2+4 = 6. Knowing this we can simply switch bit 6 from 1 to 0 and have our originally intended message!

V. Linear Algebra and Error Correction Codes Having manually seen how to encode and decode a message we can use linear algebra to give us a more general framework to understand them at a deeper mathematical level. Let’s look at the encoder, or Generator, matrix: You start with a message vector m, and a generator matrix G, where G has dimensions N (number of message bits) by K (number of total bits). When you send m through G via matrix multiplication you get c, which is a “code word”, we can write this as mG=c. C is a linear combination of m and G where G forms the basis of for all the code words in N.

In notation this looks like this:

m = [ ]

G = [ ]

The first part of G is the identity matrix for n terms, and the second part is the inverse of a part of the parity matrix. In a hamming code the non-identity part consists of what parity bits are checking what message bits.

For our [7,4,3] hamming code it G would look like this:

When you multiply mG you get the linear combination c = [m1, m2, m3, m4, (m1+m2+m4), (m1+m3+m4), (m2+m3+m4)]. These columns correspond to bits {3, 5, 6, 7, 1, 2, 4}

Now for the decoder, Parity Check, matrix: The key thing here is that G creates all the codes that become the null-space of H. This means that Hc, if c is a valid code word, will equal the zero vector. If it doesn’t, it will be a column in H corresponding to the bit with the error in it. It’s the linear dependence that exists between parity and message bits that allows this to happen. You could geometrically think of all code words being perpendicular to the vectors in H. If a code-word has an error then part of it is “tilted” into H.

In notation it looks like: c = kX1 column vector. [m1, m2, m3, m4, (m2+m3+m4), (m1+m3+m4), (m1+m2+m4)]

H = [ ]

For our [7,4,3] hamming code it H would look like this: For a hamming code, H turns out to be the bit numbers in binary, then column shifted so the identity matrix is in back. The columns are ordered by bits {6,5,3,7,4,2,1}

1 0 0 0 1 1 0

0 1 0 0 1 0 1

0 0 1 0 0 1 1

0 0 0 1 1 1 1

1 1 0 1 1 0 0

1 0 1 1 0 1 0

0 1 1 1 0 0 1

V. Vector Spaces and Sets Let’s take a look at what is going on with the different spaces to get a better feel for what is happening. K is the finite vector space consisting of all code-words {c1, c2, c3 …cn} generated by the basis G, and the coefficients m. N is the finite vector space of all N combinations possible of {0,1}. There are 2^k code-words, and 2^n possible ways to receive a message.

How the generator and parity matrix are related is what allows error detection and correction to take place. Any valid codeword c is in the rowspace of G, and the rowspace of G is the nullspace of H. The inverse is also true, the rowspace of H is the nullspace of G. This means the code-words are in the nullspace of H. This means that to construct H you just need to build a system of equations such that Hc=0.

You can also look at the relationship between what sets of parity bit are dependent on what message bits. These relationships form smaller sets of vectors within the code word vector itself and it’s this interdependence that leads to the interaction between the generator and parity matrices.

K

K

N

H 0

G

VI. Pseudo C++ Vs. Pseudo Matlab Encoding Comparison The main thing to note here is the utility of the hamming matrix . It actually has all of the functionality of the C++ code IMPLICITLY integrated into its very structure. It’s a good lesson that clever use of matrices can simplify programming.

C++ This pseudo C++ implementation of a generator matrix uses the bit shift operator to manually add a bit, then work through each bit, one at a time using the parity bit being calculated to control the number of bits shifted.

EncodeImage (inputData, parityBit) {resultBit = dataLength = inputData.length

while (0 < dataLength) {result &= inputData & 1inputData = inputData >> parityBit;

dataLength -= parityBit} return result

}

EncodeEveryThing (inputData) {outputData = 0

# Encode all our parity bits

(parityBit in parityBits)outputData |= EncodeImage(inputData, parityBit)

# Shove in our data

(dataBit in dataBits) {dataBitShift = imgageDataoutputData |= (inputData & dataBit) << dataBitShift}

return outputData

}

Matlab The pseudo matlab code is much more compact. It reads in the file, saves it as an array then loops through the array elements performing the matrix operations for each pixel, saving them to a new image matrix. function [ imageCoded ] = ImageEncode( filename )

img = filename

a = imgread(‘img’); G=ham[7,4]; dataLengthR = size(a,:);

dataLengthC = size(:,a);

i = 1;

j = 1;

while (0 < dataLengthR){

c=dec2bin(a(i,j));

bBin(i,j)=c*G;

bDec=dec2bin(bBin(i,j);

imageCoded(i,j)=bDec;

i = (i+1);

dataLengthR = (dataLengthR + 1);}end

Work Sheet 1) Given the code [7,4,3]. a) How many message bits are there? b) What is the rate? c) What is distance? d) What is total size code generated? e) Is this code more efficient than a [6,3,3] code? f) How many potential messages are possible?

2) Assume you receive this bit array [10101110] encoded with a [7,4,3] generator matrix. a) Is there an error? b) What bit has the error? 3) You have two words [001] and [101]? a) What is distance between them? b) What is the maximum distance for ANY two words in this code? c) What is total number of words in this code? 4) Given the parity matrix H as Hc= [1 0 1] a) What column of H does the syndrome correspond to? b) What row of c is the error in? 5) In the manual example above, what is the function of the eighth parity bit?

1 1 0 1 1 0 0

1 0 1 1 0 1 0

0 1 1 1 0 0 1

SourcesError–correcting codes with linear algebra http://www.math.union.edu/~jaureguj/ECC.pdf Hamming, "Error-Correcting Codes" (April 21, 1995) http://www.youtube.com/watch?v=BZh07Ew32UA Encoding and Decoding with the Hamming Code http://www.uwyo.edu/moorhouse/courses/4600/encoding.pdf The Hamming Code in Matrix Form http://www.ece.unb.ca/tervo/ee4253/hamming2.shtml Error Correction and the Hamming Code http://www.ece.unb.ca/tervo/ee4253/hamming.shtml TLT-5400/5406 DIGITAL TRANSMISSION http://www.cs.tut.fi/kurssit/TLT-5400/matlab_tehtavat/TLT5400_M5.pdf Construction of Hamming codes using Matrix http://www.gaussianwaves.com/2008/05/construction-of-hamming-codes-using-matrix/ Lecture - 15 Error Detection and Correction http://www.youtube.com/watch?v=aNqiTCZ-nko Parity Check http://en.wikipedia.org/wiki/Parity_check Linear Block Codes http://en.wikipedia.org/wiki/Linear_block_codes Generator Matrix http://en.wikipedia.org/wiki/Generator_matrix Parity Check Matrix http://en.wikipedia.org/wiki/Parity_check_matrix Hamming Code http://en.wikipedia.org/wiki/Hamming_code Systemic Code http://en.wikipedia.org/wiki/Systematic_code Forward Error Correction http://en.wikipedia.org/wiki/Forward_error_correction How To Count In Binary http://www.wikihow.com/Count-in-Binary