CSC 350 - Dr. Gary Locklair Exam #4 … CSC 350 - Dr. Gary Locklair update date on slides 5, 6, 7

CSC 350 - Dr. Gary Locklair

Exam #4 …


III. Information Theory

Computers are informational tools

Information (“I”) comes from Intelligence


A. Shannon (Info) Theory

Claude Shannon (1916-2001) American computer scientist of the 20th century


Two Statements:

“It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …”

“Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape ofDr. Ferry and assumed his role as president.”


Two Statements:One is dull:

“It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …”

The other bizarre:

“Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and

assumed his role as president.”


Two Statements:One doesn’t convey any (Shannon) information“It is Thursday, December 8, 2005, 1:15pm, S118B, CUW …”

Why? Because its probability is high - it is indeed Thursday, etc


Two Statements:The other conveys lots of (Shannon) information:

“Plasma beings from the planet Threa have infiltrated CUW. One of them has taken the shape of Dr. Ferry and assumed his role as

president.”

Why? Because its probability is low.



deals only with syntax; it does not deal with semantics.



As one example, the truth or falsity of the statement isn’t considered. For the moment, we won’t care if Plasma beings have really taken over Dr. Ferry or not! :-)


Dr. A. E. Wilder-Smith called Information the “surprise effect”


Shannon (Info) Theory attempts to:

1. Quantify, or measure, I

2. Set theoretical limits (define what’s possible) for conveying (transmitting) I


Shannon was concerned with transmitting I over phone lines.

He wanted to reliably convey I from source to destination.

In other words, we don’t want the I corrupted during transmission.


What would corrupt the I?

[student responses here :-]

How might you deal with a

noisy channel?

[student responses here :-]

Notice that all require more effort


Shannon asked

1. Is it possible to detect and correct a

corrupted message (within what limits)?

2. How can a garbled message be recovered?


Deal with noise by

1. Recognize the problem

2. Compensate for it to begin with


Situation

I must be conveyed from A B over a noisy channel

1. Reliability – message at B should be identical to message at A


Situation

I must be conveyed from A B over a noisy channel

2. Maximize Rate – effective transmission time

{These are the real world tradeoffs}


We don’t want to give up channel capacity …eg, we don’t want to repeat message 5 times to ensure it arrives since there is no new I conveyed during times 2-5


Shannon (Info) Theory shows that we don’t have to give up rate to gain reliability however, there is a price: delay – time to recover or decode the message increases due to (perhaps) longer messages


Actually two subjects

Info Theory – what is possible

Coding Theory – how to do it




Theory (fun!)

Mutual Information – I provided about event X by occurrence of event Y

I(X;Y) = LOG P(X|Y) / P(X)


Theory (fun!)

Self Info – I(X) = LOG 1 / P(X)

Entropy – “average” I = H(X) = P(X) * LOG 1 / P(X)


remember

1. don’t consider if message is T or F (yet)

2. Shannon (Info) Theory depends upon

probability of the message


Example

Self Info – I(X) = LOG 1 / P(X)


If message is certain, Info should be …? I(X) = LOG 1 / P(X) …

P(X) = 1 (100%)

we usually use LOG2 and unit of I is the bit

LOG2 1 = 0 … makes sense, there is no “info” in a certain message


Practical Example

Using Shannon (Info) Theory along with Coding Theory to see how to efficiently (and reliably) transmit I …


What if we know some messages are more likely than others?

Ex: a weather forecaster with 4 possible forecasts:


Ex: a weather forecaster with 4 possible forecasts:

Cloudy = 50% - ½

Sunny = 25% - ¼

Rainy = 12.5% - 1/8

Tornado = 12.5% - 1/8


Normally, for 4 different messages we’d need at least … how many bits?

2 bits (encode 4 possibilities as 00, 01, 10, 11)


Shannon (Info) Theory says “average info” (entropy) of this “alphabet” (set of messages) is: P(X) * LOG 1 / P(X)

sum up (probability * self I)


(½ * LOG2 1/(½)) +

(¼ * LOG2 1/(¼)) +

(1/8 * LOG2 1/(1/8)) +

(1/8 * LOG2 1/(1/8)) =

1/2 + 2/4+ 3/8 + 3/8 = 1¾


What? How can we transmit in less than 2 bits?Shannon says only 1¾ bits!On average, one message is much more likely, therefore encode it in a shorter bit string than the others.


Huffman encoding – variable length codes

Coding Theory (prefix code)

0 = cloudy10 = sunny110 = rainy111 = tornado


Zeb/Zeke Joke

Zeb and Zeke were sadly returning from an expensive fishing trip which only produced one fish.


Zeb/Zeke Joke

“The way I figure it,” said Zeke, “that lousy fish cost us $400!”

“Wow,” replied Zeb, “it’s a good thing we didn’t catch more!”


B. Information Theory

Gary Locklair - American computer scientist of the 21st century Who?

Information only comes from Intelligence


Dr A. E. Wilder-Smith

British scientist of the 20th century

Life is matter + teleonomy (Information Content)

Ultimate source of teleonomy is an omnipotent God


Dr. Werner Gitt

German computer scientist of the 21st century

“Laws of Information”

Information consists of syntax and semantics


Dr. Werner Gitt

“According to Shannon’s Theory any random sequence of symbols is regarded as information, without regard to its origin, nor whether it is meaningful or not.”


Gitt’s Levels of Information1 - Statistics (transmitted signal)

2 - Syntax (coding method)

3 - Semantics (meaning)

4 - Pragmatics (action)

5 - Apobetics (purpose)


Information Theory is more than just statistics (Shannon)There must be an associated meaning for information to be present

Example: computer program …


In Shannon (Info) Theory the truth or falsity of the statement isn’t considered!Although the “Plasma being” message has high Shannon (Info) Content

It’s a bloomin’ lie! :-)


In Shannon (Info) Theory the truth or falsity of the statement isn’t considered!Although the “Today is …” message has low Shannon (Info) Content It’s the truth! :-)


Information Theory impliesa sender and a receiver

Sender: I have a purpose in mind, that will require some action, so I will communicate my idea using a particular code and then transmit it.


Information Theory impliesa sender and a receiver

Receiver: I received a signal, now I must decode it. I can now understand the idea and implement some action to achieve the desired result.


This is Information

“For God so loved the world that he gave his one and only Son, that whoever believes in him shall not perish but have eternal life.”


From Computer Scientist Don Knuth

Documents

CSC 350 - Dr. Gary Locklair Exam #4 … CSC 350 - Dr. Gary Locklair update date on slides 5, 6, 7