Upload
ami-wilkerson
View
212
Download
0
Embed Size (px)
DESCRIPTION
LibreOffice Productivity Suite 3 My Education 2002–2005: undergraduate studies Politecnico di Milano, Italy B.A. in Computer Science 2005–2010: graduate studies M.S. in Electrical and Computer Engineering M.S. in Statistics Ph.D. in Electrical and Computer Engineering Thesis Title: “Cognition and Cooperation in Wireless networks: An information theoretic perspective” Adviser: Daniela Tuninetti Co-Adviser: Natasha Devroye
Citation preview
Information Theory
CH00-1
The instructor
• I’ve been in Taiwan for (only) two semesters• This is the third course that I teach at NCTU• I do research mostly in information theory, but I
also explore other topics
LibreOffice Productivity Suite
3
My Education
2002–2005: undergraduate studiesPolitecnico di Milano, ItalyB.A. in Computer Science
2005–2010: graduate studiesM.S. in Electrical and Computer EngineeringM.S. in StatisticsPh.D. in Electrical and Computer Engineering
Thesis Title:“Cognition and Cooperation in Wireless networks:An information theoretic perspective”
Adviser:Daniela Tuninetti
Co-Adviser:Natasha Devroye
4
2011-2012: postdoctoral student Technical University of Munich (TUM)
2012-2013: postdoctoral student Stanford University
Adviser:Gerhard Kramer
Adviser:Andrea Goldsmith
My Education
In a mapBorn Bachelor
PhD
Post Doc #1Post Doc #2 Assistant
Professor
The course is divided in two parts:
SOURCE CODING Given a source of information, how can we remove the
intrinsic redundancies, so as to obtain the most compact representation
CHANNEL CODING How do we introduce structured redundancy so as to
protect a sequence from random errors?
An overview
When Thursday13:20-16:20
Where room B07
Engineering Building Office Hours
Thursday 4pm-5pm (after class) Or email me at
Plus I’m always in the lab, always
The Course
Grading20 % Final Exam20 % Midterm Exam20 % Homework20% Quiz 10% Presentations 10 % Class participation
C0-8The Course
Final Exam Midterm Exam
Both exams will be take-home exams
The exams will be distributed on Monday morning and will be due on Friday night
The exams will be personalized up to a certain extend to avoid collaboration
The exams will contain a numerical session
CH0-9The Course
Final Exam Midterm Exam
After the exam is submitted, it will be distributed to one of your collogues for comments
I will grade both the midterm and the grades
This should help you understand how hard it is to get into someone`s mind
The Course
Homework
Homework will be assigned every week on Friday and will be due the following Friday
Homework will be posted on the course webpagehttp://stefano.cm.nctu.edu.tw/it.html
Homework will be submitted electronically at [email protected] can either scan your homework or type it
Exams will be based on the homework exercises
The Course
Quiz
The last 20mins we will have a quiz The quiz will cover the topics of the previous
lectures The quiz should help you memorize some basic
formulas. This will help in the long term (hopefully!)
The Course
Presentations
Homework will be assigned every week on Friday and will be due the following Friday
Homework will be posted on the course webpagehttp://stefano.cm.nctu.edu.tw/it.html
Homework will be submitted electronically at [email protected] can either scan your homework or type it
Exams will be based on the homework exercises
The Course
Class participation Class participation is very, very important
in the grading Let’s get over the good reasons why you
should be asking questions
CH00
The Course
1. You understand everything so completely that questions aren't necessary. In this case you should not be taking the class
2. You are so utterly lost that you don't even know where to start asking questions. In this case you should not be taking the class
3. You don't want to embarrass yourself or keep the class back with your questions. In this case you are missing an opportunity to
learn for you and your classmates
The Course
I will understand a lot better how her presentations are received. I can then adjust her lectures to the particular audience.
The questioner may receive elucidation of a particular point and benefit from the rest of this particular lesson.
The question might be of benefit to a number of other students who had not thought of it.
I might learn something and look at things in a different light
The teacher may have made a mistake that he can now recognize and recover from.
The Course
I don’t wish to enforce participation with grades or forcing people to ask questions
I’d rather you remember this
You should take control over your learning. You have the power to obtain the information you need to be successful.
…and it will make your leaning more fun!
The Course
How about breaks, professor? The class is very long
This means that you have to digest a lot of material in a single lecture
I propose 2 breaks of 15 mins each so that you can ask questions and refresh your thoughts
I tend to be absorbed, so feel free to remind me
This is the notes that I will follow
The Book
Title Information TheoryLecture notes
AuthorsStefan M. Moser
Online http://moser-isi.ethz.ch/scripts.html
Other books are:
A First Course in Information Theory Raymond W. Yueng, Plenum PubCorp, 2002.
Elements of Information Theory, Thomas M. Cover and Joy A. Thomas,John Wiley and Sons, 1991.
Principles and Practices of Information Theory, Richard E. Blahut,Addison Wesley, 1988.
The Book
How many of you are familiar with Matlab?
Matlab
Each week one exercise will be a numerical exercise to be performed in Matlab Matlab is the industry standard for numerical
simultaions. The earlier you learn to master it, the better
I don’t think we need any particular introduction
You will be given a function template which you have to complete
Matlab
An important note: Whenever you write the code, make sure that it
can be run using the Matlab distribution made available by NCTU
The version in the school computers does not contain any package (communication package, fitting packages and so on) If I can’t run the code, the grade is zero
Matlab
I think that numerical simulations are a key way to test ideas
I will try to have use this approach as often as possible So let’s start with a warm up exercise
Matlab
ASCII table
Morse Code
Do the following exercise Think of the following problem
Given a text of a 5000/10.000 words, develop a Morse encoder then answer the following questions
How many bits are required to save the text in ASCII code and how many bits are needed to save the text in Morse code?
Call the above ratio the compression ration. Take a text in German, Italian and Spanish: how is the compression rate different in different languages?
Plot the compression ration for the different texts
Do the following exercise Now the fun part
Next assume that the bits in the Morse coded message are flipped from zero to one with a certain probability p
Using the above decoder, calculate the number of error in the reconstructed text for flipping probability in the interval [0,1/4]
Repeat the above calculation for the case in which the message is coded using ASCII code
Plot all the error probabilities
Another Exercise
Information Theory is similar to other branches of science of engineering in the following peculiarity: They were bootstrapped by the efforts of a single
individual For this reason you should read the first paper in
information theory
Claude ShannonThe mathematical theory of communication
Available online here: http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf
Another ExerciseClaude ShannonThe mathematical theory of communication
Read the paper and answer the following questions:
What kind of problems are defined as communication problem?
What kind of engineering applications did Shannon have in mind?
What kind of mathematics did he developed to study the problem at hand?