Xin Li LDCSEE WVU Spring 2009 2015-6-3 From Gene to Meme Example of Memes: -Cultural ideas, symbols -Religions -Languages -Mathematics (e.g., Gaussian

Xin Li LDCSEE WVU Spring 2009Xin Li LDCSEE WVU Spring 200923/4/1823/4/18

From Gene to MemeFrom Gene to Meme

Example of Memes:

-Cultural ideas, symbols-Religions-Languages-Mathematics (e.g.,Gaussian distribution)-Scientific understanding (e.g.,Quantum mechanics)-Technological achievements (e.g.,Atomic force microscopy)-Engineering designs (e.g., ipod,Wii, iphone)


An Objective Measurement of An Objective Measurement of Meme’s FitnessMeme’s Fitness

Industry Academia


What is in Common?

Steve Jobs Herbert Simon


Xin Li LDCSEE WVU Spring 2009Xin Li LDCSEE WVU Spring 2009

Theme of This Talk“Life is about connecting dots”– in “Staying Hungry, Staying Foolish” Stanford

Commencement Address by Steve Jobs in 2005

Scientific research is also about connecting dots– Search is an important component part of

scientific research (where are the dots? how are they related?)

– Re-search often reveals hidden relationship among isolated dots that is not known before


Image processing

Statisticalphysics

Communication

Networking

CognitiveScience

geometry

PRAM/Microarray

STM/AFM

MRI/PET

Control

analysis

Chemicaloscillation

algebra

statistics

Image Processing as the ShowcaseScience Technology

Engineering

Mathematics


Image Processing: at the Intersection Image Processing: at the Intersection of Science, Technology, Engineering of Science, Technology, Engineering

and Mathematics (STEM)and Mathematics (STEM)

+

our starting point


So how should you choose your So how should you choose your technical field?technical field?

Outside environment plays some roleOutside environment plays some role– Emerging areas tend to attract more resources Emerging areas tend to attract more resources

than traditional fieldsthan traditional fields– Every dept. has its focused areasEvery dept. has its focused areas

Learn yourself betterLearn yourself better– Good at theory or experiment/application?Good at theory or experiment/application?– Good at algebraic or geometric thinking?Good at algebraic or geometric thinking?– Good at depth-first or width-first reasoning?Good at depth-first or width-first reasoning?

Find a good matchFind a good match23/4/1823/4/18


Taking Myself as an Example

Entered Princeton ISS group in 1996 Very little research experience in my undergraduate study (BS thesis is on speech coding)Information theory or signal processing?– Princeton EE is really strong in theory (not to mention

Math and Physics)– Majority of ISS students will take the theory path even

by doing TAs due to limited research funding in the area of information theory

– Graduate students in Princeton EE from India are also really good at theory


Information Theory vs. Image Processing

IEEE TIP– H-index=148– 1992-present– Most influential papers:

image watermarking, image coding, image segmentation

IEEE TIT– H-index=214– 1963-present– Most influential papers:

cryptography, space-time codes, error-correcting codes, wavelets, ...


Highly-cited Papers related to Highly-cited Papers related to Image ProcessingImage Processing

Markov Random Field (Geman and Markov Random Field (Geman and Geman 1984) >8000 citationsGeman 1984) >8000 citations

Wavelet theory (Daubechies, Mallat, Wavelet theory (Daubechies, Mallat, Vetterli …) >180,000 citationsVetterli …) >180,000 citations

Why do they last?Why do they last?

23/4/1823/4/18

“the most fruitful areas for the growth of sciences were those which had been neglected as a no-man’s land between the various established fields.”

–Norbert Wiener


Connection 1: MRFConnection 1: MRFPixels vs. ParticlesPixels vs. Particles

23/4/1823/4/18

Pixel value = 0 or 1 Spin direction = up or down


A Little Bit History of Ising A Little Bit History of Ising ModelModel

Proposed by Ising in his PhD thesis in 1925Proposed by Ising in his PhD thesis in 1925

2D Ising model was analytically solved by L. 2D Ising model was analytically solved by L. Onsager in 1944 (who won the Nobel Prize Onsager in 1944 (who won the Nobel Prize in 1968)in 1968)

Phase transition behavior investigated by Phase transition behavior investigated by Yang and Lee in 1950sYang and Lee in 1950s

related to renormalization theory pioneered related to renormalization theory pioneered by RG Wilson (who won the Nobel Prize in by RG Wilson (who won the Nobel Prize in 1982)1982)


Apply Ising Model to ImagesApply Ising Model to Images

Applied to image restoration by Geman Applied to image restoration by Geman and Geman in 1984 (bring statistical and Geman in 1984 (bring statistical mechanics to engineering)mechanics to engineering)

Stirred up lots of interestStirred up lots of interest– More powerful image models (line process, More powerful image models (line process,

higher-order MRF)higher-order MRF)– More efficient optimization algorithms (Gibbs More efficient optimization algorithms (Gibbs

sampling, Swendsen-Wang, Wolff algorithm)sampling, Swendsen-Wang, Wolff algorithm)– New applicationsNew applications


Image ExampleImage Example

23/4/1823/4/18

original

noisy restored

Monte-CarloOptimization(minimize E)


Connection to Hopfield NetworksConnection to Hopfield Networks

Why is this model so influential?

The first-order approximation ofassociative memory in brain theory

Prof. Hopfield gave a talk at WVU on Mar. 13, 2007 titled “How Do We Think So Fast? From Neurons to Brain Computations,”


Statistical Mechanics and ITStatistical Mechanics and ITShannon was the first to recognize the connection Shannon was the first to recognize the connection between statistical mechanics and communication theorybetween statistical mechanics and communication theory

Connection with statistical mechanics also exists for Turbo Connection with statistical mechanics also exists for Turbo codes (belief propagation is related to Bethe free energy)codes (belief propagation is related to Bethe free energy)

““Multiuser detection and statistical mechanics” (Guo and Multiuser detection and statistical mechanics” (Guo and Verdu’ 2003)Verdu’ 2003)

““Evolution and structure of the Internet: A statistical Evolution and structure of the Internet: A statistical physics approach” (R Pastor-Satorras and A Vespignani‘ physics approach” (R Pastor-Satorras and A Vespignani‘ 2004)2004)

““Statistical mechanics of complex networks” (R. Albert’ Statistical mechanics of complex networks” (R. Albert’ PhD thesis in 2001)PhD thesis in 2001)

23/4/1823/4/18


If you think you have understood If you think you have understood entropyentropy

23/4/1823/4/18

““My greatest concern was what to call it. I thought of calling it My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it a better idea. Von Neumann told me, ‘You should call it entropyentropy, , for two reasons. In the first place your uncertainty function has for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already been used in statistical mechanics under that name, so it already has a name. has a name. In the second place, and more important, nobody In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have knows what entropy really is, so in a debate you will always have the advantagethe advantage. . ””

-Conversation between Claude Shannon and -Conversation between Claude Shannon and John von NeumannJohn von Neumann regarding what name to give to the “measure of uncertainty” or regarding what name to give to the “measure of uncertainty” or

attenuation in phone-line signals (1949)attenuation in phone-line signals (1949)


Connection II: Wavelet Theory Connection II: Wavelet Theory and Image Processingand Image Processing

Wavelet theory was established in late Wavelet theory was established in late 1980s by mathematicians, computer 1980s by mathematicians, computer scientists and electrical engineers togetherscientists and electrical engineers together

The most successful application of The most successful application of wavelets is likely to be lossy image wavelets is likely to be lossy image compression (e.g., JPEG2000)compression (e.g., JPEG2000)– Also popular in other processing tasks such Also popular in other processing tasks such

as segmentation, denoising and retrievalas segmentation, denoising and retrieval

The question is: Why? The question is: Why?


Where do Wavelets Come from?Where do Wavelets Come from?Before wavelet, people used Short-Time FT Before wavelet, people used Short-Time FT to analyze transient signalsto analyze transient signals

J. Morlet – a geophysical engineer at a J. Morlet – a geophysical engineer at a French oil company came up with an French oil company came up with an alternative approach which was recognized alternative approach which was recognized by Grossmann – Daubechies’ advisorby Grossmann – Daubechies’ advisor

S. Mallat – a graduate student at Penn met S. Mallat – a graduate student at Penn met Y. Meyer’s student and recognized its Y. Meyer’s student and recognized its connection to multi-resolution analysisconnection to multi-resolution analysis

23/4/1823/4/18


Laplacian Pyramids invented by RCA Engineers


At the Intersection of Math, CS At the Intersection of Math, CS and EEand EE

Math: construction of basis functions with good Math: construction of basis functions with good localization property in both time and frequencylocalization property in both time and frequency

CS: decomposes images under a multi-resolution CS: decomposes images under a multi-resolution analysis framework in analogy to HVSanalysis framework in analogy to HVS

EE: analysis-and-synthesis filter banks used by TV EE: analysis-and-synthesis filter banks used by TV engineersengineers

Merge of roots: a new tool for data/signal analysisMerge of roots: a new tool for data/signal analysis

Different perspectives: deterministic (Besov-space Different perspectives: deterministic (Besov-space functions) vs. statistical (heavy-tail distributions)functions) vs. statistical (heavy-tail distributions)

23/4/1823/4/18


Why Wavelets for Images?Why Wavelets for Images?

23/4/1823/4/18

Math: Besov-space function, statistics: sparse component analysis, neuroscience:Independent components of natural scenes


Beyond Image ProcessingBeyond Image ProcessingStatistics: nonparametric regressionStatistics: nonparametric regression

Graphics: progressive mesh compressionGraphics: progressive mesh compression

Turbulence: one of the most complicated Turbulence: one of the most complicated phenomenon in naturephenomenon in nature

Astronomy: hierarchical clustering theory of Astronomy: hierarchical clustering theory of galaxy formationgalaxy formation

Biomedical: MRI, EEG, PET, mammographyBiomedical: MRI, EEG, PET, mammography

Acoustic: computer music analysis Acoustic: computer music analysis

23/4/1823/4/18


What is Missing in Wavelet What is Missing in Wavelet Models?Models?

23/4/1823/4/18

DWT

sign flip

IWT


Connection III: Complex Connection III: Complex Networks and Image processingNetworks and Image processing

Common assumption made by MRF and Common assumption made by MRF and wavelet models: locality or Markovianwavelet models: locality or Markovian

Most existing physical laws are defined locally; Most existing physical laws are defined locally; but what about but what about nonlocalitynonlocality??

A great mystery in brain science is how it A great mystery in brain science is how it collectively processes local informationcollectively processes local information– Speed of nerve impulse transmission is much Speed of nerve impulse transmission is much

slower than that of logic gatesslower than that of logic gates– The power consumption of neural system is also The power consumption of neural system is also

much more efficient much more efficient

23/4/1823/4/18


Networks of NeuronsNetworks of Neurons

23/4/1823/4/18


Complex NetworksComplex Networks

Internet Internet

World Wide WebWorld Wide Web

Movie actor collaboration networkMovie actor collaboration network

Science collaboration networkScience collaboration network

Citation networksCitation networks

Cellular networksCellular networks

Ecological networksEcological networks

Power networksPower networks23/4/1823/4/18


How is it related to Image How is it related to Image Processing?Processing?

23/4/1823/4/18

B11

B22

B14B13B12

B41

B31

B21

B33B32

B23 B24

B34

B44B43B42

f3f2

f1

Bij

i

j1

2

3

4

1 2 3 4

Parallel and Distributed Processing (PDP)or connectionism was at the foundation of neural networks


How is it Different from NN?How is it Different from NN?What role does time play?What role does time play?– Temporal binding hypothesis in neuroscienceTemporal binding hypothesis in neuroscience– Synchronization of nonlinear oscillators in chemical, Synchronization of nonlinear oscillators in chemical,

biological and physical systemsbiological and physical systems

What role does feedback play?What role does feedback play?– As important as feedforwardAs important as feedforward– Mountcastle’s uniformity principle in psychologyMountcastle’s uniformity principle in psychology

Why does the network have to be hierarchical?Why does the network have to be hierarchical?– Natural world is organized in a hierarchical fashionNatural world is organized in a hierarchical fashion– Our perception of natural world is the consequence of mapping Our perception of natural world is the consequence of mapping

from outside (physical stimuli) to inside (synaptic connections)from outside (physical stimuli) to inside (synaptic connections)

23/4/1823/4/18


29.06dB 31.56dB 34.96dB

x y DT KR FG1

28.46dB 31.16dB 36.51dB

17.90dB 18.49dB 29.25dB

26.04dB 24.63dB 29.91dB

Experiment 1: Compressed SensingExperiment 1: Compressed Sensing

DT- DelauneyTriangle-based(griddata under MATLAB)

KR- KernalRegression-based(Takeda et al.IEEE TIP 2007w/o parameteroptimization)

1X. Li, “Patch-based image interpolation: algorithms and applications,” Inter. Workshop on Local and Non-Local Approximation (LNLA)’2008

25% kept

Xin Li

Nothing new here - just confirm nonuniform sampling could work better with our reconstruction algorithm.


Experiment 2: Image CodingExperiment 2: Image Coding

JPEG-decoded at rate of 0.32bpp(PSNR=32.07dB)

SFG-enhanced at rate of 0.32bpp(PSNR=33.22dB)

SPIHT-decoded at rate of 0.20bpp(PSNR=26.18dB)

SFG-enhanced at rate of 0.20bpp(PSNR=27.33dB)

Maximum-Likelihood (ML) Decoding

Maximum a Posterior (MAP) Decoding


Unfulfilled Connections (I)Unfulfilled Connections (I)

23/4/1823/4/18

Bayer Pattern(US3,971,065)

Cone distribution in human retina

CCD sensor design (engineering) could benefit from the organizationalprinciple of cones in human retina (biology)

Xin Li LDCSEE WVU Spring 2009Xin Li LDCSEE WVU Spring 2009 3434

Unfulfilled Connection (II)Unfulfilled Connection (II)

Q: Can we generate a HDR image (16bpp) by a standard camera?A: Yes, adjust the exposure and fuse multiple LDR images together

Xin Li LDCSEE WVU Spring 2009Xin Li LDCSEE WVU Spring 2009 3535

High Dynamic Range ImagingHigh Dynamic Range Imaging

Note that any commercial display devices we see these days are NOT HDR


Unfulfilled Connection (III)Unfulfilled Connection (III)

23/4/1823/4/18

Visual perception might be the first small step towards human intelligencebut it will be a huge leap in human intelligence (can brain understand brain?)


Summary and ConclusionsSummary and ConclusionsEvery theory, technology or system has its own Every theory, technology or system has its own evolution pathevolution path– Understand its connection is the most difficult yet important Understand its connection is the most difficult yet important

task task – My learning about image processing is still evolving, but I My learning about image processing is still evolving, but I

am hoping its principle can be also applied to other am hoping its principle can be also applied to other technical field such as communicationtechnical field such as communication

““If you truly believe that God creates this world in a If you truly believe that God creates this world in a unified fashion, when you get stuck with a problem, unified fashion, when you get stuck with a problem, seek your inspiration from around: nature, art and seek your inspiration from around: nature, art and other sciences. Essentially, the principles are the other sciences. Essentially, the principles are the same. ”same. ”


Research vs. Development

Good development/programming skills are a plus but secondary to good analytical/logical reasoning skills in my own assessment

Implementation skills should be viewed at the same level as mathematical skills; they are both technical tools but cannot replace scientific vision/understanding

``Knowledge and productivity are like compound interest.'' –Richard Hamming


How Good do You Need to at How Good do You Need to at Mathematics?Mathematics?

““Never be overwhelmed by the Never be overwhelmed by the mathematics other people are boasting in mathematics other people are boasting in their papers if you are in engineering: the their papers if you are in engineering: the more equations, the fewer ideas”more equations, the fewer ideas”

Mathematics is a language – you cannot Mathematics is a language – you cannot communicate well if you don’t master it; communicate well if you don’t master it; but you cannot advance science by simply but you cannot advance science by simply playing with mathematics playing with mathematics

23/4/1823/4/18

http://masterxinli.wordpress.com/2008/09/15/how-good-do-you-need-to-be-at-mathematics/


Why does engineering/math/science education in the US suck?

http://headrush.typepad.com/creating_passionate_users/2006/11/why_does_engine.html

Documents

Xin Li LDCSEE WVU Spring 2009 2015-6-3 From Gene to Meme Example of Memes: -Cultural ideas, symbols -Religions -Languages -Mathematics (e.g., Gaussian