View
3.980
Download
9
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Towards a Theory of Semantic Communication
Jie Bao, RPI
Joint work with Prithwish Basu, Mike Dean, Craig Partridge, Ananthram
Swami, Will Leland and Jim Hendler 1
Outline
• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
2
Shannon, 1948
“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;... These semantic aspects of communication are irrelevant to the engineering problem.”
3Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 625-56, 1948.
message
message
Signal
Signal
But, are these just sequences of bits?
• Movie streams• Software codes• DNA sequences• Emails• Tweets• ……
4
“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;..”“These semantic aspects of communication are irrelevant to the engineering problem”?
Between a Talent Manager & Me
“Are you open to discuss greener pastures”?
5
“Thanks for contacting me. However, I'm not sure if my research is related to "greener pastures". I'm a computer scientist.”
Misunderstanding can be costly
6
Mars Climate Orbiter (1998-1999), $125 million
Expressed
Pound (lbF)
Interpreted
Newton (N)
Image Source: Wikipedia, http://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Communications_loss
Misunderstanding can be deadly
Afghan National Army (ANA) to ISAF• “Launch flares over the left side of the village”
Received and Understood as• “fire on the left side of the village”
Alternative semantic coding (e.g., illuminating shell) may save lives!
7Scenario based on report from http://www.closeprotectionworld.co.uk/security-news-asia/37466-afghanistan-war-what-happens-when-war-interpreter-doesnt-know-language.html
(Noisy) Battlefield Communication (Noisy) Battlefield Communication
Our Contributions
• We develop a generic model of semantic communication, extending the classic model-theoretical work of (Carnap and Bar-Hillel 1952) ;
• We discuss the role of semantics in reducing source redundancy, and potential approaches for lossless and lossy semantic data compression;
• We define the notions of semantic noise, semantic channel, and obtain the semantic capacity of a channel.
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
9
(Classical) Information Theory Semantic Information Theory
Shannon, 1948
message
message
Shannon ModelShannon Model
Signal
Signal
ExpressedMessage(e.g., commands and reports)
Expressed Message
Semantic Channel
From IT to SIT
A Three-level Model (Weaver)
Transmitter Receiver
Destination Destination Source Source
Physical Channel
Technical message
Technical Noise
Intended message
Expressed message
Semantic Transmitter
Semantic Transmitter
Semantic ReceiverSemantic Receiver
Semantic Noise
Semantic Noise
Shared knowledge
Shared knowledge
Local knowledge
Local knowledge
Local knowledge
Local knowledge
(effectiveness factors)
C: Effectiveness
B: Semantic
A: Technical
A Semantic Communication Model
12
Message generator
World model
Background Knowledge
Inference Procedure
Messages
Sender
Message interpreter
World model
Background Knowledge
Inference Procedure
Receiver
Ws Wr
Ks KrIs Ir
{m}
World
M: Message Syntax
Feedback (?)
observations
Ms Mr
Semantic Sources
• Key: A semantic source tells something that is “true”– Engineering bits are neither true or false!
• Goal: 1) more soundness (sent as “true”->received as “true”); 2) less ambiguity
13
Outline
• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
14
Measuring Semantic Information
• Basic Problem: What is the amount of “semantics” carried by a source and its messages?
15
Measuring Semantic Information
• Statistical approach: Inference may change the distribution of symbols, hence the entropy of the source.
• Model-theoretical approach: The less “likely” a message is to be true, the more information it contains.
• Algorithmic approach: What’s the minimal program needed to describe messages and their deductions?
• Situation-theoretical approach: measuring the divergence of messages to “truth”.
16
Our ApproachOur Approach
Shannon: Information = “surpriseness”
17
H(tyrannosaurus) > H(dog)H(tyrannosaurus) > H(dog)
Captured from: http://www.wordcount.org/main.php
Which sentence is more “surprising”?
18
``Rex is not a tyrannosaurus''
``Rex is not a dog''
????
Model Semantics
• tyrannosaurus • dog
19
??
“Semantics” of DNA
20Image courtesy: http://www.yourdictionary.com/dna http://www.pnl.gov/biology/images/protein_molecule.jpg
“Syntax” Model (“Semantics”)
Gene expression
Stone-age Semantic Communication
• Semantic communication predates symbolic communications
21Altamira Cave Painting http://mandyking.files.wordpress.com/2011/02/altamira-cave.jpg
Semantics of Messages
• Messages are expressions, not just sequences of symbols– E.g., Saturday->Weekend, Sunny & Cold
• If an expression is more commonly true, it contains less semantic information– inf (Sunny & Cold) > inf (Cold)– inf (Cold) > inf (Cold or Warm)
22
Semantics of Messages
• Carnap & Bar-Hillel (1952) - “An outline of a theory of semantic information”
m(exp) = |mod(exp)| / |all models|
inf(exp) = - log m(exp)
• Example– m(A v B) = ¾, m(A ^ B)=1/4– Inf(A v B)=0.415, inf(A^B )= 2
23
Knowledge Entropy
• Extending Carnap & Bar-Hillel (1952) – Models have a distribution– Background knowledge may present
Weekend=2/7, Saturday=1/7
Knowledge Entropy
• Logical prob. and knowledge entropy of Messages
• Model entropy of an information source
25
model distribution
logical probability
Semantic Information Calculator (Demo)
• http://www.cs.rpi.edu/~baojie/sit/index.php
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
27
Conditional Knowledge Entropy
• When there is background knowledge, the set of possible worlds decreases.
28
Model Compression with Shared Knlg
• Background knowledge (A->B), when shared, help compress the source– Side information in the form of entailment
29
Lossless Message Compression
• Theorem : There is a semantically lossless code for source X, with message entropy H >= H(Xeq); no such code exists for H < H(Xeq)
– Xeq are equivalent classes of X
• Example: no need for coding both “pig” and “swine”, using one of them is sufficient.
• Example 2: a->(a^b)v(b^c) = a->b• Sometime, the loss is intentional compression
– Textual description of an image– Abstract of a paper
Other Source Coding Strategies
• Lossless model compression– E.g., using minimal models
• Lossy message compression– E.g., compressing based on semantic similarity
• Leave as future work
31
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
32
Semantic Noise
Examples
• The meaning of a message is changed due to technical noises, e.g., from ``flare'' to ``fire'‘;
• Semantic mismatch: The source / receiver use different background knowledge or inference (e.g., during the loss of the Mars Climate Orbiter);
• Lost in translation: “Uncle” in English has no exact correspondence in Chinese.
33
Semantic Noise and Channel Coding
34
“coffee machine”“copy machine”
“Xerox” “Xerox”
“copy machine”
p->ff
?
?
0.9
0.1
1.0
W X Y W’
Scenario developed based on reports in http://english.visitkorea.or.kr/enu/AK/AK_EN_1_6_8_5.jsp and http://blog.cleveland.com/metro/2011/03/identifying_photocopy_machine.html
Semantic Channel Coding Theorem
• In the simplified model, assume no semantic mismatch (Ks=Kr, Is=Ir)
• Theorem 3: If transmission rate is smaller than Cs (semantic channel capacity), error-free coding exists
• Semantic channel capacity may be higher or lower than the engineering channel capacity (sup I(X;Y)) !– H(W|X) stands for encoder’s semantic ambiguity – avg(inf(Y)) is receiver’s “smartness”
35
Outline
• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel
coding) • Path ahead
36
Application in Coding & Validation
• Hypothesis 1: using semantics we can achieve better data compression
• Hypothesis 2: using semantics we can achieve more reliable communication
• Validation with comparison to non-semantic algorithms
Extensions
• Extensions & connections to other fields
– First-order languages [probabilistic logics]– Inconsistent KBs (misinformation) [paraconsistent
logics]– Lossy source coding [clustering and similarity
measurement]– Semantic mismatches [extending Juba & Sudan
2011]– … …
Path ahead – Broad Impact
– Communications (e.g., coding)– Linguistics (e.g., entropy of English)– Biology (e.g., semantics of genes)– Economics – ….– Areas wherever Shannon’s theory applies – And beyond (e.g., Semantic Web, ontology
engineering)
Questions?
40Image courtesy: http://www.addletters.com/pictures/bart-simpson-generator/900788.htm