Compression 1 and 2

8/9/2019 Compression 1 and 2

http://slidepdf.com/reader/full/compression-1-and-2 1/55

296.3 Page 1

Algorithms in the Real World

Data Compression: Lectures 1 and 2



296.3 Page 2

Compression in the Real World

Generic File Compression– Files: gzip (LZ77), zip (!"rro#s$Wheeler),

!%A (&&')

– Archivers: ARC (LZW), &Zip (LZW)

– File systems: *+F

Comm"nication

– Fax: -+.$+ Gro"p / (r"n$length 0"11man)

– Modems: 2345is protocol (LZW),'*&6 (r"n$length0"11man)

– Virtual Connections



296.3 Page 3

Compression in the Real World

'"ltimedia– Images: gi1 (LZW), ig (conte8t),

peg$ls (resid"al), peg (trans1ormRLarithmetic)– V: 09+2 (mpeg$4)

– !ound: mp/An e8ample

%ther str"ct"res– Indexes: google, lcos

– Meshes "#or graphics$: edgerea;er– %raphs– Data&ases:

http://www.luratech.com/

http://www.luratech.com/



296.3 Page 4

Compression %"tline

Introduction:– Lossless <s3 loss

– 'odel and coder

– !enchmar;s

In#ormation heory: =ntrop, etc3

'ro&a&ility Coding: 0"11man Arithmetic Coding

Applications o# 'ro&a&ility Coding: &&' others

Lempel()iv Algorithms: LZ77, gzip, compress, 333*ther Lossless Algorithms: !"rro#s$Wheeler

Lossy algorithms #or images: >&=G, '&=G, 333

Compressing graphs and meshes: !!



296.3 Page 5

=ncoding?9ecoding

Will "se @message in generic sense to mean the datato e compressed

Encoder Decoder Input

Message

Output

MessageCompressed

Message

The encoder and decoder need to understand

common compressed ormat.



296.3 Page 6

Lossless <s3 Loss

Lossless: -np"t message B %"tp"t messageLossy: -np"t message ≈ %"tp"t message

Loss does not necessaril mean loss o1 "alit3 -n1act the o"tp"t co"ld e @etter than the inp"t3– 9rop random noise in images (d"st on lens)

– 9rop ac;gro"nd in m"sic

– Fi8 spelling errors in te8t3 &"t into etter1orm3

Writing is the art o1 loss te8t compression3



296.3 Page !

0o# m"ch can #e compressD

For lossless compression, ass"ming all inp"t messagesare <alid, i1 e<en one string is compressed, someother m"st e8pand3

Consider the shortest string m that is compressed, #here m

has length n3 "ppose m is mapped to mE o1 length nE n3 -1 allstrings o1 length nE or less are mapped to strengths o1 length nEor less, then #e ha<e a total o1

5 5H I 5nE H B 5nEH

inp"t strings mapped onto onl 5 5H I 5nE B 5nEH$H o"tp"tstrings3 o some string o1 length nE or less m"st e mapped to a

string o1 length greater than nE3

empt string strings o1 length H thro"gh nE m



296.3 Page "

'odel <s3 Coder

+o compress #e need a ias on the proailit o1messages3 +he model determines this ias

Mode#Coder

Pro$s.Compressed Message

Messages

Encoder

=8ample models:– imple: Character co"nts, repeated strings– Comple8: 'odels o1 a h"man 1ace

Input Message



296.3 Page 9

J"alit o1 Compression

R"ntime <s3 Compression <s3 Generalite<eral standard corp"ses to compare algorithms

e+g+ Calgary Corpus

5 oo;s, 6 papers, H iliograph,H collection o1 ne#s articles, / programs,H terminal session, 5 oect 1iles,H geophsical data, H itmap # image

+he Archive Comparison est maintains a

comparison o1 "st ao"t all algorithms p"licla<ailale

http://compression.ca/





296.3 Page 1%

Comparison o1 Algorithms

Program &#gor'thm T'me (PC )core

*+ ,- PPM 111115 1.!9 43%

(O& PPM /ar. 949! 1.91 4%!

PPMD PPM 112% 2.%! 265

IMP (0 1%3 2.14 254

(-IP (0 2%6 2.19 2!3

-IP ,-!! /ar. 195 2.59 31"

,-!! ,-!! 3.94



296.3 Page 11

Compression %"tline

Introduction: Loss <s3 Lossless, !enchmar;s, IIn#ormation heory:

– =ntrop

– Conditional =ntrop

– =ntrop o1 the =nglish Lang"age'ro&a&ility Coding: 0"11man Arithmetic Coding







296.3 Page 12

-n1ormation +heor

An inter1ace et#een modeling and coding,ntropy

– A meas"re o1 in1ormation content

Conditional ,ntropy

– -n1ormation content ased on a conte8t

=ntrop o1 the =nglish Lang"age

– 0o# m"ch in1ormation does each character in

@tpical =nglish te8t containD



296.3 Page 13

=ntrop (hannon HK4)

For a set o1 messages #ith proailit p(s), s ∈ ,the sel# in#ormation o1 s is:

'eas"red in its i1 the log is ase 53

+he lo#er the proailit, the higher the in1ormation

,ntropy is the #eighted a<erage o1 sel1 in1ormation3

H S p s p s s S

#og

=∈

∑ 1

i s p s

p s #og

#og = = −1



296.3 Page 14

=ntrop =8ample

25.25.25.257. =S p

24#og25.4 =×=S H

1256.1256.256.57.43 =S p

%625.%625.125.!57. =S p

!5.1"#og125.24#og25.2#og5. =×++=S H

2.116#og%625.2"#og125.34#og!5. =×++=S H



296.3 Page 15

Conditional =ntrop

+he conditional pro&a&ility p(sMc) is the proailit o1 s in a conte8t c 3 +he conditional sel# in#ormation is

+he conditional in1ormation can e either more or lessthan the "nconditional in1ormation3

+he conditional entropy is the #eighted a<erage o1 theconditional sel1 in1ormation

∑ ∑∈ ∈

=

C c S s c s pc s pc pC S H

8

1#og88

i s c p s c 8 #og 8 = −



296.3 Page 16

=8ample o1 a 'ar;o< Chain

b

p$8

p8$

p8 p$8$

.9

.1

.2

.8



296.3 Page 1!

=ntrop o1 the =nglish Lang"age

0o# can #e meas"re the in1ormation per characterDAC-- code B 7

=ntrop B 436 (ased on character proailities)

0"11man codes (a<erage) B 437

.ni8 Compress B /36

Gzip B 53N

!zip B H3K

=ntrop B H3/ (1or @te8t compression test)'"st e less than H3/ 1or =nglish lang"age3



296.3 Page 1"

hannonEs e8periment

As;ed h"mans to predict the ne8t character gi<enthe #hole pre<io"s te8t3 0e "sed these asconditional proailities to estimate the entropo1 the =nglish Lang"age3

+he n"mer o1 g"esses re"ired 1or right ans#er:

From the e8periment he predicted-",nglish$ . +/(1+0

: o guesses 1 2 3 4 5 ; 5

Pro$a$'#'t< .!9 .%" .%3 .%2 .%2 .%5



296.3 Page 19

Compression %"tline

Introduction: Loss <s3 Lossless, !enchmar;s, IIn#ormation heory: =ntrop, etc3

'ro&a&ility Coding:

– &re1i8 codes and relationship to =ntrop

– 0"11man codes

– Arithmetic codes







296.3 Page 2%

Ass"mptions and 9e1initions

Comm"nication (or a 1ile) is ro;en "p into piecescalled messages3

=ach message comes 1rom a message set B Os H ,I,s n P #ith a pro&a&ility distri&ution p(s)3

&roailities m"st s"m to H3 et can e in1inite3Code C"s$ : A mapping 1rom a message set to

codeords, each o1 #hich is a string o1 its

Message seuence: a se"ence o1 messages

3ote: Adacent messages might e o1 di11erenttpes and come 1rom di11erent proailitdistri"tions



296.3 Page 21

9iscrete or !lendedWe #ill consider t#o tpes o1 coding:

Discrete: each message is encoded a distinct seto1 its

– 0"11man coding, hannon$Fano coding

4lended: its can e @shared among messages

– Arithmetic coding

01001 11 0110001

message: H 5 / 4

010010111010

message: H,5,/, and 4



296.3 Page 22

.ni"el 9ecodale Codes

A varia&le length code assigns a it string (code#ord)o1 <ariale length to e<er message <al"e

e3g3 a = 1, b = 01, c = 101, d = 011

What i1 o" get the se"ence o1 its

1011 D-s it aba, ca, or, adD

A uniuely decoda&le code is a <ariale length code in#hich it strings can al#as e "ni"el decomposed

into its code#ords3



296.3 Page 23

&re1i8 Codes

A pre#ix code is a <ariale length code in #hich nocode#ord is a pre1i8 o1 another code#ord3

e3g3, a B , B HH, c B HHH, d B H

All pre1i8 codes are "ni"el decodale



296.3 Page 24

&re1i8 Codes: as a tree

Can e <ie#ed as a inar tree #ith message <al"esat the lea<es and s or Hs on the edges:

a B , B HH, c B HHH, d B H

b c

a

d

0

1

0 1

1

0



296.3 Page 25

ome &re1i8 Codes 1or -ntegers

n Binary Unary Gamma1 ..001 0 0|

2 ..010 10 10|0

3 ..011 110 10|1

4 ..100 1110 110|00

5 ..101 11110 110|01

6 ..110 111110 110|10

'an other 1i8ed pre1i8 codes:Golom, phased$inar, s"e8ponential, 333





296.3 Page 2!

Relationship to =ntrop

heorem "loer &ound$: For an: proailit:distri"tion p() #ith associated "ni"el:decodale code C,

heorem "upper &ound$: For an: proailit:distri"tion p() #ith associated optimal pre1i8code C,

H S l C a

≤

l C H S a ≤ +

1



296.3 Page 2"

ra1t 'c'illan -ne"alit

heorem "5ra#t(McMillan$: For an: "ni"el:decodale inar: code C,

Also, 1or an: set o1 lengths L s"ch that

there is a pre1i8 code C s"ch that

12

≤∑∈

−

C c

cl

12 ≤∑∈

−

Ll

l

88...1 Lil cl ii

==



296.3 Page 29

&roo1 o1 the .pper !o"nd (&art H)

Assign each message a length:We then ha<e

o the ra1t$'c'illan ine"alit there is a pre1i8

code #ith lengths l(s) 3

( ) 1#og s p sl =

( )

( )

2 2

2

1

1

1

−

∈

−

∈

−∈

∈

∑ ∑

∑∑

=

≤

=

=

l s

s S

p s

s S

p s

s S

s S

p s

#og =

#og =



296.3 Page 3%

&roo1 o1 the .pper !o"nd (&art 5)

( )

1

=1#og1

=1#og1

=1#og

S H

s p s p

s p s p

s p s p

cl s pS l

S s

S s

S s

S c s

a

+=+=

+⋅≤⋅=

=

∑

∑∑

∑

∈

∈

∈

∈

*o# #e can calc"late the a<erage length gi<en l(s)

And #e are done.



296.3 Page 31

Another propert o1 optimal codes

heorem: -1 C is an optimal pre1i8 code 1or theproailities Op H , I, p n P then p i Q p 7 implies l(c i ) ≤ l(c 7 )

'roo#: ( contradiction)

Ass"me l(c i ) Q l(c 7 )3 Consider s#itching codes c i andc 7 3 -1 l a is the a<erage length o1 the original code,the length o1 the ne# code is

+his is a contradiction since la is not optimal

l l p l c l c p l c l c

l p p l c l cl

a a j i j i j i

a j i i j

a

>

= + − + −

= + − −<



296.3 Page 32

0"11man Codes

-n<ented 0"11man as a class assignment in HK63.sed in man, i1 not most, compression algorithms

gzip, zip, peg (as option), 1a8 compression,I

'roperties:

– Generates optimal pre1i8 codes

– Cheap to generate codes

– Cheap to encode and decode

– la B 0 i1 proailities are po#ers o1 5



296.3 Page 33

0"11man Codes

-u##man Algorithm:tart #ith a 1orest o1 trees each consisting o1 a

single <erte8 corresponding to a message s and#ith #eight p(s)

6epeat until one tree le#t:– elect t#o trees #ith minim"m #eight roots p H

and p 5

– >oin into single tree adding root #ith #eight

p H p 5



296.3 Page 34

=8ample

p(a) B 3H, p() B 35, p(c ) B 35, p(d) B 36

a(.1) b(.2) d(.5)c(.2)

a(.1) b(.2)

(.3)

a(.1) b(.2)

(.3) c(.2)

a(.1) b(.2)

(.3) c(.2)

(.5)

(.5) d(.5)

(1.0)

a=000, b=001, c=01, d=1

0

0

0

1

1

1)tep 1

)tep 2)tep 3



296.3 Page 35

=ncoding and 9ecoding

,ncoding: tart at lea1 o1 0"11man tree and 1ollo#path to the root3 Re<erse order o1 its and send3

Decoding: tart at root o1 0"11man tree and ta;eranch 1or each it recei<ed3 When at lea1 can

o"tp"t message and ret"rn to root3

a(.1) b(.2)

(.3) c(.2)

(.5) d(.5)

(1.0)0

0

0

1

1

1

+here are e<en 1aster methods that

can process or /5 its at a time



296.3 Page 36

0"11man codes are @optimalheorem: +he 0"11man algorithm generates an optimal pre1i8

code3'roo# outline:-nd"ction on the n"mer o1 messages n3 !ase case n B 5 or nBHDConsider a message set #ith nH messagesH3 Least proale messages mH and m5 o1 are neighors in the

0"11man tree, constr"ction53 Replace the t#o least proale messages #ith one message#ith proailit p(mH) p(m5) ma;ing ne# message set E

/3 Can trans1orm an optimal pre1i8 code 1or so that mH andm5 are neighors3

43 Cost o1 optimal code 1or is cost o1 optimal code 1or E

p(mH) p(m5)63 0"11man code 1or E is optimal ind"ctionN3 Cost o1 0"11man code 1or is cost o1 optimal code 1or E

p(mH) p(m5)3



296.3 Page 3!

&rolem #ith 0"11man Coding

Consider a message #ith proailit 3KKK3 +he sel1in1ormation o1 this message is

-1 #e #ere to send H s"ch messages #e mighthope to "se H3H4 B H344 its3

.sing 0"11man codes #e re"ire at least one it permessage, so #e #o"ld re"ire H its3

%%144.999#og. =−



296.3 Page 3"

Arithmetic Coding: -ntrod"ction

Allo#s @lending o1 its in a message se"ence3%nl re"ires / its 1or the e8ample

Can o"nd total its re"ired ased on s"m o1 sel1in1ormation:

.sed in 9'', >&=G?'&=G (as option), zip (&&'d)

'ore e8pensi<e than 0"11man coding, "t integer

implementation is not too ad3

∑=

+<n

ii

sl 1

2



296.3 Page 39

Arithmetic Coding: message inter<als

Assign each proailit distri"tion to an inter<alrange 1rom (incl"si<e) to H (e8cl"si<e)3

e3g3

a ? .2

c ? .3

$ ? .5

0.0

0.2

0.7

1.0

f(a) = .0, f(b) = .2, f(c) = .7

∑

−

==

1

1

i

j j pi f

+he inter<al 1or a partic"lar message #ill e calledthe message interval (e3g3, 1or the inter<al is S35,37))



296.3 Page 4%

Arithmetic Coding: se"ence inter<als

Code a message se"ence composing inter<als3For e8ample: &ac

+he 1inal inter<al is 7+289+0$

We call this the seuence interval

a ? .2

c ? .3

$ ? .5

%.%

%.2

%.!

1.%

a ? .2

c ? .3

$ ? .5

%.2

%.3

%.55

%.!

a ? .2

c ? .3

$ ? .5

%.2

%.21

%.2!

%.3



296.3 Page 41

Arithmetic Coding: se"ence inter<als

+o code a se"ence o1 messages #ith proailities pi (i = 1..n) "se the 1ollo#ing:

=ach message narro#s the inter<al a 1actor o1 p i 3

Final inter<al size:

iii

iiii

p s s p s

f sl l f l

111

1111

−

−−

==

+==

∏==

n

iin p s 1

ottom o1 inter<al

size o1 inter<al



296.3 Page 42

Warning

+hree tpes o1 inter<al:– message interval : inter<al 1or a single message

– seuence interval : composition o1 messageinter<als

– code interval : inter<al 1or a speci1ic code "sedto represent a se"ence inter<al (disc"ssedlater)



296.3 Page 43

.ni"el de1ining an inter<al

Important property:+he se"ence inter<als 1ordistinct message se"ences o1 length n #ill ne<ero<erlap

here#ore: speci1ing an n"mer in the 1inal

inter<al "ni"el determines the se"ence39ecoding is similar to encoding, "t on each step

need to determine #hat the message <al"e is andthen red"ce inter<al





296.3 Page 45

Representing Fractions

!inar 1ractional representation:

o ho# ao"t "st "sing the smallest inar 1ractionalrepresentation in the se"ence inter<al3

e3g3 S,3//) B 3 S3//,3NN) B 3H S3NN,H) B 3HH!"t #hat i1 o" recei<e a HD

ho"ld #e #ait 1or another HD

1%11.16=11

%1%1.3=1

11.!5.

=

==



296.3 Page 46

Representing an -nter<al

Can <ie# inar 1ractional n"mers as inter<als considering all completions3 e3g3

We #ill call this the code interval+

m'n ma@ 'nterAa#

. . . B. .

. . . B. . 11 11% 111 !51%1%1 1%1% 1%11 625 !5



296.3 Page 4!

Code -nter<als: e8ample

+r code S,3//) B 3 S3//,3NN) B 3H S3NN,H) B 3HH

%

1

3I

3HHI3HI

*ote that i1 code inter<als o<erlap then one code is

a pre1i8 o1 the other3Lemma: -1 a set o1 code inter<als do not o<erlap

then the corresponding codes 1orm a pre1i8 code3



296.3 Page 4"

electing the Code -nter<al

+o 1ind a pre1i8 code 1ind a inar 1ractional n"mer#hose code inter<al is contained in the se"enceinter<al3

Can "se the 1raction l s?5 tr"ncated to

its

.61

.!9

.625

.!5)euence InterAa# Code InterAa# .1%1

s s #og12#og −+=−



296.3 Page 49

electing a code inter<al: e8ample

S,3//) B 3H S3//,3NN) B 3H S3NN,H) B 3HH

3H

3HH

e3g: 1or S3//I,3NNI), l B 3//I, s B 3//I

l s?5 B 36 B 3HI

tr"ncated to its is 3H

-s this the est #e can do 1or S,3//) D

3//

3NN

H

3H

333#og.1#og1 =−+=−+ s



296.3 Page 5%

6ealArith =ncoding and 9ecoding

6ealArith,ncode:9etermine l and s "sing original rec"rrences

Code "sing l s?5 tr"ncated to H$log s its

6ealArithDecode:

Read its as needed so code inter<al 1alls #ithin amessage inter<al, and then narro# se"enceinter<al3

Repeat "ntil n messages ha<e een decoded 3



296.3 Page 51

!o"nd on Length

heorem: For n messages #ith sel1 in1ormation Os H ,I,s n P RealArith=ncode #ill generate at most

its3

'roo#: 1 1

1

1

2

1

1

1

1

+ − = + −

= + −

= +

< +

=

=

=

=

∏∑

∑∑

#og #og

#og

s p

p

s

s

ii

n

ii

n

ii

n

ii

n

∑=

+n

i

i s1

2



296.3 Page 52

-nteger Arithmetic Coding

&rolem #ith RealArithCode is that operations onaritrar precision real n"mers is e8pensi<e3

5ey Ideas o# integer version:

eep integers in range S33R) #here RB5;

.se ro"nding to generate integer se"ence inter<al

Whene<er se"ence inter<al 1alls into top, ottom ormiddle hal1, e8pand the inter<al 1actor o1 5

+his integer Algorithm is an appro8imation or thereal algorithm3



296.3 Page 53

-nteger Arithmetic Coding

+he proailit distri"tion as integers&roailities as co"nts:

e3g3 c(a) B HH, c() B 7, c(c) B /

+ is the s"m o1 co"nts

e3g3 4 (HH7/)&artial s"ms 1 as e1ore:

e3g3 1(a) B , 1() B HH, 1(c) B H

Re"ire that R Q 4+ so that

proailities do not get ro"nded tozero a ? 11

c ? 3%

$ ? !

%

11

1"

T?4"

*?256



296.3 Page 54

-nteger Arithmetic (contracting)

l . 9 u . 6(1

T m f sl l

T m f sl u

l u s

iiii

iiii

iii

=

1=1

1

1

1

11

⋅+=

−+⋅+=+−=

−

−

−−

%

*?256

u

l



-nteger Arithmetic (scaling)

-1 l ≥ R?5 then (in top hal#)%"tp"t H 1ollo#ed m set m B cale message inter<al e8panding 5

-1 " R?5 then (in &ottom hal#)%"tp"t 1ollo#ed m Hset m B cale message inter<al e8panding 5

-1 l ≥ R?4 and " /R?4 then (in middle hal#$-ncrement mcale message inter<al e8panding 5

Documents

Compression 1 and 2