A Transpositional Redundant Data Update Algorithm for Growing Server-less Video Streaming Systems

T. K. Ho and Jack Y. B. LeeDepartment of Information Engineering

The Chinese University of Hong Kong

Presented by T.K. HoP2P 2003, 1-3 Sep, Linköping, Sweden

Outline

� Background� Previous Works� Transpositional Redundant Data

Update (TRDU)� Multiple Redundant Nodes� Performance Evaluation� Conclusion and Future Works

Background� Server-less Video Streaming Systems[1]

� Pool of user hosts, or call nodes� Each node acts as both sender and receiver� Data redundancy to achieve reliability

Data Nodes

Redundant Nodes

ReceiverNetwork

......

[1] Jack Y. B. Lee and W. T. Leung, “Study of a Server-less Architecture for Video-on-Demand Applications,” Proc. IEEE International Conference on Multimedia and Expo., August 2002.

Background

� Data Placement Policy

v0 v1 v2 v3 c0,0 c0,1

d2d1d0 d3 r0 r1

v4

v8

v12

c1,0

c2,0

c3,0

c1,1

c2,1

c3,1

v5 v6 v7

v9 v10 v11

v13 v14 v15

v16 c4,0 c4,1v17 v18 v19

Same Parity Group

Data Nodes Redundant Nodes

Q-bytes Data Blocks

Q-bytes Redundant Data Blocks

v20 c5,0 c5,1v21 v22 v23

Background� New nodes join the system� Video data blocks are reorganized to utilize their

storage and streaming capacity� Video data blocks in the parity group change� Redundant data blocks need updating� Transmitting video data blocks to redundant nodes

v0 v1 v2 v3 c’0,0 c’0,1

d2d1d0 d3 r0 r1

v6

v12

v18

c’1,0

c’2,0

c’3,0

c’1,1

c’2,1

c’3,1

v7 v8 v9

v13 v14 v15

v19 v20 v21

v4

d4

v10

v16

v22

Data Blocksin Parity GroupCHANGE!

Redundant Data Blocks CHANGE!

v5

v11

v17

v23

d5

Previous Works

� Redundant Data Regeneration� Data blocks are fully distributed

• Transmit all data blocks to redundant node• Compute the new redundant

data in redundant node• Overhead is too high

RedundantNode

Data Nodes ...

� Redundant Data Regeneration� All data blocks stored in central server

• Regenerate new redundant data in server• Transmit them to redundant node• Overhead is low• Costly for server, not desirable/ feasible

Redundant Node

Central Server

Previous Works

Previous Works� Sequential Redundant Data Update (SRDU)

� Old redundant blocks contain valuable information� Possible to reuse for linear code, e.g. RSE code� Vandermonde matrix

• Number of nodes, N• Number of redundant nodes, h• Element at row i, column j = ji-1

block from data node 0

Redundant block in redundant node 0

,0 ,0

,1 ,1

1 1 1, 1 , 1

1 1 1 11 2 3

1 2 3 ( )

i i

i i

h h hi h i N h

c dc dN h

c dN h− − −− − −

− = −

…

…

� �� � � �

…

Previous Works

v0 v1 v2 v3 c’0,0 c’0,1v4

d2d1d0 d3 r0 r1d4

v0 v1 v2 v3 c0,0 c0,1

d2d1d0 d3 r0 r1

0

0,0 1

0,1 2

3

1 1 1 11 2 3 4

vc vc v

v

=

0

1

0,0 2

0,1 3

4

5

' 1 1 1 1 1 1' 1 2 3 4 5 6

vv

c vc v

vv

=

0,1 0 1 2 31* 2* 3* 4*c v v v v= + + +

Only need to send v4, v50,1 0,1 4 5' 5* 6*c c v v= + +

� Updating parity group 0� Before addition of nodes

� After addition of two nodes

v5

d5

Previous Works

� Updating parity group 1� Before addition of nodes

8

2,0 9

2,1 10

11

1 1 1 11 2 3 4

vc vc v

v

=

8

9

1,0 10

1,1 11

6

7

' 1 1 1 1 1 1' 1 2 3 4 5 6

vv

c vc v

vv

=

2,1 8 9 10 111* 2* 3* 4*c v v v v= + + +

1,1 2,1 6 7' 5* 6*c c v v= + +

v8 v9 v10 v11 c2,0 c2,1

d2d1d0 d3 r0 r1

� After addition of two nodes

v6 v7 v8 v9 c’1,0 c’1,1v10

d2d1d0 d3 r0 r1d4

v11

d5

Only need to send v6, v7

Previous Works� Problem in SRDU

� Reusing of old redundant block is limited� c’0,1 can only be in term of one of c0,1 and c1,1

� Both c0,1 and c1,1 are constructed by multiplying 1to 4 with the data blocks

7

0,1 0,14

' ( 1)* jj

c c j v=

= + +∑3

0,1 1,10

' ( 5)* jj

c c j v=

= + +∑

v0 v1 v2 v3 c’0,0 c’0,1

d2d1d0 d3 r0 r1

v8

v4

v12 c’1,0

c’2,0

c’1,1

c’2,1

v9 v10 v11

v5 v6 v7

v13 v14 v15

v16 v17 v18 v19

d4 d5 d6 d7

v20 v21 v22 v23

Transpositional Redundant Data Update� Compute redundant block using

transposed coefficients � Vandermonde matrix

� (Nmax-h) columns� Nmax is max system size, irrespective of current size N

max

,0,0

,1,1 max

1 1 1, 1, 1 max

1 1 1 11 2 3

1 2 3 ( )

ii

ii

h h hi N hi h

dcdc N h

dc N h− − −− −−

− = −

…

…

�� � � � �

…

Transpositional Redundant Data Update� For parity group i, the starting position is

shifted by (i(N-h) mod (Nmax-h))� All other elements are zeroed� Example

� Let (Nmax-h) be 10� Parity group 0

v0 v1 v2 v3 c0,0 c0,1

d2d1d0 d3 r0 r1

v4

v8

c1,0

c2,0

c1,1

c2,1

v5 v6 v7

v9 v10 v11

0

1

20,0

30,1

1 1 1 1 1 11 2 3 4 5 10

0

0

vvv

cv

c

=

�

�

�

0,1 0 1 2 31* 2* 3* 4*c v v v v= + + +

Transpositional Redundant Data Update� Parity group 1 and 2

� Repeat coefficient in c2,1

� Max size of 10 prevents v0 and v10 in residing in the same parity group

v0 v1 v2 v3 c0,0 c0,1

d2d1d0 d3 r0 r1

v4

v8

c1,0

c2,0

c1,1

c2,1

v5 v6 v7

v9 v10 v11

4

7

0000

00

v

v

�

1,1 4 5 6 75* 6* 7* 8*c v v v v= + + +

2,1 8 9 10 119* 10* 1* 2*c v v v v= + + +

10

11

8

9

0

0

vv

vv

�

Transpositional Redundant Data Update

� Two nodes are added� Updating parity group 0

v0 v1 v2 v3 c’0,0 c’0,1

d2d1d0 d3 r0 r1

v6 c’1,0 c’1,1v7 v8 v9

v4

d4

v10

v5

v11

d5

0

1

2

30,0

40,1

5

' 1 1 1 1 1 1' 1 2 3 4 5 10

0

0

vvvv

cv

cv

=

�

�

�

0,1 0 1 2 3 4 5

0,1 4 5

' 1* 2* 3* 4* 5* 6* 5* 6*c v v v v v v

c v v= + + + + += + + Need to send v4, v5

Transpositional Redundant Data Update

v0 v1 v2 v3 c’0,0 c’0,1

d2d1d0 d3 r0 r1

v6 c’1,0 c’1,1v7 v8 v9

v4

d4

v10

v5

v11

d5

10

11

1,0

1,16

7

8

9

0

' 1 1 1 1 1 10

' 1 2 3 4 5 10

vv

cc

vvvv

=

��

�

1,1 6 7 8 9 10 11

1,1 4 5 2,1

' (7* 8* ) (9* 10* 1* 2* ) ( 5* 6* )c v v v v v v

c v v c= + + + + += − − +

No need to send

Cached in previous update

� This equation is not allowed in SRDU, as v6 shares the same coefficient with v10

� Updating parity group 1

Transpositional Redundant Data Update

1,1 8 9 10 11 6 7

2,1 6 7

' (1* 2* 3* 4* ) (5* 6* ) 5* 6*c v v v v v v

c v v= + + + + += + +

8

9

1,0 10

1,1 11

6

7

' 1 1 1 1 1 1' 1 2 3 4 5 6

vv

c vc v

vv

=

1,1 4 5 6 71* 2* 3* 4*c v v v v= + + +

2,1 8 9 10 111* 2* 3* 4*c v v v v= + + +

� In SRDU� Before addition of nodes

� After addition of two nodes

Transpositional Redundant Data Update� Zero overhead point

� Number of additional nodes equals integral multiples of original system size

v0 v1 v2 v3 c’0,0 c’0,1

d2d1d0 d3 r0 r1

v4

d4

v5

d5

� Tradeoffs� Specify the max system size Nmax

� Larger Vandermonde matrix increases the computational complexity in decoding

v6

d6

v7

d7

0,1 0,1 1,1'c c c= +

Multiple Redundant Nodes� Total number of data blocks, B� Redundant Data Regeneration

� Fully distributed data blocks• Total overhead = B + (h-1)(B/(N-h))

All data blocks are sent, overhead = B

All redundant data are constructed here overhead = B / (N-h)

......

...

Multiple Redundant Nodes

� Data blocks are stored in central server• Total overhead = h * B/(N-h)

overhead = B / (N-h)

All redundant data are constructed in server

......

Multiple Redundant Nodes� SRDU/TRDU

� Total overhead = blocks under SRDU/TRDU + (h-1)(B/(N-h))

Blocks under TRDU or SRDU are transmitted

Compute partial result of each new redundant blocks

overhead = B / (N-h)

Combine the old redundant data and received partial result

1,1 1,1'c c P= +

4 55* 6*P v v= +

......

...

Multiple Redundant Nodes

� Overhead is dominated by updating the first redundant node

Performance Evaluation

� Simulation Details� One redundant node� One video title� 40,000 data blocks

� Metric� Redundancy Update Overhead

• Number of video data blocks needed to transmit for generating the new redundant data

Performance Evaluation� Continuous system growth

• Grow from 1 node to 400 nodes• One node added each time

100

1000

10000

100000

0 100 200 300 400

System Size (nodes)

Red

unda

ncy

Upd

ate

Ove

rhea

d (b

lock

s)

Regeneration by Fully Distributed Data

SRDU

TRDU

Regeneration by Archive Server

Performance Evaluation� Batched redundancy update

• Defer update until a fixed number of nodes, W• Initial system size = 80, W ranges from 1 to 200

10

100

1000

10000

100000

0 50 100 150 200

Batch Size W (nodes)

Per-n

ode

Red

unda

ncy

Upd

ate

Ove

rhea

d (b

lock

s)

TRDU, Starting at 80

SRDU, Starting at 80

Zero Overhead Points

Conclusion

� Reduce the redundancy update overhead significantly by� TRDU� Batched update

� Overhead in multiple redundant nodes is investigated� Additional overhead is insignificant

Future Works

� Node Removal� Addition of redundant nodes� Scheduling problem� Optimal batch size

Thank you

� Q&A Section