View
6
Download
0
Category
Preview:
Citation preview
© 2011 Toshiba Corporation
Evaluation of TIPCMitsuhiro KimuraTOSHIBA CORPORATION
CELF Japan Technical Jamboree #37, May 20, 2011
2© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Background
• Integrating one more SoC in the embedded system for adding more functions.
• We will discuss the interconnection of two SoCs:– It needs to choose a bus from available interface buses in
SoCs.– Ethernet is more commonly available.
• Required a quick and lightweight protocol stack on Ethernet.
SoC A
SoC BProduct
SoC: System-on-a-chip
3© 2010 Toshiba Corporation© 2011 Toshiba Corporation
TIPC : Transparent Inter-Process Communication
• The protocol specifically designed for cluster communication – quick and lightweight
• Supposing little packet loss and infrequent retransmission– Socket I/F
• sockets corresponding TCP, UDP.– Built in Linux Kernel.
TIPC:Kernel: 2.6.35 ~
2.0.01.6, 1.72.6.16 ~ 34
1.5.122.6.9 ~ 14
1.4.32.4, 2.6
5/14 1.7.7
4© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Evaluation environment
• Interconnection of SoCs in embedded systems is similar to cluster connections in that both of them are little packet loss intra-communications.
• Testing SoC’s used for embedded systems– (reference [2], evaluation with Xeon, gigabit ethernet)
• Benchmarking throughput as well as CPU load– MIPS32®24K® 533MHz ×2, 100Mbps Ethernet– Linux Kernel 2.6.20.19, TIPC Version 1.7.7-rc1
SoC A
BoardSoC B
5© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Test cases
• (1) Max throughput– Send data in one direction.– Measure throughput and CPU load
per message size• (2) Latency
– Data flows in both directions alternately in a ping-pong fashion.
– Measure the roundtrip time per message size.
• (3) CPU load for Streaming– Send data in one direction at a constant rate.– Measure the CPU load.
6© 2010 Toshiba Corporation© 2011 Toshiba Corporation
(1) Max throughput, CPU load
• Message size: 512, 1024, 1536, … 40448Byte– Watch a change depending on message size.– Compare throughput and CPU load of TIPC with TCP/IP.
• Measure the time of send()/recv() 10,000 times.
512×n Byte
SoC A SoC B
7© 2010 Toshiba Corporation© 2011 Toshiba Corporation
(1) Max throughput
• TCP/IP < TIPC– Header: TCP/IP:40Byte, TIPC:24Byte, MTU:1500Byte– Small header high transfer efficiency
77
79
81
83
85
87
89
512 10624 20736 30848 40960
Thro
ughp
ut [M
bps]
Message size [Byte]
Message size - Throughput
8© 2010 Toshiba Corporation© 2011 Toshiba Corporation
(1) Max throughput
• TIPC throughput drops in case one more packet needs to be sent.
8383.5
8484.5
8585.5
8686.5
8787.5
88
2048 2432 2816 3200 3584
Thro
ughp
ut [M
bps]
Message size[Byte]
Message size - Throughput 2952 Byte= (1500-24)×2
3449
packet: 23
9© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Reason of TIPC throughput drop
• 1500Byte – 24Byte = 1476Byte.2953Byte = 1476Byte × 2 + 1Byte
TIPC:
TCP/IP:
1460 14601460SoC A SoC B1460 1460
send() send()
• 1500Byte – 40Byte = 1460Byte.
1476 14761SoC A SoC B1476 14761
send() send()
10© 2010 Toshiba Corporation© 2011 Toshiba Corporation
(1) CPU load
• CPU load of TIPC is less than TCP/IP
0
10
20
30
40
50
512 10624 20736 30848 40960
CPU
load
(%)
Message size[Byte]
send
0
20
40
60
80
100
512 10624 20736 30848 40960
CPU
load
(%)
Message size[Byte]
recv
11© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Packets observance
• TIPC performance in case of small size packets– Counting RX/TX packets.
SoC A SoC BMessage
eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XXinet addr:X.X.X.X Bcast:X.X.X.X Mask:X.X.X.XUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:0 errors:0 dropped:0 overruns:0 frame:0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
ifconfig
12© 2010 Toshiba Corporation© 2011 Toshiba Corporation
TCP/IP TIPC
Message Packets: TCP/IP < TIPC
0
50000
100000
150000
200000
250000
300000
0 50 100 150 200
Num
ber o
f pac
kets
Time [sec]
0
50000
100000
150000
200000
250000
300000
0 20 40 60 80
Num
ber o
f pac
kets
Time [sec]
• Message size: 512 Byte• TCP/IP small packets coalesce.
Send packetsRecv packets
13© 2010 Toshiba Corporation© 2011 Toshiba Corporation
TCP/IP TIPC
ACK: TCP/IP > TIPC• Message size: 4096 Byte• TIPC is less frequent ACK.
0
10000
20000
30000
40000
50000
0 20 40 60 80
Num
ber o
f pac
kets
Time [sec]
ACK
05000
1000015000200002500030000350004000045000
0 20 40 60 80
Num
ber o
f pac
kets
Time [sec]
ACK
Low CPU load
Send packetsRecv packets
14© 2010 Toshiba Corporation© 2011 Toshiba Corporation
(2) Latency
• TCP/IP’s latency nearly equals TIPC’s.
00.0010.0020.0030.0040.0050.0060.0070.0080.009
512 10624 20736 30848 40960
Late
ncy[
sec]
Message size [Byte]
Message size - Latency SoC B
SoC A
・・・
Latency
15© 2010 Toshiba Corporation© 2011 Toshiba Corporation
CPU load[%]
(3) CPU load in sending streaming data.
• Data transmission rate: 25Mbps
TCP/IP TIPC TCP/IP - TIPCSoC A(send) 9.16 5.18 3.98SoC B(recv) 19.25 10.25 9.00
SoC BSoC A
nanosleep()80msec
256KByte
16© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Use TIPC (1/2)
• TIPC doesn’t support SO_SNDTIMEO. Blocking-send is hard to use. Nonblocking-send repeat. CPU load increases. (test program: ~5%)
17© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Use TIPC (2/2)
• Processes/threads issue a lot of send request TIPC send queue is filling up.
(congestion control)– Increase processes waiting for queue to decrease.– Even if queue decreases, and processes get up, wait
again without issue a send request.– Need to schedule between processes.
TIPCsend queue
Send packets
20© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Conclusion
• TIPC is faster than TCP/IP– Smaller header size– TCP/IP is faster in case of small message size.
• Merging small messages would improve the throughput.
• TIPC has lower CPU load than TCP/IP– Less frequent ACK (other reasons would exist)
• Verified that TIPC is useful for embedded systems.– (e.g. Reducing CPU load when transferring streaming
data.)
21© 2010 Toshiba Corporation© 2011 Toshiba Corporation
Reference• Reference:1. TIPC Home Page, http://tipc.sourceforge.net/2. Florian Westphal, 2007, TIPC Analysis and Optimization,
http://webuser.hs-furtwangen.de/~reich/AdvancedMiddlewareWorkshop.SS07
3. Mitsuhiro Kimura: “Application of TIPC to communicate between CPUs in an embedded device and performance comparison to TCP/IP”, The 73rd National Convention of IPSJ 6G-3 1-43 – 1-44 (2011-3)
Recommended