Performance Evaluation of JXTA-* Communication Layers

Mathieu JanPARIS Research Group

Paris, November 2004

2

Outline

Overview of JXTA communications layers

Experimental setup and benchmarks used

Evaluation of JXTA-J2SE Fast-Ethernet Myrinet

Evaluation of JXTA-C Fast-Ethernet Myrinet

3

Motivations: performance of JXTA?

Papers from Emir Halepovic Pipe service only No in-depth explanations about the results No precise details about the experimental

setup Mainly JXTA 1.x Complex topologies (relays, etc)

Not clear what is benchmarked

Results from the P3 project =~ XtremWeb over JXTA

4

5

6

Project JuxTest

A set of benchmarks for JXTA-J2SE 2.x Focuses of JXTA-J2SE Communications layers Uses JXTA Distributed Framework project (JDF)

Co-written with David A. Noblet Undergraduate at the University of New

Hampshire

Source code soon available! (22/11) http://jdf.jxta.org/ Results already on-line

7

JXTA-J2SE communications layers

Endpoint service Endpoint service

Pipe service Pipe service

JXTA Socket JXTA Socket

TCP, HTTP, etc

8

JXTA-J2SE communication layers

JXTA Socket Java Socket interface Data-stream interface Reliability

Pipe service Pipe ID Unreliable

Endpoint service Peer ID Unreliable

9

Experimental setup

Nodes 2.4GHz Intel P4, 1GB of RAM, linux 2.4.22

Networks Fast-Ethernet (MTU of 1500) Myrinet (GM 2.0.11, MTU of 9000)

JVMs Sun 1.4.2_01-b06 (-server -Xms256M -

Xmx256M) IBM 1.4.1 (-Xms256M -Xmx256M)

JXTA-J2SE peers via JDF 1 rdv peer and 2 edge peers

Benchmarks used RTT and protocol efficiency

Performance Evaluation of JXTA-J2SE on Fast-

Ethernet

11

Raw bandwidth results of JXTA Socket

Java socket = 11.22 MB/s

JXTA 2.3 = 9.72 MB/s

JXTA 2.2.1 = 9.48 MB/s

12

Raw latency results of JXTA Socket

Java socket = < 0.1 ms

JXTA 2.3 =~ 3 ms

JXTA 2.2.1 =~ 4 ms

13

Protocol efficiency of JXTA Socket

4 message elements in a JXTA message ACK_NUMBER

Message payload + reliability EndpointRouterMsg (XML doc)

Used by the endpoint router service 565 bytes! Not even needed!

EndpointSourceAddress TCP address

EndpointDestinationAddress TCP address + EndpointService (Net group) +

Endpoint Router

1 byte = 909 bytes sent on the wires!

14

Tunning the output buffer size of JXTA Socket

output buffer size =

size of a pipe message

Java socket = 11.22 MB/s

JXTA 2.2.1 (512KB) = 11.12 MB/s

15

Raw bandwidth results of unicast pipe

Java socket = 11.22 MB/s

JXTA 2.3 = 9.59 MB/s

JXTA 2.2.1 = 10.74 MB/s

16

Raw latency result of unicast pipe

Java socket = < 0.1 ms

JXTA 2.3 =~ 2 ms

JXTA 2.2.1 =~ 35 ms

17

Protocol efficiency of unicast pipe

4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress

1 byte = 877 bytes sent on the wires!

18

JXTA Socket vs unicast pipe (bandwidth)

JXTA Socket = 9.72 MB/s

JXTA unicast pipe = 9.59 MB/s

19

JXTA Socket vs unicast pipe (latency)

JXTA Socket =~ 3 ms

JXTA unicast pipe =~ 2 ms

20

Removing limit on message size (unicast

pipe)

Java socket = 11.22 MB/s

JXTA 2.2.1 = 11.14 MB/s

21

IBM JVM vs Sun JVM on unicast pipe

JXTA 2.3 IBM =~ 1.2 ms

JXTA 2.3 SUN =~ 1.8 ms

22

Raw bandwidth results of endpoint service

Java socket = 11.22 MB/s

JXTA 2.3 = 10.47 MB/s

JXTA 2.2.1 = 11.01 MB/s

23

Raw latency results of endpoint service

Java socket = < 100 us

JXTA 2.3 IBM =~ 480 us

JXTA 2.3 SUN =~ 730 us

Issue 1228: 2 TCP messages for 1 JXTA message!

24

Protocol efficiency of endpoint service

3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress

1 byte = only 300 bytes sent on the wires! If we can say only …

No EndpointRouterMsg message element Removed 565 bytes! Removed XML parsing

25

Cost of each layer (bandwidth)

26

Cost of each layer (latency)

Performance Evaluation of JXTA-J2SE on Myrinet

Or how to desperately reach the Gb/s!

28

Experimental setup

« Ethernet emulation » feature of GM No OS-bypass mode :-(

Same benchmarks A different IP address when configuring JXTA

Focus on bandwidth Removed limit on message size

29

JXTA Socket bandwidth results

Results are not very

good!Java socket = 159.20 MB/s

JXTA 2.3 = 14.70 MB/s

JXTA 2.2.1 = 12.10 MB/s

30

Tuning the output buffer size of JXTA Socket

Above 80 MB/s

31

Unicast pipe bandwidth results

Above 1 Gb/s! :-)

Java socket = 159.20 MB/s

JXTA 2.3 = 106.54 MB/s

JXTA 2.2.1 = 136.78 MB/s

32

Endpoint service bandwidth results

Java socket = 159.20 MB/s

JXTA 2.3 = 112.32 MB/s

JXTA 2.2.1 = 144.97 MB/s

Performance Evaluation of JXTA-C on Fast-

Ethernet

34

Short overview of JXTA-C

Needs Apache Portable Runtime (APR) Threads, Network, etc

Newly added support for TCP communications

PeerCom, Seoul (South Korea) Does not support rdv peers

Need to use a J2SE rdv peer No JXTA Socket layer

Only pipe service and endpoint service Several bug fixes

Mostly EndpointRouterMsg TCP_NODELAY on

Several TCP messages

35

Raw bandwidth results of unicast pipe

Similar results compared to JXTA-J2SE

JXTA-J2SE 2.3 = 11.1 MB/s

JXTA-J2SE 2.2.1 = 11.1 MB/s

JXTA-C = 11.1 MB/s

36

Raw latency results of unicast pipe

JXTA-J2SE 2.3 =~ 1.8 ms

JXTA-J2SE 2.2.1 =~ 35 ms

JXTA-C =~ 2 ms

37

Protocol efficiency of unicast pipe

4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress

1 byte = 834 bytes sent on the wires! JXTA-J2SE = 877 bytes + GatewayForward tag in EndpointRouter

Waiting for Bernard Traversat answear - charset tag for each message element

38

Raw bandwidth results of endpoint service

JXTA-J2SE 2.3 = 11.15 MB/s

JXTA-J2SE 2.2.1 = 11.19 MB/s

JXTA-C = 11.16 MB/s

39

Raw latency results of endpoint service

JXTA-J2SE 2.3 =~ 0.8 ms

JXTA-J2SE 2.2.1 =~ 1 ms

JXTA-C =~ 0.8 ms

40

Protocol efficiency of endpoint service

3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress

1 byte = only 239 bytes sent on the wires! If we can say only … JXTA-J2SE = 300 bytes

No EndpointRouterMsg Removing 595 bytes! Removed XML parsing

41

Unicast pipe vs endpoint service

Performance Evaluation of JXTA-C on Myrinet

Still using « Ethernet emulation » feature of GM

43

Unicast pipe bandwidth results

JXTA-J2SE 2.3 = 106 MB/s

JXTA-J2SE 2.2.1 = 136 MB/s

JXTA-C = 107 MB/s

44

Endpoint service bandwidth results

JXTA-J2SE 2.3 = 112.32 MB/s

JXTA-J2SE 2.2.1 = 144.97 MB/s

JXTA-C = 109 MB/s

45

JXTA-C on top of PadicoTM :-)

JXTA-C is working on top of PadicoTM! (17/11) « sed » modifications in apr

Using marcel threads instead of pthread JXTA-C configured to use GDBM instead of SDBM

Goal: stop using « ethernet emulation » of GM …

But JXTA-C is not yet zero-copy :-(

Performance evaluation in progress …

46

Conclusion

Performance evaluation of the communication layers of JXTA-* (J2SE and C)

Fast-Ethernet Myrinet

Much better results compared to other related work and explained results

Known costs of JXTA JXTA-J2SE nearly saturate a Fast-Ethernet link JXTA-J2SE on Myrinet bandwidth greater than

1Gb/s JXTA-J2SE > JXTA-C on Myrinet

JXTA-C is not zero-copy

47

Ongoing work

Performance evaluation of JXTA-C on top of PadicoTM

JuxMem client in C for DIET Better knowledge of JXTA-C tricks ;-)

Write papers GP2PC (1/12) on JXTA raw performance

Fast-ethernet ? on JXTA tuned performance for grid

computing Myrinet

48

Future work

Benchmark secure and propagate pipes

Update JuxTest results JXTA 2.3.1

Unidirectional throughput test

Plug-in for Ethereal An easier analysis of JXTA protocols