General Performance Tuning Published on 10-01-2010 01:12 AM  Number of Views: 863  

Below are several suggestions for tuning Wowza Server on your production hardware. If you would like more specific instructions for configuring Wowza Server on your specific hardware configuration, send an email with a detailed description of your hardware (# of CPU/cores, memory, operating system, Java version...) to support@wowzamedia.com:1. It is generally best to run the most recent Sun Java JDK (http://java.sun.com). 

Note: If you are on a Mac with OS X 10.5.x or greater (Leopard) with 64-bit hardware you can run the Java 6 VM. To change your default Java VM in OSX, open the Java preferences application /Applications/Utilities/Java/Java Preferences and drag the Java SE 6 item to the top of theJava Applications list. More information here: Changing the default JVM on OSX Leopard

2. It is best to run a 64-bit operating system with the 64-bit Java VM. The 64-bit VM enables Java heap sizes greater than 2GB.

Note: Wowza Media Server 2 will run properly on 64-bit Windows platforms.

3. The most recent development builds can be found here:

http://www.wowzamedia.com/devbuild.html

4. It is best to run the -server version of the Java VM. The instructions for configuring the server version is described in these two forum threads:

Linux and OSX tuning, running the "server" Java VM (tuning)Windows tuning, running the "server" Java VM (tuning)

5. The default Java heap size is set to 768MB which is great for development but not large enough for high volume production use. If you are running the 64-bit version of the Java VM and have 4GB or more of RAM in your machine, we suggest a Java heap size of between 3000MB to 5000MB. If your are running the 32-bit version of the Java VM or have less than 3GB of RAM in your machine, we suggest a Java heap size of 1500MB. Do not attempt to set a heap size greater than 1800MB when using the 32-bit VM. On many Linux platforms the server will start but will fail with anOutOfMemory error after a short period of time. Memory settings can be adjusted in the following script files (based on platform):

1. [install-dir]/bin/setenv.sh (Linux, Mac OSX and Solaris) 

JAVA_OPTS="-Xmx3000M"

2. [install-dir]/bin/setenv.bat (Windows) 

set JAVA_OPTS=-Xmx3000M

3. [install-dir]/bin/WowzaMediaServer-Service.conf (Windows) 

wrapper.java.additional.1=-Xmx3000M

2. Garbage collection (GC) tuning in Java is a tricky subject. What works best in one server setup may not work in another. Through trial and error and customer feedback we have several suggest approaches:

Garbage Collection Options:

1 / 4

We strongly recommend No Addtional Settings. The default -server tuning (see step #4 above) will work quite well for many stream situations. So in most cases this tuning works great without additional settings.

Only if you are experiencing problems with the Java Heap, such as a build up of memory usage, you should try implementing one of the following Garbage Collection options:a. The Concurrent Collector: The concurrent collector is designed for

applications that prefer shorter garbage collection pauses and that can afford to share processor resources with the garbage collector while the application is running. The suggested additional settings are:

Code:

-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

b. Expiremental: Garbage-First garbage collector (G1): In the Sun Java 6 Update 14 or greater release there is a new garbage collector that is in the expirmental stage. It is designed for low pause time high throughput applications such as Wowza Media Server. It is an option to explore. There is more information below. To enable this garbage collector the addtional settings are:

Code:

-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

3.To change the garbage collection settings, modify the following scripts:1. [install-dir]/bin/setenv.sh (add at line 10, Linux, Mac OSX and

Solaris). For example:

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

2. [install-dir]/bin/setenv.bat (add at line 8, Windows). For example:

set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

3. [install-dir]/bin/WowzaMediaServer-Service.conf (Windows). For example:

wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

Note: Set [n] to the next available number in the list of wrapper.java.additional items. So if the last uncommented-outwrapper.java.additional item is 8 then this item needs to be one item below it with an index of 9.

There is more detailed information here:

Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection TuningJava SE 6 Performance White PaperThe G1 Garbage Collector: Coming in Java 6 Update 14

2. For low latency chat applications, it is best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml:

Code:

2 / 4

<ReceiveBufferSize>16000</ReceiveBufferSize><SendBufferSize>16000</SendBufferSize>

3. On Linux, increase the maximum number of file descriptors. This will address the "Too many files open" error message. To do this edit the following two files:

/usr/local/WowzaMediaServer/bin/wms.shCode:

from:#ulimit -n 20000

to:ulimit -n 20000

/usr/local/WowzaMediaServer/bin/startup.shCode:

from:#ulimit -n 20000

to:ulimit -n 20000

Note: This may fail on some versions of Linux or if you do not have permission to change the descriptor limit. If after making this change Wowza Server will not start, consult the documentation for your Linux distribution to see how to increase the descriptor limit.

Note: On some versions of Linux there is also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file:

Code:

fs.file-max=20000

4. On Linux, switch to using the Anticipatory elevator (as) elevator algorithm. It is described here:

Choosing an I/O Scheduler for Red Hat

5. On Linux, mount your disk with the noatime option. This operation will differ based on your Linux distribution. Here is a basic description of the setting:

Using noatime Attribute

6. To tune your server based on the available CPU resources of your server, use the following guidelines (this assumes a single virtual host, if you are running more than one virtual host contact support@wowzamedia.com for more detailed instructions). 

The [total-core-count] refers to the total number of cpu cores in your server. For example if you have dual quad core processors (two quad core processors) the [total-core-count] is:

2x4 = 8

With the number of cores and threads per physical processing growing and growing we suggest a maximum number of threads for each value below:

3 / 4

In the configuration file [install-dir]/conf/VHost.xml:HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24) Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") should not be modified.IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)

NetConnections/ProcessorCount: 2x[total-core-count] (maximum of 24)

RTP/DatagramConfiguration/UnicastIncoming/ProcessorCount: [total-core-count] (maximum of 12)RTP/DatagramConfiguration/UnicastOutgoing/ProcessorCount: 2x[total-core-count] (maximum of 24)

RTP/DatagramConfiguration/MulticastIncoming/ProcessorCount: [total-core-count] (maximum of 12)RTP/DatagramConfiguration/MulticastOutgoing/ProcessorCount: [total-core-count] (maximum of 12)

HandlerThreadPool/PoolSize: (300x[total-core-count])/5 (maximum of 480)TransportThreadPool/PoolSize: (200x[total-core-count])/5 (maximum of 320)

This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above.

7. If you are not doing low latency streaming and you have a client side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the machine and handle more concurrent sessions by editing [install-dir]/conf/VHost.xml and changing the following two values:

Code:

IdleWorkers/CheckFrequency: 100Client/IdleFrequency: 500

8. If you are having problems with multiple incoming multicast streams interfering with each other on Linux you may need to set the Java propertyjava.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13):

Code:

JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

9. This really isn't tuning but it seems like a good place... When streaming to the Adobe Flash player, it is important to try and avoid RTMPT(tunneling version of RTMP) as much as possible. RTMPT uses a polling mechanism which is very chatty and CPU intensive. We suggest employing a protocol rollover strategy so that only connections that need RTMPT use RTMPT. Is is described here:

http://www.wowzamedia.com/forums/content.php?50

4 / 4