Performance Tuning NGINX...Proxy 2 24 2 Intel(R) Xeon(R) Platinum ... Linux Sysctl Settings 26 • Disable TCP timestamps sysctl -w net.ipv4.tcp_timestamps=0 • Defines the local

Performance Tuning NGINX

Name: Amir Rawdat

Currently: Technical Marketing Engineer at NGINX inc.

Previously: - Customer Applications Engineer at Nokia inc.

Multi-Process Architecture with QPI Bus

Web Server Topology

wrk nginx

Reverse Proxy Topology

�6

wrk nginx nginx

Technical Specifications

# Sockets # Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

Client 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

128 GB Ubuntu Xenial

40GbE QSFP+

Web Server & Reverse Proxy

2 24 2 Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz


40GbE QSFP+

Multi-Processor Architecture

Duplicate NGINX Configurations

#1

�9

Multi-Processor Architecture

NGINX Configuration (Instance 1)

�11

user root; worker_processes 48 ; worker_cpu_affinity auto

000000000000000000000000111111111111111111111111000000000000000000000000111111111111111111111111; worker_rlimit_nofile 1024000; error_log /home/ubuntu/access.error error;

….. …….

NGINX Configuration (Instance 2)

�12


111111111111111111111111000000000000000000000000111111111111111111111111000000000000000000000000; worker_rlimit_nofile 1024000; error_log /home/ubuntu/access.error error;

……. …….

Deploying NGINX Instances

�13

$ nginx –c /path/to/configuration/instance-1 $ nginx –c /path/to/configuration/instance-2 $ ps aux | grep nginx

nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_0.conf nginx: worker process nginx: worker process

nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_1.conf nginx: worker process nginx: worker process

$ pkill nginx

Additional NGINX Configuration Directives

#2

�14

Web Server (Instance 1)

events { worker_connections 1000000; } http { access_log off; keepalive_timeout 315; keepalive_requests 10000000; sendfile on; tcp_nopush on; tcp_nodelay on;

server { listen 10.10.16.10:443 backlog=250000 reuseport; root /usr/share/nginx/bin; } }


�16



Reverse Proxy (Instance 1)

�17

server { listen 10.10.10.18:443 ssl backlog=102400 reuseport; ssl_certificate /etc/ssl/certs/nginx.pem; ssl_certificate_key /etc/ssl/private/nginx.key; ssl_session_cache off; ssl_session_tickets off; location / { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass http://webserver_0; } } upstream webserver_0 { server 10.10.10.11:80; keepalive 200; } }

Reverse Proxy (Instance 2)

�18


Performance Test Results

�19





�21


server { listen 10.10.11.23:443 backlog=250000 reuseport; root /usr/share/nginx/bin2; } }


�22


�23

Performance Tip: Linux Sysctl Settings

#3

�24

Linux Sysctl Settings

�25

• Increase memory thresholds to prevent packet dropping ◦ sysctl -w net.ipv4.tcp_rmem=”4096 87380 4194304” ◦ sysctl -w net.ipv4.tcp_wmem=”4096 65536 4194304”

• Increase the size of the processor queues ◦ sysctl -w net.core.net_dev_max_backlog=250000

• Setting the maximum TCP buffer sizes ◦ sysctl -w net.core.rmem_max=4194304 ◦ sysctl -w net.core.wmem_max=4194304


�26

• Disable TCP timestamps ◦ sysctl -w net.ipv4.tcp_timestamps=0

• Defines the local port range that is used by TCP and UDP to choose the local port ◦ sysctl -w net.ipv4.ip_local_port_range = 32768 60999

• Enable reuse of TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. ◦ sysctl -w net.ipv4.tcp_tw_reuse = 1


�27

Performance Tip: Enabling RSS and TPS

4

�28

Enabling RSS and TPS

�29

• /etc/init.d/irq_balance stop • git clone https://github.com/ANLAB-KAIST/mlnx-en.git • cd /mlnx-en/ofed-scripts • ./set_irq_affinity_bynode.sh <numa node id> <interface-name> • set_irq_affinity -x local <interface-name>

https://github.com/ANLAB-KAIST/mlnx-en.git


�30


�31


�32

Performance Numbers with NGINX and Intel QuickAssist

�33

Performance with QuickAssist

Performance with QuickAssist

�35

Summary

�36

• Deploy two nginx instances • Using additional nginx configuration directives • Linux sysctl parameter tuning • Setting IRQ affinity using RSS and TPS • Consult appendix for additional information and

performance tips

Contact information here

Thank you

[email protected]

Appendix7

Client Traffic Script

taskset -c 0-21,44-65 wrk -t 44 -c 1000 -d 180s -H 'Connection: Close' https://10.10.16.10:443/$1 >> output.txt &

taskset -c 22-43,66-87 wrk -t 44 -c 1000 -d 180s -H 'Connection: Close' https://10.10.11.23:443/$1 >> output1.txt &

## $1 is the requested static file size

Architectural Specifications

Architectural Specifications

�41


# Sockets

# Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

nbdw32 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz


40GbE QSFP+

Web Server



40GbE QSFP+


# Sockets # Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

Client 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz


40GbE QSFP+

Reverse Proxy



40GbE QSFP+

Web Server 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz


40GbE QSFP+

Reverse proxy (Instance 1)

�44


000000000000000000000000111111111111111111111111000000000000000000000000111111111111111111111111; worker_rlimit_nofile 1024000; error_log /home/ubuntu/access.error crit; events { worker_connections 1000000; }

http { access_log off; keepalive_timeout 315; keepalive_requests 10000000; sendfile on; tcp_nopush on; tcp_nodelay on;


�45



�46


111111111111111111111111000000000000000000000000111111111111111111111111000000000000000000000000; worker_rlimit_nofile 1024000; error_log /home/ubuntu/access.error crit; events { worker_connections 1000000; }

http { access_log off; keepalive_timeout 315; keepalive_requests 10000000; sendfile on; tcp_nopush on; tcp_nodelay on;


�47


Documents

Performance Tuning NGINX...Proxy 2 24 2 Intel(R) Xeon(R) Platinum ... Linux Sysctl Settings 26 • Disable TCP timestamps sysctl -w net.ipv4.tcp_timestamps=0 • Defines the local