May 2013

May 2013

SUM410Getting the Best Performance with Citrix NetScaler

Edward Targonski

© 2013 Citrix

Agenda

• Netscaler Model and Network Deployment Options• Performance Enhancing Features• Commonly Used Troubleshooting Tools and Commands

• Questions?• Conclusion

Netscaler Models

© 2013 Citrix

NetScaler VPX

NetScaler Models

NetScaler MPX

NetScaler SDX

© 2013 Citrix

Differences Between MPX and VPX

• Three main differences exist between MPX and VPX:ᵒ System capacityᵒ Performanceᵒ Tagged VLAN Configuration

• NetScaler VPX system capacity:ᵒ No hardware SSL accelerationᵒ Processing not offloaded to dedicated silicon

© 2013 Citrix

When to Use Which?

NetScaler Appliances NetScaler VPX

• Gig+ performance

• High volume SSL Offload

• >100 SSL VPN CCUs

• FIPS requirements

• Physical device security

• Labs/test environments

• Development environments

• “Datacenter-in-a-box”

• CPU-intensive workloads

• Frequently moved apps

• Fast/remote deployment

© 2013 Citrix

NetScaler SDX

• Instances, not partitions

• Complete CPU isolation

• Complete memory isolation

• Version independence

• High availability independence

• Lifecycle independence

© 2013 Citrix

Network TopologiesOne-Armed

If you are able to, one-armed topologies are the preferred method of deploying NetScaler in most environments.

© 2013 Citrix

Network TopologiesTwo-Armed

1. User Request

3. Response4. Response

2. User Request

Public/Front VLAN Private/Server

VLAN

The most common implementation of two-armed topologies are when a NetScaler is replacing another legacy two-armed device in a network

Performance Enhancing Features and Settings

© 2013 Citrix

TCP Connection without NetScaler

Server sees eleven packets

Client ServerSYN

ACK

SYN+ACK

GET

FIN

ACK

ACK

Data

DataData

FIN

Server de-allocates storage for the connection

Server allocates storage for connection

© 2013 Citrix

Transaction with NetScaler

Server sees

four packets

Client ServerNetScalerSYN

ACK

SYN+ACK

GET

FINACK

ACK

Data

DataData

GET

Data

DataData

FIN

Global Performance Settings

© 2013 Citrix

Global Settings

•Surge Protection

•Path MTU discovery

© 2013 Citrix

HTTP Parameters

• Client IP Insertion• Cookie Version • Requests/Responses:

ᵒ Drop invalid HTTP requestsᵒ Mark CONNECT request as invalidᵒ Mark HTTP/0.9 request as invalidᵒ Log HTTP error responses

• Server Header Insertion

© 2013 Citrix

TCP Parameters

• Window Scaling

• Selective Acknowledgments

• Nagle’s Algorithm

• SYN Attack Detection

© 2013 Citrix Citrix Confidential - Do Not Distribute

Performance Enhancing Features


• Reduce Server Load

• Higher TPS

• Central Certificate Management

• Central Cipher Management

Performance Enhancing Features – SSL Offload

© 2013 Citrix

• In end-to-end, use low-level ciphers in NS-to-service communication

• Cipher selection depends on client-needs, and security considerations.

• Can be combined with IC and Compression for maximum impact

Citrix Confidential - Do Not Distribute

Advanced Optimization: SSL Offload


• Faster response

• Fewer bytes on-wire

• Better response for low-bandwidth clients

• Policy-based rules

Performance Enhancing Features – Compression

© 2013 Citrix

Compression

• NetScaler supports various ways of compressing traffic

• HTTP traffic can easily be compressed by NetScalerᵒ Less work for the web serverᵒ Client can understand and de-compress (accept-encoding header)

• Compression governed via policies

• Preconfigured policies exist


• Reduce server load

• Faster response

• Policy-based controls

Performance Enhancing Features – Caching

© 2013 Citrix

• Use Content-Group settings to optimizefor min/max content size, or overallnumber of hits.

• Use parameterization to optimize cache retrieval or invalidation.

• Prioritize NO_CACHE policies before CACHE policies

• Use multiple Content-Groups to allow for specific cache-clearing


Advanced Optimization: Caching


• Reduce server load

• Faster server response

• Full Traffic Optimization and Traffic Security Feature Sets

Performance Enhancing Features – TCP Session Mangement

Results of Performance Enhancing Feature Configuration

© 2013 Citrix

“Sharepoint” SSL+HTTP Load Balancing ConfigurationStandard HTTP Load Balancing


SSL Handling on Servers

Doc. Size Baseline

987 kB .doc 16.34s

5.29 MB .doc 89.86s

1.75 MB .pdf 28.62s

5.10 MB .pdf 80.28s

Source: Citrix Application Optimization for MOSS 2007 Performance Assessment - http://support.citrix.com/article/ctx120235

*Times based on 1.5mbps connection with 0.7%packet loss.

http://support.citrix.com/article/ctx120235

© 2013 Citrix

SSL-Offload + Compression Load Balancing ConfigurationSSL-Offloaded HTTP Load Balancing


SSL Handling on NetScalerStatic/Dynamic content

compressed

Doc. Size BaselineSSL Offload

+ Compress

987 kB .doc 16.34s 12.29s

5.29 MB .doc 89.86s 56.20s

1.75 MB .pdf 28.62s 18.87s

5.10 MB .pdf 80.28s 70.36s

Servers configured as plaintext HTTP



© 2013 Citrix

SSL offload + Compression + Integrated CachingLoad Balancing ConfigurationSSL-Offload + Cmp +Caching HTTP Load Balancing


Doc. Size BaselineSSL Offload

+ CompressCaching

987 kB .doc 16.34s 12.29s 8.62s

5.29 MB .doc 89.86s 56.20s 42.78s

1.75 MB .pdf 28.62s 18.87s 14.51s

5.10 MB .pdf 80.28s 70.36s 60s

SSL Handling on NetScaler + Compression with Integrated

Caching

*Cache object max. limit set to 10MB



Troubleshooting Tools and Commands

© 2013 Citrix

•Primary tool for detailed analysis

•NetScaler logs all statistics every 7 seconds

•Uses logs from /var/nslog

•Logfiles are gzipped (use zcat)

•Some stats now available via GUI(System > Diagnostics)


NSCONMSG

© 2013 Citrix

Scenario: Testing reports problems with SSL VIP earlier. What happened?


NSCONMSG – Examples

nsconmsg –K newnslog –g ssl_err –d stats

Displaying current counter value informationNetScaler V20 Performance DataNetScaler NS9.3: Build 57.53.nc, Date: Jul 20 2012, 07:26:39

reltime:mili second between two records Fri Feb 5 10:31:31 2010Index reltime counter-value symbol-name&device-no 0 0 0 ssl_err_ssl3_badversion 1 0 0 ssl_err_cavium_random_seed_failed 2 0 0 ssl_err_ubsec_card_reset 3 0 0 ssl_err_ssl3_send_server_hello 4 0 0 ssl_err_ssl3_send_server_certificate 5 0 0 ssl_err_ssl3_send_server_key_exchange 6 0 0 ssl_err_ssl3_send_certificate_request 7 0 0 ssl_err_ssl3_send_server_done

Current logfile

Grep for ‘ssl_err’

View initial statistics

© 2013 Citrix

Scenario: Testing reports problems with SSL VIP earlier. What happened?



nsconmsg –K newnslog –s disptime=1 –g ssl_err_ssl3 –d current

Index rtime totalcount-val delta rate/sec symbol-name&device-no&time 108 0 78 1 0 ssl_err_ssl3_get_client_hello Fri Feb 5 12:01:06 2010 109 14000 11 2 0 ssl_error_cvm_bad_record Fri Feb 5 12:01:20 2010 110 7000 79 1 0 ssl_err_ssl3_badversion Fri Feb 5 12:01:27 2010 111 0 79 1 0 ssl_err_ssl3_get_client_hello Fri Feb 5 12:01:27 2010 112 28000 81 2 0 ssl_err_ssl3_badversion Fri Feb 5 12:01:55 2010 113 0 81 2 0 ssl_err_ssl3_get_client_hello Fri Feb 5 12:01:55 2010 114 7000 83 2 0 ssl_err_ssl3_badversion Fri Feb 5 12:02:02 2010

View historic statistics

View timestamps

© 2013 Citrix

Scenario: Testing reports problems with SSL VIP earlier. What happened?NSCONMSG – Examples

nsconmsg –K newnslog -s csv=1 –g ssl_err_ssl3_badversion –d current > sslv3.csv

Grep specific counter

Output to csv

Write to file

© 2013 Citrix

Checking for distribution and performance



nsconmsg –K newnslog –s ConLb=3 –d distrconmsg

VIP(1.1.1.1:636:UP:WEIGHTEDRR): Hits(2506) Pers(OFF) PersHits(0:0%) Err(0:0%) Ovrride(0:0%)S(1.1.1.100:636:UP) Hits(835:33%) PHits(0:0%) LbHits(835:100%)S(1.1.1.101:636:UP) Hits(836:33%) PHits(0:0%) LbHits(836:100%)S(1.1.1.102:636:UP) Hits(835:33%) PHits(0:0%) LbHits(835:100%)VIP(2.2.2.2:389:UP:WEIGHTEDRR): Hits(6) Pers(OFF) PersHits(0:0%) Err(0:0%) Ovrride(0:0%)S(2.2.2.100:389:UP) Hits(2:33%) PHits(0:0%) LbHits(2:100%)S(2.2.2.101:389:UP) Hits(2:33%) PHits(0:0%) LbHits(2:100%)S(2.2.2.102:389:UP) Hits(2:33%) PHits(0:0%) LbHits(2:100%)VIP(3.3.3.3:123:UP:WEIGHTEDRR): Hits(180) Pers(SOURCEIP) PersHits(180:100%) Err(0:0%) Ovrride(0:0%)S(3.3.3.100:123:UP) Hits(42:23%) PHits(42:100%) LbHits(0:0%)S(3.3.3.101:123:UP) Hits(49:27%) PHits(49:100%) LbHits(0:0%)S(3.3.3.102:123:UP) Hits(46:25%) PHits(46:100%) LbHits(0:0%)S(3.3.3.103:123:UP) Hits(43:23%) PHits(43:100%) LbHits(0:0%)

© 2013 Citrix




nsconmsg –K newnslog –s ConLb=3 –d oldconmsg

current time is Thu Apr 8 14:45:28 2010-------------------------------------------------------NATSession : Free(19644)A(21845)InUse(2201)NATSession: Cur(Tcp[194] Udp[2007] Icmp[0] Other[0])NATSession: Op/s(Tcp[3] Udp[436] Icmp[1] Other[0])Session: A:9187 F:4604 IUse:4583 SEs: SIP:4582 C:0 SSL:0 Svr:1 UserId:0 SIPDIP:0 DIP:0 SO:0SSF: Conn (Srvr 0 Clnt 1) U:0CM: Conn (Srvr 0 Clnt 1) Sessions PCB 0 NATPCB 0Z(SIP[68307], C[0], SSL[0] Server[22] SIPDIP[0] DIP[0] SO[0])Mon: Probes: 24303862, Failed: 3757181

© 2013 Citrix




nsconmsg –K newnslog –s Con???=3 –d oldconmsg

ConDebug - DebuggingConLb - Load BalancingConMon - Monitoring ProbesConMEM - Memory ManagementConCSW - Content SwitchingConSSL - SSL OffloadConCMP - CompressionConIC - Integrated Caching

© 2013 Citrix

• Nstrace supports filtering beginning in 9.x


nstrace.sh


nstrace -size 0 -filter "SOURCEIP == 10.1.2.3 && SOURCEPORT == 8080" -link ENABLE

Packet-size limit Filters in standard NS policy format

Automatically capture linkedclient/server connections

Filter on: SOURCEIPSOURCEPORTDESTIPDESTPORTSVCNAMEVSVRNAMESTATE

Booleans supported!


© 2013 Citrix

• nstrace files now officially supported in Wireshark!

• Available in latest Stable release

• Includes ns.pdevno and ns.l_pdevno filtering


Wireshark

Citrix AutoSupport Introduction

© 2013 Citrix

Citrix AutoSupport Analysis

© 2013 Citrix

Graph Generated by AutoSupport Tools

Resources

© 2013 Citrix

Resources

• Netscaler HTTP Profiles

• Netscaler TCP Profiles

• Tune NetScaler TCP Stack

• Netscaler Advanced SSL Settings

• Nsconmsg to Excel Tool

• Netscaler SSL Offload

© 2013 Citrix

Resource – 2

• Netscaler Integrated Caching

• Netscaler Compression

• Netscaler CPU Profiling

• Citrix AutoSupport (TaaS)

• Netscaler Datasheet - Models and Specs

• Citrix Application Optimization for MOSS 2007 Performance Assessment

© 2013 Citrix

Conclusion

© 2013 Citrix

Question

© 2013 Citrix

Before you leave…

52

•Conference surveys are available online at www.citrixsynergy.com starting Friday, May 24 at 9:00 a.m. PT

ᵒ Provide your feedback by 4:00 p.m. PT that day and you’ll receive a $30 Amazon.com gift card via email

•Download presentations starting Monday, June 3, from your My Conference Planning tool located within the My Account section

Work better. Live better.

Documents

May 2013