Top 10 Ways to Optimize Network Performance

Carrie Higbie, Siemon

Global Network Applications Market Manager

Ask the Expert , TechTarget – SearchNetworking, SearchEnterprise Voice, SearchDataCenter

President, BladeSystems Alliance

Top 10 ways to optimize performance

• Understand your performance

• What is harming optimal performance?

• Physical layer

• Routing and switching

• Applications

• Environmental concerns

• How to fix performance

• What does poor performance cost?

1. Bandwidth bandits – Find ‘em, slay ‘em!

• Begin with auto-discovery

• Start at the physical layer

• Common to all systems

• Can cause the most problems

• According to LAN Magazine – 70% of all downtime!

• Can be 80% of all troubleshooting costs

• Only 20% of troubleshooting time is spent fixing the problem

• Documentation of the physical layer may be required for compliance

• Absolute necessity for security reasons

Top offenders

• Improperly terminated cables• Improperly terminated patch cords• Lengths exceeded specified maximum• Cabling was improperly or not labeled (troubleshooting

problems)• Electronics and closets in bad locations (humidity, EM, RF)• Cables bunched to tightly causing the pairs to be flattened• Cables tied to electrical conduits or run too close to power panels• Cabling that did not pass testing due to various issues• Closet spaghetti• CAT 3 cables terminated to CAT 5 – 100M switches• Bends exceeding bend radius• Cables not to spec• Racks not grounded

Active electronics and networking layers

• Flat networks with large address ranges

• Subnets are helpful

• Poor VLAN management

• Intruders and security breaches

• Build a historic reference

• Duplex mismatches

• Ports forced to low speeds

• Half-duplex connections

• May be electronics; may be cabling!

2. Building in resiliency

• Load balancing and load sharing will help performance

• Proper subnets and/or VLANs can help

• If you build a monster, it will be a monster!

• Use the electronics to your advantage

• Price/performance ratios are key!

• Interoperability must be tested, not just stated

• Failover must be tested

3. Pay attention to applications

Application sizes double every 18 months

• More capability and application bloat

• Process increasingly larger file sizes

• Shared applications and server-side processing can enhance or deplete resources

• QoS attributes can be set in applications

• Don’t let them step on each other

• Increased demands on storage, backups and network resources

With growth comes problems

• According to Nucleus Research (2004)• Average storage budget = $2.3 Million• Average storage capacity = 115 TB• Troubleshooting = Average 14 hours per month

• Among companies with budgets > $1B• 85% report service degradation• 51% said poor application performance• 82% said problems impact employee productivity• 79% said customer service quality suffers!

• Study from Network World (2003)

• According to Gartner, downtime will increase 200% in Q1 2005

• 20% of all IT expenditures are for things that DON’T work

Sample views - SNMP

Node Index Mtu Speed In Octets In Errors

192.168.1.91 23 1514 200000000 3,609,426,192 6,975,445

192.168.1.9 11 1514 10000000 3,116,050,917 5,761,862

192.168.2.11 7 1514 200000000 2,841,840,312 1,507,894

192.168.1.92 6 1514 200000000 619,369,476 2,302,031

192.168.2.11 5 1514 200000000 2,099,071,371 4,227,351

192.168.1.94 22 1514 200000000 1,274,947,502 3,155,459

192.168.1.104 1 1514 200000000 2,490,061,169 5,247,177

192.168.1.97 2 1514 200000000 742,847,434 3,437,434

192.168.1.14 3 1514 200000000 162,306,489 8,258,906

192.168.2.11 4 1514 200000000 1,130,881,154 7,166,234

192.168.1.124 6 1514 200000000 54,190,705 11,145,670

192.168.1.114 15 1514 200000000 445,094,503 1,315,031

192.168.1.105 18 1514 200000000 152,346,003 6,514,778

192.168.2.11 16 1514 200000000 1,077,239,333 1,913,864

192.168.2.11 2 1514 200000000 3,455,568,628 14,413,615

192.168.1.108 21 1514 200000000 266,461,364 1,789,253

192.168.1.102 9 1514 200000000 50,564,444 3,115,889

192.168.1.101 10 1514 200000000 53,670,114 1,112,455

192.168.2.11 15 1514 10000000 159,618,694 1,109,967

Actual audit data

Note: Proprietary And confidential information

removed

4. Examine utilization

• Snapshots do little good

• Must be done over time

• Sample period should include all normal business functions

• Closing out accounting

• End-of-month processes

• Payroll runs

• High-traffic periods such as high customer service demands

• Forget averages – watch peak periods!

What is utilization really? What do I look for?

• Make sure that periods viewed are consistent with hours worked

• Averages do you little good

• For real-time applications always plan on highest utilization numbers

• Includes VoIP/IPT, video, etc.

• Group utilization needs by class of user, not department

• Environmental conditions• Temperature and humidity variations

• EM and RF interference

• High network traffic

• Outdated, slow PCs and NICs

• Poor installation• Inferior patch cords

• Damaged cable due to pulling, bending

• Too many splices

• Poor cable management

• Inferior network cabling• ACCORDING TO ESTIMATES GIVEN TO IEEE, OVER 50% OF ALL

CAT 5E WON’T PASS 5E TESTING!

What causes slow response

The cost of a slow network

Examples:

Company A:Number of Employees: 500

Average Hourly Wage: $15.00

Hours of Productivity Lost per Year: 30

Network Slow Cost = $225,000.00

Company B:Number of Employees: 1,000

Average Hourly Wage: $18.00

Hours of Productivity Lost per Year: 52

Network Slow Cost = $936,000.00

Company C:Number of Employees: 5,000

Average Hourly Wage: $20.00

Hours of Productivity Lost per Year: 20

Network Slow Cost = $2,000,000.00

Calculate network slow cost:

Cost = P x W x E

P = Total Number of hours lost Productivity per year (weekly minutes/60 x 52)

W = Average hourly Wage E = Number of Employees

on the network

Formulas

• Revenue per hour

• Total revenue / 2080 hour work week

• Revenue per employee per hour

• Total revenue / Number of employees / 2080

• Salary expense per hour (weighted)

• Average hourly wage * 1.4 (to include overhead) / 2080

• Salary expense plus lost revenue

• Total revenue per hour + weighted salary expense per hour * % of workforce down at any given time (we used 15%)

5. Performance optimization: Tricks of the trade

• Audit your infrastructure – cabling, electronics, etc.

• Use a Certified Infrastructure Auditor

• Trained to understand relationships between electronics and physical layer

• Omitting either one is only half an audit

• Audit your ports, not the entire switch

• Use RMON to help determine bottlenecks

Adding new applications

• BEWARE – minimums are dangerous!

• Utilize a test bed to help understand needs

• Multiply all results by the number of end users

• Assume concurrent operations for all on the same shifts

• Don’t forget additional loads for replication, redundancy and backups

• There is a fine line between not enough and too much

• Aim for a little too much

• Don’t forget to account for growth!

Products to help

• Bandwidth managers

• Layer 7 products• Application classification

• Routing based on need

• Dynamic bandwidth allocation

• Don’t stop at Layer 3

Courtesy of Packeteer

6. Trends: Know your bandwidth needs

Courtesy of Packeteer

Determine your need for speed

• 10G products are currently available, with new copper-based products expected this year

• Your network may be a combination of speeds to the desktop

• If a user can’t fill an entire gig channel but you have discards at 100 Mbps, gig should be used

• Same applies to 10G

• Watch power users

• CAD/CAM/CAE

• Imaging and graphics

• Video

7. Predict the future – The crystal ball

• Examine your past 5 years if possible

• Application changes

• Speed changes

• Hardware upgrades

• Server upgrades

• Memory upgrades

• OS upgrades

Trends for ‘05

• Gates’ Law revisited

• “640k ought to be enough for anybody”

-- Bill Gates, 1981

Application growth/bloat

Office 95 Office XP Windows 95 Windows XP

386 – 486 recommended

PIII recommended

133MHz min

486 recommended

233MHz min

300MHz rec.

8 MB RAM 128 MB RAM

+ 8MB for each program

4 MB RAM, 8 recommended

128 MB RAM+ recommended

64MB min

40 MB min disk space

128 all features

245MB disk space +

115MB +50 MB

50-55 MB 1.5 GB

Disk space

8. Know your security challenges

• #$%^&* hackers!

• Spyware and malware

• Breaches

• Reporting structure

• Weaknesses

• Logging

• Compliance

IT management and security management

• Shift in duties

• Other spending is often derailed in lieu of security expenditures

• Consumes many resources

• Patch management adds overhead

• WAN links can be hindered or halted

• Utilize the best tools

• ROI based on cost avoidance

• Beware of target size!

9. Revisit and revise

• Any and all measures mentioned

• Quarterly health checks a must

• Downtime is not the only factor

• Slow performance is also very costly

• Put in the best if you expect the best

• The level of support you can give is equal to the level of support you can receive

10. Evaluate products: What are the extras in a top-of-the-line system?

• R&D

• Standards participation

• Support on a global scale

• Tried-and-true support

• Product expediting

• Value-added services

• Warranty

• Partner as opposed to seller

Evaluate your vendors properly

• Check non-vendor references

• Pose some questions and validate the answers

• Training and transfer of knowledge

• Quality of training

• Certification levels

• Training for their resellers

• Back-end support – knowledge bases

• Pricing is only one factor and should never be the deciding factor

• MTBF/MTTR – every single component!

• Upgrade paths and future capabilities

Have a bake-off

• Invite your potential vendors in for testing

• If they state it will work – make them show you

• Verify fail-over/redundancy/resiliency

• Look at the management tools

• Play with the technology

• If you find a weakness in one product, see how/if it is addressed in another

MOST IMPORTANT!

• Understand what you are getting for your money

• Front end

• Back end

• In between

• Understand market share and marketing information

• Understand where the standards are going. Today’s investment could kill support tomorrow!

Thank you

Carrie Higbie, Siemon

Global Network Applications Market Manager

Ask the Expert , TechTarget – SearchNetworking, SearchEnterprise Voice, SearchDataCenter

President, BladeSystems Alliance