Data Center Testing - BICSI · Data Center Testing: The Hidden Impairment David SchellDavid Schell Principal Member of the Technical Staff Fluke NetworksFluke Networks July 2011 –

Data Center Testing:The Hidden Impairment

David SchellDavid SchellPrincipal Member of the Technical Staff

Fluke NetworksFluke NetworksJuly 2011 – Baltimore Technical Education Summit

Introduction• Datacenter owners and operators have long used

fiber for storage networks and campus backbonesfiber for storage networks and campus backbones. • The constant thirst for bandwidth has increased the

use of fiber and driven a shift towards 10Gigabit anduse of fiber and driven a shift towards 10Gigabit and faster fiber technology.

• Preterminated factory tested “plug-and-play” fiberPreterminated, factory tested plug and play fiber systems have enabled much of the shift.

• The fiber’s full capacity may not be achieved unlessThe fiber s full capacity may not be achieved unless pre-emptive assurances are incorporated into your network’s design and installation criteria.

MPO/MTP Connectors• Released in the late 1990’s.

l fi d f 2 fib• Commonly configured as 1 row of 12 fibers.• Factory terminated and tested.• Data center installations make use of preterminated

trunks viableR k/ bi t l ti k h d f ti– Rack/cabinet locations are known ahead of time

• Installation time is reduced by up to 96% compared with field terminating a 960 port installation *with field-terminating a 960-port installation. *

* Congdon II, Herbert V. http://www.cablingtrends.com/news-a-issues/industry-innovations/105-with-high-density-connectivity-the-data-centre-of-the-future-is-here-now.html

Pre-terminated Systems Dominatey

Connectivity method Connectivity methodyfor datacenters:50 respondents

yfor LAN:

46 respondents

Pre-polished Mech Conn

10%

Other8%

Puck & Polish14%

Pre-polishedMech Conn

11%

Other4%

Puck & Polish12%

Pre-term Pre-term cables/pigtails

68%cables/pigtails

63%

4

Data from The Fiber Optics LAN Section (FOLS) of TIA Webinar Surveys

www.fols.org

MPO and LC are Everywhere

Connector types Connector typesypin datacenters:49 respondents

in the LAN: 48 respondents

FC0%

ST4%

MPO/MTP28%

FC2%

ST2%

MPO10%

LC52% SC

16%

LC67%

SC19%

5

Data from The Fiber Optics LAN Section (FOLS) of TIA Webinar Surveys

www.fols.org

What is the Correct Test Method?• Don’t test: It’s Plug-and Play?

• Just verify continuity with a VFL?

• Just clean and inspect?

• Full loss certification?

• Other?

Plug-and-Play is Plug-and-Prayg y g y• It’s factory tested, why would I need to field test?

Th f i t ll ti i t d i k t bl d– The process of installation introduces risks to cable and connectors

– Warranties usually require testingy q g– Excessive bending losses that impede performance– Connector contamination or even damage– Hope is not a valid strategy

Verification with a VFL• It’s factory tested; don’t I just need continuity?

Q lit t b d t i d ith VFL– Quality cannot be determined with a VFL

12 dB Loss! 0.5 dB Loss!

Inspect and Cleanp• It’s factory tested; do I really need to inspect?

S!– YES!Contamination can be collected from dust caps, fingers, air particulates etcair particulates, etc.

Dirt could move to core Will damage other connectors

Inspect and Clean (MPO)• Guard against redistribution of contamination

B i i d l i i h

p ( )

• But inspection and cleaning is not enough

Full Certification is Requiredq• Standards require it

ANSI/BICSI 002 2011 Data Center Design and Implementation– ANSI/BICSI 002-2011 Data Center Design and Implementation Best Practices

– ANSI/TIA-568-C Generic Telecommunications Cabling for Customer Premises

– ANSI/TIA-526-14-B Optical Power Loss Measurements of Installed Multimode Fiber Cable Plant

– ISO/IEC 11801 Generic cabling for customer premises– ISO/IEC 14763-3 Implementation and operation of customer

premises cabling Part 3: Testing of optical fiber cablingpremises cabling -- Part 3: Testing of optical fiber cabling

• You should want it– Rock-solid field testing with proof of workmanship is the firstRock solid field testing with proof of workmanship is the first

line of defense against claims of non-performance.

Confusion with fiber standards test limitstest limits

• Test requirements and limits:

ANSI/BICSI002

ANSI/TIA568-C

ISO/IEC11801

• Test methods

ANSI/TIA526-14-A

ISO/IEC14763-3

ANSI/TIA526-14-B

Limits for ISO/BICSI/TIA

• First and last connectors

BICSI ≤ 0.1 dBISO ≤ 0.1 dBTIA ≤ 0.1 dB


Test Reference CordMultimode Connector

InstalledMultimode Connector

TIA ≤ 0.1 dB TIA ≤ 0.75 dB

Permitted Mated Loss BICSI ≤ 0.30 dBBICSI ≤ 0.30 dBISO ≤ 0.30 dBTIA ≤ 0.75 dB

Limits for ISO/BICSI/TIA

• Random mated pair (within the channel)





TIA ≤ 0.75 dB TIA ≤ 0.75 dB

Permitted Mated Loss BICSI ≤ 0.50 dBBICSI ≤ 0.50 dBISO ≤ 0.75 dBTIA ≤ 0.75 dB

ISO/IEC 11801:2010 Test Limit• Example calculation of loss budget

850 nm:

AdaptersSplices = 0 * 0 3 dB

0.60 dB0 00 dB

= 2 * 0.30 dBp

Fiber= 0 * 0.3 dB= 0.1 km * 3.5 dB

0.00 dB0.35 dB

Allowable loss = 0.95 dBAllowable loss 0.95 dB

Custom Test Limit• You may be required to test to a limit tighter than the

values given by the standardsvalues given by the standards• For example:

MTP Cassette loss (one adapter) ≤ 0 50 dB– MTP Cassette loss (one adapter) ≤ 0.50 dB– Fiber loss @ 850 nm ≤ 2.3 dB per km– Fiber loss @ 1300 nm ≤ 0.6 dB per kmFiber loss @ 1300 nm ≤ 0.6 dB per km

• A user defined custom limit is required for automatic PASS/FAIL analysis/ y

Duplex Testing ofMPO/MTP CassettesMPO/MTP Cassettes

Set the Reference• One jumper from Source to Meter (no intermediate

adapter)adapter)– Duplex testers have two sources and two meters– Meter port must match the cabling plant’s connector typeMeter port must match the cabling plant s connector type

Set the Reference• Disconnect each jumper from the meter but not

from the sourcefrom the source– The meter has a large area detector so that measured

power is not affected by moving the jumperp y g j p

Set the Reference• Connect jumpers to both meters

Verify the Test Jumpersy p• Connect the test jumpers together with one adapter

f l• Perform a loss test• Test jumpers are good if loss is <0.1 dB

Prepare to Testp• Disconnect the test jumpers: protect the connector end faces• Go to opposite ends of the link under testGo to opposite ends of the link under test• Inspect and clean the jumpers (dust caps often contaminate

connectors)

Test the Link• Perform the loss test as you would on any other

duplex linkduplex link.• How many adapters are in this test?

How Many Adapters?y p• Are there four adapters?

≤0.75 dB ≤0.75 dB

300 m

≤0.75 dB ≤0.75 dB

≤ 1.05 dB

10GBASES-SR requires <2.6 dB @ 850 nm

The loss here would be 4.05 dB; not good enough

How Many Adapters?y p• The cassettes need to be treated as a single adapter

≤0 75 dB 300 m ≤0 75 dB≤0.75 dB≤ 1.05 dB

≤0.75 dB

10GBASES-SR requires <2.6 dB @ 850 nm

Th l h ld b 2 55 dB thi i dThe loss here would be 2.55 dB; this is good

MPO/MTP Cassette Specificationsp

MPO/MTP Cassette Specificationsp

THE HIDDEN IMPAIRMENTTHE HIDDEN IMPAIRMENT

What is reflectance?• When light moves from a medium of a given refractive index n1 into a second

medium with refractive index n2, both reflection and refraction of the light may occur

• These Fresnel reflections are what you see when looking at a window– Caused by the refractive index difference

between air and glass– If not too bad, you can still see

through the glassthrough the glass• An air gap between the end faces of a fiber

also cause Fresnel reflections to occur

Why should you care?y y• Reflections cause increased Bit Error Rates on the network

– Laser light reflected back into the transmitter increases the Relative Intensity Noise (RIN) of the laser.

– If the reflection is caused by contamination that results in Mode Selective Losses (e.g. caused by dirt) then Modal Noise is increased.

– Noise on the network increases Bit Error Rates (adversely affecting the user’s experience)

What do those numbers mean?• Reflectance is the preferred term when characterizing a single connector.

– It is a measure of the amount of power reflected by a connection.p y– It includes one connector– It is always negative.– Smaller is better (e.g. -35dB is better than -20dB)

Refl =10logPreflected

Pincident

• Return Loss is the preferred term when characterizing an entire link– It is a measure of the amount of power NOT reflected by a link.– Includes all connections and fiberIncludes all connections and fiber– It is always positive.– Bigger is better (e.g. +35dB is better than +20dB)

P

=

reflected

incident

PPlog10ORL

Reflectance in connectors• If light sees a change in refractive index, there will be a reflection.• The most common causes are:

– Air gap between the connectors– Dirt/contamination– Residue left behind by the cleaning solution

f ld h ld b b h d b• In a perfect world, there would be no air gap between the mated connectors but in reality, there is always a small air gap, also known an “undercut”:

• The very best factory terminated connectors will have an undercut better than 50 nm (that’s 0.05 µm).

• The amount of undercut you see will depend on your polishing technique• The amount of undercut you see will depend on your polishing technique.

Reflection Limits (Cabling Standards)

• ANSI/BICSI 002 – An OTDR shall be used … to evaluate uniformity of connectionsuniformity of connections.– Multimode: -40 dB (Recommendation)– Singlemode: -50 dB (Recommendation)

• ANSI/TIA-568-C.0 – Not for link testing.ANSI/TIA-568-C.3 – Connector specification– Multimode: -20 dB– Singlemode: -26 dB

• ISO/IEC 11801:2002 – Connector specification– Multimode: -20 dB– Singlemode: -35dB

• ISO/IEC 11801:2010 Amd 2 – Yes, but limits are unclear/ ,

Reflection Limits (Network Applications)

• IEEE 10GBASE-SR (Table 52-7, 52-9)T itt ORL T l 12 dB– Transmitter ORL Tolerance: 12 dB

– Receiver Reflectance Max: -12 dB

• IEEE 10GBASE LR (Table 52 12 52 13)• IEEE 10GBASE-LR (Table 52-12, 52-13)– Transmitter ORL Tolerance: 12 dB– Receiver Reflectance Max: -12 dBReceiver Reflectance Max: 12 dB

• IEEE 10GBASE-LRM (Table 68-3, 68-5)– Transmitter ORL Tolerance: 20 dBTransmitter ORL Tolerance: 20 dB– Receiver Reflectance Max: -12 dB

Effect of Reflectanceon Network Applicationson Network Applications

Effects of Connector Reflectance on Total Optical Return Loss at 850nmFiber = 100m, 50um, 1.5 dB/km

Connectors = 4x , equal reflectance (x axis), 0.5 dB loss

8

9

B)

Transceiver = 10GBASE-SR with -12 dB reflectance

1.7dB over limit if ConnectorReflectance is -20 dB

10

11

turn

Los

s (dB

10GBASE SR ORL Li i i 12 dB

No margin if ConnectorReflectance is -30 dB

12

13l Opt

ical

Ret 10GBASE-SR ORL Limit is 12 dB

13

14

Tota

l

15-45 -40 -35 -30 -25 -20 -15

Connector Reflectance (dB)

Effect of Reflectanceon Network Applicationson Network Applications

Effects of Connector Reflectance on Total Optical Return Loss at 1310nmFiber = 6km, Singlemode,0.3 dB/km

Connectors = 4x , equal reflectance (x axis), 0.5 dB loss

10

12

14B)

Transceiver = 40GBASE-LR4 with -26 dB reflectance

4.2 dB over limit if ConnectorReflectance is -20 dB

16

18

turn

Los

s (dB

40GBASE-LR4 ORL Limit is 20 dB

20

22

24l Opt

ical

Ret

4.0 dB margin if ConnectorReflectance is -30 dB

26

28

Tota

l

30-45 -40 -35 -30 -25 -20 -15

Connector Reflectance (dB)

Why Does FNET Suggest -35 dB?y gg• The network standards require -35 dB to provide a

reasonable margin relative to their maximum limitsreasonable margin relative to their maximum limits• Most (all) suppliers of OTDRs specify deadzones with

a connector reflectance of 35 dBa connector reflectance of -35 dB• The testing standards support this specificationANSI/TIA 526 14 BANSI/TIA 526-14-B

D.2.2 OTDRThe OTDR shall be capable of using a short pulse width (≤20 ns) and have sufficient dynamic range (> 20 dB) to achieve a measurement typically in lengths of up to 2 000 m The OTDR shouldrange (> 20 dB) to achieve a measurement typically in lengths of up to 2 000 m. The OTDR should have an attenuation dead zone (see G.2.4) less than 10 m following standard connectors (i.e. reflectance of –35 dB).

MEASURING REFLECTANCEMEASURING REFLECTANCE

Tools To Measure Reflectance• Optical Time Domain Reflectometer

– The most versatile fiber test toolThe most versatile fiber test tool– Today’s automated setup and analysis make it simple– Test times have been reduced to <15 seconds

d/ bl– See traces and/or Event Tables

Tools To Measure Reflectance• Optical Fiber Troubleshooter• Not your father’s “fault finder”• Not your father s fault finder

– Locates multiple reflective and/or loss incidents– No setup required– Test times = 6 seconds– Numeric read-out

Testing With An Optical Fiber TroubleshooterTroubleshooter

Testing With An OTDRg• A dirty OTDR port will cause a FAIL result• The port needs to be inspected and cleaned before you connect the p p y

launch fiber– All fiber connections need to be cleaned before connecting

• Some OTDRs have a grading scale for the OTDR Port (based upon itsSome OTDRs have a grading scale for the OTDR Port (based upon its reflectance)

Cleaning and Inspectingg p g• Isopropyl Alcohol (IPA)

– Must use as a minimum 98% IPA• Hybrid Cleaners

– Evaporate much faster– Anything less will leave a thin film behind,

causing a change in reflectance.

• But IPA has issues– Relatively slow evaporation result in halo

– Are more aggressive at cleaning– Have antistatic properties

• Having a cleaner with an antistaticproperties reduces the chance of

effect

• There is a preference among professionals to move to hybrid cleaners.

p pdust being attracted to the endface of the connector.

– According to the EPA, the typicallydust particle in the office is between2 and 10 µm.

Launch + Receive Compensationp• Required by standards

h l h i f h l• The only way to measure the properties of the last connector is to have a link of fiber attached to it

The Measurement• Good example• Spikes are small (good reflectance)Spikes are small (good reflectance)• Launch + Receive Compensation enabled• Configure a custom test limit with a -32dB

reflection limit for multimode fiber

Traces From Two Typical Multimode LinksTypical Multimode Links

• Poor reflectance widens the deadzone– Short multimode links require shorter deadzonesShort multimode links require shorter deadzones

• Poor reflectance may cause “negative” loss– Or the measurement may be impossible to make

Link 01 Link 02

The height of the event indicates the amount of reflectance.

Traces From Two Typical Multimode LinksTypical Multimode Links

• Poor reflectance widens the deadzone– Short multimode links require shorter deadzonesShort multimode links require shorter deadzones

• Poor reflectance may cause “negative” loss– Or the measurement may be impossible to make

Link 01 Link 02

The height of the event indicates the amount of reflectance.

Event Table From TwoTypical Multimode LinksTypical Multimode Links

• The OTDR event table is generated from automatic analysisanalysis– The measured reflectance value may be inspected– Pass/Fail details may be inspectedPass/Fail details may be inspected

Unable to measure

Link 01 Link 02

Excellent reflectance

Poor reflectance

Traces From Two TypicalMultimode Links– Digging DeeperMultimode Links– Digging Deeper

• The use of the receive fiber enables measurement of the receive event• The tailing caused by the poor reflectance limits the useful information g y p

available• Note the “ghost” caused by the poor reflectance

– All OTDRs have ghost.All OTDRs have ghost.

Link 01 Link 02





Link 01 Link 02





Link 01 Link 02





Link 01 Link 02





Link 01 Link 02





Link 01 Link 02

Attenuation Dead Zone vs. Event Dead ZoneDead Zone

• Attenuation Dead Zone is the minimum distance between two events on an OTDR where the OTDR can measure thetwo events on an OTDR where the OTDR can measure the event loss

• In this example, the following event is too close to first event

• The second event is within theAttenuation Dead Zone so we areAttenuation Dead Zone, so we are unable to measure the event loss of the first event at 0 ft/ 0 m/

Attenuation Dead Zone vs. Event Dead ZoneDead Zone

• Event Dead Zone is the minimum distance it can detect an event after the preceding event on an OTDRevent after the preceding event on an OTDR

• In this example, we can see that thereis an event 32 ft/9.8 m after the first event at 0 ft/0 m

• The Event Dead Zone depends on– The pulse width usedThe pulse width used – The reflectance of the preceding event

How Many Connectors In This 10BASE LR Link?10BASE-LR Link?

• To see connections close together we use a narrow pulse width– This will reduce the dynamic range and possibly reduce the accuracy ofThis will reduce the dynamic range and possibly reduce the accuracy of

your reflectance measurements, but allows a better view of what is in the channel

123 63 4 5

6













So there is a crossSo there is a cross connect here – two

connections

How Many Connectors In This 10GBASE LR Link?10GBASE-LR Link?

• If you need an accurate count of the connections in the channel just use ChannelMap™channel just use ChannelMap

• There are 13 fiber connections in this 10GBASE-LR channel• Not all OTDRs know how many connections are in a channel

Summaryy• Do not Plug-and-Pray• VFL testing is not testing• VFL testing is not testing• Perform full Certification Testing

– Inspect and Clean– Proper loss referencing (1-Jumper)– Meticulous loss testing

• Pre emptive OTDR testing for Event Loss and Connector• Pre-emptive OTDR testing for Event Loss and Connector Reflectance provides a rock solid testing strategy– Inspect and Clean the OTDR Port– Use Launch and Receive Cords– Strive for multimode connector reflectances <-35 dB

Documents

Data Center Testing - BICSI · Data Center Testing: The Hidden Impairment David SchellDavid Schell Principal Member of the Technical Staff Fluke NetworksFluke Networks July 2011 –