Public 14.02.2012

VISIT Deliverable Report Cover Sheet

Workpackage: WP5: Packaging, System-level Evaluation,

Standardization, and Exploitation Deliverable number D5.3a Deliverable name Final assessment of VISIT Project-related Standards Lead beneficiary: INT Workpackage leader (name): VIS Description writer (name): Prof. Dr. Nikolay N. Ledentsov Date planned: 31.10.2011 Date finished: 14.02.2012

Deliverable reached not reached

Deliverable description and summary of achieved results (max. 2400 char.): This Deliverable D5.3a is a supplement to the original Deliverable D5.3 report (dated 30 May 2010). Therefore the purpose of this Deliverable report is to provide a final update, assessment, and commentary on the current, on-going, and future planned Standards activities that are relevant to the component and systems technology that was (and continues to be through post-project exploitation) devleoped within the EC-funded FP7 VISIT project. As before in this report we briefly discuss progress on IEEE-directed Standards for optical data transfer such as the IEEE 802.3ba 40G and 100G Ethernet Standard. We also comment on the evolving Standards for the Common Electrical Interface, Infiniband, Fibre Channel, and consimer markets that include Light Peak (and Thunderbolt) optical (and electrical) interconnects.

Contributors: VIS

Public

VISIT Deliverable Report Technical Annex

Deliverable number: D5.3a

Deliverable name: Final assessment of VISIT Project-related Standards

An update on Standards for next-generation systems that include high-speed VCSEL-based optical interconnects and an assessment of Standards that may be impacted by the technology developed in the VISIT Project

Introduction

As silicon scaling continues, the density of transistors on a chip doubles each two years. The

computational power of the processor doubles allowing a dramatic increase in the computational

power. Within 2008-2010 the peak computational power of petascale1 supercomputers increased 10-

fold form 1 petaflops (IBM Roadrunner 2008) up to peak power of 8-10 petaflops (K-computer2,

Fujistu, 2011). Furthermore, construction of 50 petaflops (PF) supercomputers is already announced3.

At the same time reaching this performance is a challenging task, particularly for interconnects, which

may contribute for exascle supercomputers to above 80% of the total power consumption and up to

90% of the performance4. According to IBM, 5 million optical links may be needed already for 10PF

supercomputers5.

With silicon scaling and the related increase in the computational power of processors, it is important

to keep the same physical space for interfaces or in other words scale the bandwidth density with each

generation by a factor of two. As the number of channels may hardly be increased, the speed per

channel must double. This “scaling” is already an extremely challenging task. 1 petascale refers to the speed of the computer and means it could perform a thousand trillion mathematical operations a second

2 http://www.nsc.riken.jp/K/diary_eng.html

3 http://www.zdnet.co.uk/news/emerging-tech/2011/05/25/cray-taps-gpus-for-50-petaflop-supercomputer-40092887/

4 Alan Benner, IBM, "Optical Interconnects for HPC" Short-Distance High-Density Optical Interconnects. An OIDA Roadmapping Workshop, Stanford Photonics Research Center, Stanford University, CA, April 12-13, 2011

5 Bert Jan Offrein, IBM Silicon Photonics Workshop, Munich, 23 May 2011 http://www.siliconphotonics.eu/munich_slides/2_IBM.pdf

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 3/14 – Public –

Common Electrical Interface

Common electrical interfaces (CEI) are being developed by the Optical Internetworking Forum (OIF),

which promotes the development and deployment of interoperable networking solutions and services

through the creation of Implementation Agreements (IAs) for optical networking products, network

processing elements, and component technologies. The Common Electrical Interface Standard

supports the chip-to-chip and chip-to-module links allowing connections between different elements on

a printed circuit board (PCB) that services optical networking components. This has sparked great

interest in the development of suitable optical circuit boards and optical backplanes for high-speed

optical interconnects.

Presently ten OIF members are uniting to showcase multi-vendor participation in OIF Interoperability

2012 – Enabling High-Speed Dynamic Services. The OIF’s Physical and Link Layer (PLL)

demonstration will showcase interoperability of the Forum’s Common Electrical Interface (CEI) 28G

(28 Gbit per second or 28 Gbps) Very Short Reach (VSR) draft implementation agreement that defines

chip-to-module electrical interfaces. Demonstrations of the CEI-25G-LR signal for backplane interfaces

will also be tested. These demonstrations will be on display at the Optical Fiber Conference

(OFC/NFOEC) during 06-08 March 2012 at the OIF booth #713.

The associated marketing literature notes that, “This demonstration of CEI-28G-VSR shows the ability

to reach 100G for next-generation 4 x 25Gb/s based optical transceivers,” said Ed Frlan of Gennum

and the OIF PLL Interoperability Working Group Chair. “The CEI-25G-LR and CEI-28G-VSR electrical

interface and signaling schemes being tested for interoperability support multiple 100G applications.”

The following companies are participating in the OIF Interoperability 2012 during OFC 2012: Altera,

Amphenol, Fujitsu Optical Components, Gennum, IBM, Inphi, Luxtera, Molex, Tyco Electronics (TE)

Connectivity, and Xilinx. The equipment that will be used for a systems demonstration includes host

integrated circuits (ICs) with VSR SERDES ICs, host PCB traces, optical module connectors, module

re-timers, and optical transceivers. The test equipment used in the demonstration is supplied by

Tektronix, Inc. “This interoperability test and demonstration shows a critical mass, with multiple

vendors supporting the CEI-28G-VSR,” said Rod Smith of TE Connectivity and the OIF Marketing

Awareness & Education Committee co-chair. “There is momentum in the marketplace to incorporate

higher speed signals.”

The summary of the time frames for the related CEI-28G-VSR Standards acceptance is given next in

Figure 1.

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 4/14 – Public –

Figure 1. A Roadmap that illustrates timeframes for different Standard interfaces. Within 2012 four single channel electrical

interfaces at 25 Gb/s – 28 Gb/s are to be standardized.

In 2012 four types of copper interfaces at 25 to 28 Gbps are to be standardized: CEI-25G-LR, CEI-

28G-SR, CEI-28G-VSR, and FC32G6. The roadmap for 50 Gbps is already set (CEI-50G-SR, due for

certification in 2015). One should mention that the introduction of 28G interfaces will enable the

planned Fibre Channel 32GFC Standard in 2012 (which already includes optical interfaces) and

25Gx4 links for 100G IEEE next generation interfaces over copper (IEEE 802.3bj) and over optical

fiber (IEEE NextGenOPIX). The latter two Standards are expected to be ratified in 2014.

6 http://www.lsi.com/AIS2011/Documents/LSI_Deploying100GtoPreparing400G.pdf

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 5/14 – Public –

Infiniband

Infiniband represents the most important interface in high-performance (“super”) computing. As

Gordon Moore’s Law continues to be valid the computational power of computers continues to rapidly

increase (by about three orders of magnitude per decade reaching presently 10-20 petaflops7). To

match the demand the bandwidth density of interconnects must scale up with the serial data bit rate

per channel increasing 4-fold each 5 years. Copper now gives us data rates at about and above 10

Gigabits per second (Gbps). Already in the 10PF-scale IBM machine the number of optical links at 10

Gbps has increased to 5 million8. In 2012 the first two supercomputer systems at 20PF (IBM9 and

Cray) are due.

In Figure 2 an evolution of the data bit rate per single channel and per electrical link for the most

relevant high-performance computing Infiniband standard is shown. The roadmaps provided by the

Infiniband Industry Association in June 2010 and June 2011 are given. As one can see from the

comparison the 26 Gb/s data bit rate (eight data rate or EDR or 8X) expected for 2011 has been

shifted towards 2013. Instead an intermediate speed of 14 Gb/s will be applied. Even the reduced

speed is partially compensated by the improved efficiency of coding (97% for 64/66 versus 80% for

8/10), still there is no doubling of the affective data bit rate. To compensate the delay 26 Gb/s

(equivalent to 32 Gb/s for the old coding scheme) is planned for 2013. It is also important to note that

the HDR (hexadecimal data rate or 16X) Standard expected at ~50 Gb/s is still planned for use

starting in 2014. The delay with the EDR Standard clearly indicates that no adequate solution for such

an interconnection has been found. However, EDR was announced to be used in a 20PF

supercomputer this year (Sequoia by IBM in 2012). Thus either the solution will be outside of the

Standard, or the Sequoia Project will fail, similar to the failed 10PF Blue Waters project of IBM10 which

was still based on 10 Gb/s links.

7 http://www.top500.org/static/lists/2011/11/TOP500_201111_Poster.pdf

8 Bert Jan Offrein, IBM Silicon Photonics Workshop, Munich, 23 May 2011.

9 http://www.hpcwire.com/hpcwire/2012-01-12/first_racks_of_20-petaflop_sequoia_supercomputer_arrive_at_llnl.html

10 http://insidehpc.com/2011/08/08/ibm-cancels-blue-waters “Rumors at ISC’11 centered on problems with the Blue Waters proprietary interconnect.”

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 6/14 – Public –

Figure 2. Roadmap illustrating the increase in the serial transmission data bit rate for the most important Standard

(Infiniband) for high-performance computing. The serial data bit rate doubles each 2-2.5 years. Note the delay in the

introduction of the EDR standard in the roadmap of 2011. The scaling of the bandwidth density did not occur for 14 Gb/s

interfaces even assuming a 20% margin due to the improved coding efficiency for FDR versus QDR (97% versus 80%).

Fibre Channel

Fibre Channel is an important standard which defines data rates in Storage Area Networks (SANs).

Short-distance links are traditionally served with 850 nm VCSEL links with single receive and transmit

channels (see also Figure 1). The actual Fibre Channel roadmap is presented in Figure 3.

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 7/14 – Public –

Figure 3. Roadmap illustrating the increase in the serial transmission data bit rate for the Fibre Channel standard (Fibre Channel Speed Road Map v1311). 28 Gb/s bit data rate is to be accepted in 2012 causing a significant market demand by 2014.

As it follows from Figure 3 in three years after 2012 another doubling of throughput is to take place and

data bit rates beyond 50 Gb/s are to be targeted. However, already the 28 Gb/s data rate has caused

significant technical problems.

The major manufacturers of transceivers claimed that a 28 Gb/s solution will require additional

electronics because a fast and reliable VCSEL may not be available, "It is expected that in high

volume production, VCSEL-based, 28 Gb/s transmitters will have fall times longer than 18 ps.” (David

Cunningham, Avago "Example Calculation for 32GFC VCSEL Based Links", Fibre Channel Industry

Association, Plenary Meeting / T11 June 6 - 10, 2011 Coeur d'Alene, ID, USA, 11-230v0)12. “32GFC

VCSELs will likely not be twice as fast as 16GFC VCSELs and electronic dispersion compensation will

be required” (Richard V. Johnson, Finisar "32GFC Auto-negotiation considerations" Fibre Channel

Industry Association, Plenary Meeting / T11 June 6 - 10, 2011 Coeur d'Alene, ID, USA, 11-240v0)13.

11 http://www.fibrechannel.org/roadmaps

12 http://www.t11.org/ftp/t11/pub/fc/pi-6/11-230v0.pdf

13 http://www.t11.org/ftp/t11/pub/fc/pi-6/11-240v0.pdf http://www.t11.org/ftp/t11/pub/fc/pi-6/11-241v0.pdf

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 8/14 – Public –

Fibre Channel is another important opportunity for high speed VCSELs. There may be a delay in

moving forward with the Standards also here as with Infiniband. Some companies claim that reliable

25-28 Gb/s VCSELs will not be available and they seek to employ or re-consider the use of EDC, and

retiming circuits as part of optical fiber interconnections. This will certainly affect the power efficiency

and cost and will adversely impact the bandwidth density scaling requirements. See the following

presentations: http://www.t11.org/ftp/t11/pub/fc/pi-6/11-230v0.pdf, http://www.t11.org/ftp/t11/pub/fc/pi-

6/11-240v0.pdf, and http://www.t11.org/ftp/t11/pub/fc/pi-6/11-241v0.pdf. Some companies are seeking

to continue using 12.5 Gb/s VCSEL technology (operating at 14 Gb/s) from circa 2003. It is now clear

to all concerned that the Fibre Channel standards are becoming severely limited by the optical

components.

Ethernet

The latest update on 850 nm and 1310 nm Standards development for Ethernet is given at:

http://www.ieee802.org/3/100GNGOPTX/public/nov11/index.html and includes a presentation from

IBM. A new Study Group was launched in July 2011 and may last only for 6 months. The work of this

group is described at:

http://www.ieee802.org/3/100GNGOPTX/public/nov11/dove_01_1111_NG100GOPTX.pdf. The final

meeting of the Study Group to fix everything of key importance will take place in early 2012. This

group will fix the basics of the Standards for 850 nm and 1300 nm transmission.

Some companies argue for Standards at a wavelength of 1060 nm or 1100 nm VCSEL for single mode

fiber applications due to the lack of high-speed and robust 1300 nm VCSELs. These aspects are

described at: http://www.lightreading.com/document.asp?doc_id=38194&site=nfoec and

http://www.jdsu.com/News-and-Events/news-releases/Pages/jdsu-completes-acquisition-of-

picolight.aspx.

The 25G VCSEL is not dead despite the demise of the Infiniband EDR. There is a new wave of 100

Gb/s Ethernet standards coming soon including 25 Gb/sx4-850 nm VCSEL-based links and possibly

compact 1300 nm 25Gx4 interfaces. The wave of interest in these topics was launched at the July

2011 IEEE Plenary Meeting described at: http://www.ieee802.org/3/100GNGOPTX/public/index.html.

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 9/14 – Public –

100 Gb/s Backplane and Copper Cable Study Group

The tasks set by the 100 Gb/s Backplane and Copper Cable Study Group are presented in Figure 4

and the timeline is given in Figure 5. Four channels at a 25 Gb/s data bit rate are targeted 14.

Figure 4. Objectives for the 100 Gb/s Backplane and Copper Cable Study Group.

Figure 5. Standard time frames for the 100 Gb/s Backplane and Copper Cable Study Group

14 John D'Ambrosia (Dell) "Agenda and General Information", 100 Gb/s Backplane and Copper Cable Study Group Interim

Meeting Material 13-14 September 2011 Chicago, IL, USA, http://www.ieee802.org/3/100GCU/public/sep11/agenda_01_0911.pdf

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 10/14 – Public –

Next Generation 100 Gb/s Optical Ethernet Study Group

This group targets optical interfaces at 25 Gb/s x 4. The aim is to create more compact and smaller

cost interfaces opposite to the 802.3ba Standard (Figure 6).

Figure 6. Evolution of data traffic and developments on the stock exchange15.

However, the trend is counteracted by the proposed solutions of some key transceiver manufacturers

claiming the need for electronics (retiming, clock and data recovery, equalization, etc.) to compensate

the VCSEL deficiency16 as shown in Figure 7.

15 Mark Nowell, Matt Traverso, Gary Nicholl (Cisco)"Next-Gen 100G PMDs: Considerations when defining objectives", Next

Generation 100Gb/s Optical Ethernet Study Group Interim Meeting Material Sept 14th - 15th, 2011 Chicago, IL, USA) http://www.ieee802.org/3/100GNGOPTX/public/sept11/nowell_01_0911_NG100GOPTX.pdf

16 John Petrilla (Avago Technologies) Jonathan King (Finisar) "100G Next Gen Multi-Mode Optics" Next Generation 100Gb/s Optical Ethernet Study Group Interim Meeting Material Sept 14th - 15th, 2011 Chicago, IL, USA

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 11/14 – Public –

Figure 7. Impact of Clock and Data Recovery and equalization needed because of VCSEL deficiency on the power budget.

It is claimed as also for FC standardization that the transient time of VCSEL cannot be made lower

than 20 ps (Finisar, Avago) even VCSELs with deconvoluted rise time of 10 ps suitable for error-free

operation up to 40 Gb/s was already reported by VIS in 200917 and 40 Gb/s error free operation was

also reported by CTU in 201018.

Interestingly, VIS data was shown to illustrate the need in the electronics19 (Figure 8). Eyes obtained at

very low optical modulation power and using detectors with external amplifiers are certainly not

suitable for standard applications. Limiting amplifiers with the optimized bandwidth are to be

incorporated to reduce the noise and sensitivity dramatically and also to reshape the optical eye.

17 S. A. Blokhin, J. A. Lott, A. Mutig, G. Fiol, N. N. Ledentsov, M. V. Maximov, A. M. Nadtochiy, V. A. Shchukin, and D. Bimberg, “850 nm VCSELs operating at bit rates up to 40 Gbit/s,” Electronics Letters, 45, 501-503 (2009).

18 Westbergh, P., Gustavsson, J.S., Kögel, B., Haglund, Å., Larsson, A., Mutig, A., Nadtochiy, A., Bimberg, D., and Joel, A.: "40 Gbit/s error-free operation of oxide-confined 850 nm VCSEL" Electron. Lett. 46, 1014 (2010).

19 Ali Ghiasi (Broadcom) "100G-Optics Next Gen Possible PMD Set" Next Generation 100Gb/s Optical Ethernet Study Group

Interim Meeting Material Sept 14th - 15th, 2011 Chicago, IL, USA http://www.ieee802.org/3/100GNGOPTX/public/sept11/ghiasi_01_0911_NG100GOPTX.pdf

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 12/14 – Public –

(a)

(b)

Figure 8. Eye diagram obtained with a VIS PD having an external amplifier and an impact of integration of PD and TIA into a

40Gb/s- module having a small (TO-compatible) form factor. Eye opening is significantly improved by inserting the TIA into the

module even when the bandwidth of the TIA is not optimal for 25 Gb/s applications.

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 13/14 – Public –

Consumer Interfaces

Light Peak

The first optical interface presently applied in consumer applications is Light Peak. It provides two

channels at 5 Gb/s20. Sony is using an optical version of the Light Peal input/output (IO) interface

(Figure 9), rather than the copper cable supported by Apple’s thunderbolt implementation.

Figure 9. Light Peak optical interface used in Sony Vaio Z series.

Thunderbolt

Thunderbolt initially was developed in the copper cable version (two channels at 10 Gb/s). However,

presently an optical version is being prepared21. Fiber-optic cabling may allow for faster speeds across

even longer distances (up to "tens of meters," according to Intel). Thunderbolt was designed to scale

much higher up to 50 Gbps by 2015 and 100 Gbps being Intel's goal by the end of the decade (2019).

Standards Activities

Within the final period of the VISIT Project the VISIT partners continued their active participation in

Standards meetings and related activities. This included the following two invited talks:

1. A. Mutig, D. Bimberg, and J. A. Lott, “850 nm and 980 nm VCSELs,” in Proceedings OIDA Roadmapping Workshop: Short-

Distance, High-Density Optical Interconnects, Stanford, CA (12-13 April 2011).

2. J. A. Lott, N. N. Ledentsov, J.-R. Kropp, G. Kuyt, D. Molin, and A. Amezcua, “Progress on high speed 850 nm VCSELs for efficient multimode optical fiber data communications,” 11-263v1, Fibre Channel Standards FC-PI-6 Meeting, Coeur d‘Alene Idaho, USA (08 June 2011).

20 http://www.techhapp.com/2011/08/sony-uses-light-peak-in-vaio-z-laptop-series/

21 Thunderbolt Set to Go Optical in 2012 http://www.pcmag.com/article2/0,2817,2393677,00.asp

Deliverable number: D5.3a (a supplement to D5.3) Deliverable name: Final assessment of VISIT Project-related Standards

14 February 2012 14/14 – Public –

Although after the formal end of the VISIT project the partners have received additional requests to

participate in several up-coming Standards and related activities including the following confirmed

invited talk:

3. J. A. Lott, “Perspectives on 850 nm VCSEL-based extended-reach multimode optical fiber links at 25-56 Gb/s,” OIDA

Optical Fiber Communications Workshop, Shanghai, PR China (20-22 March 2012).

Conclusions

A Common Electrical Interface (CEI) electrical interconnection standard for 25-28 Gb/s is expected in

2012. Within 2012 four single channel electrical interfaces at 25 Gb/s – 28 Gb/s are to be

standardized. These standards for short-distance “electrical” interconnections will greatly influence

near-term optical interconnect standards at the same and faster data bit rates. The 850 nm VCSEL,

VCSEL packaging, and systems testing technologies developed within the VISIT Project are vital to

meet the future needs of the ultrashort (intra- and inter-chip) to short-reach (300 to 1 km) optical

interconnections.

High performance computing (“supercomputing”) continues to evolve with the planned deployment of

petaflops and exaflops systems using Infiniband Standards. This has give rise to a new emphasis on

optical “computer communication” where in the near-future each supercomputer will employ more than

a billion optical interconnections using high-speed (25 Gb/s) VCSEL technology. It is critical to

increase the data rate to 25 Gb/s or faster per channel and to reduce the power consumed by the

VCSELs and other components to on the order of a few milliwatts per channel. The expected

ratification date of the Infiniband EDR (8x or eight data rate) Standard has been moved from 2011 to

2013.

Progress on the next Fiber Channel Standard (FC32G – with VCSEL operation at about 28 Gb/s)

remains sluggish. Development of reliable VCSELs capable of operation at 28 Gb/s remains a difficult

commercial task. The FC 16G Standard still prevails, although eventually the 32G Fibre Channel

Standard will be ratified. The VCSEL technology developed within the VISIT Project is crucial for

progress toward FC32G and eventually toward a FC64G Standard.

Progress on short-reach Ethernet Standards toward 4x25G optical links at 850 nm and possibly at

1300 nm continues with vigor. Efficient, low-cost, and low-power consuming high-speed optical fiber

links at up to 1 km for use within data centers remains a critical as part of the deployment of new large-

scale data centers around the world in low power cost areas. The VCSEL technology developed within

the VISIT Project is critical for the continued expansion of the Internet and optical communications

worldwide.

Deliverable number: D5.1 Deliverable name: Assessment 1 of IEEE standard definition

14 February 2012 15/14 – Public –

New opportunities for VCSELs are emerging in several areas of optical communication including near-

term application in computer links (850 nm Light Peak-based systems and 850 nm optical USB-based

systems), in fiber-to-the-home systems (using 1300 nm-range high-speed VCSELs), but also in

sensing (environmental controls, gesture recognition, and many more).