Bcaf Nutshell

BCAF 2013 in a Nutshell Study Guide for Exam 143-130

Brocade University

Revision 0213

Corporate Headquarters - San Jose, CA USAT: (408) [email protected]

European Headquarters - Geneva, SwitzerlandT: +41 22 799 56 [email protected]

Asia Pacific Headquarters - SingaporeT: [email protected]

© 2013 Brocade Communications Systems, Inc. All Rights Reserved.

ADX, AnyIO, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, and Vyatta are registered trademarks, and HyperEdge, The Effortless Network, and The On-Demand Data Center are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of their respective owners.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.

Revision 0213

Brocade Certified Architect for FICON 2013 in a Nutshell First Edition

Introduction to BCAF 2013 in a Nutshell First Edition

Objective: The BCAF 2013 Nutshell guide is designed to help you prepare for the BCAF 2013 Certification, exam number 143-130.

Audience: The BCAF 2013 Nutshell self-study guide is intended for those who have successfully completed the CAF200 Fundamentals of Brocade Mainframe Networking course, and who wish to undertake self-study or review activities before taking the actual BCAF 2013 exam. The BCAF 2013 guide is not intended as a substitute for classroom training or hands-on time with Brocade products.

How to make the most of the BCAF 2013 guide: The BCAF 2013 guide summarizes the key topics on the BCAF 2013 exam for you in an easy to use format. It is organized closely around the exam objectives. We suggest this guide be used in conjunction with our free online knowledge assessment test. To benefit from the BCAF 2013 guide, we strongly recommend you have successfully completed the CAF200 Fundamentals of Brocade Mainframe Networking course.

We hope you find this useful in your journey towards BCAF 2013 Certification, and we welcome your feedback by sending an e-mail to [email protected].

Joe CannataCertification Manager

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. iii

mailto:[email protected]

mailto:[email protected]


iv © 2013 Brocade Communications Systems, Inc. All Rights Reserved.

Brocade Certified Architect for FICON 2013 16Gbps in a Nutshell Second Edition

Table of Contents1 — Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

System z Host, I/O Channel Subsystem and FICON Protocol Architecture . . . . . . . . . . . . . . . 1Mainframe Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Mainframe Models: zSeries to System z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3What is FICON? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

FICON Frame Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3FICON Operating Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4FCP and FICON Mode Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Channel Subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Multiple CSS Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Logical Channel Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Channel Spanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Open Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7FICON Connection Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8FICON Express Cards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Brocade MLX Series Multiservice Ethernet routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

z/OS Discovery and Auto-configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Parallel Sysplex and Coupling Links. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11zHPF - High Performance FICON. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Brocade Network Advisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Event Notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Reasons to Migrate from ESCON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16FICON CUP Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Features of CUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Prohibit Dynamic Connectivity Mask (PDCM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Mainframe Reporting with RMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Creating RMF 74-7 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

FICON CUP and FOS Firmware Upgrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2 — Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Single Director vs. Cascaded Director Design Requirements . . . . . . . . . . . . . . . . . . . . . . . . 212-byte Link Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Configuring CHPIDs Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Scalability Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Buffer-to-Buffer Credits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Long Distance Link Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

FICON Emulation Requirement for a Determinate Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27FCIP Tunnel Between Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Brocade FCIP Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. v


High Performance FCIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Traffic Isolation Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

FC Long Distance Cascading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30FCIP Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Protocol Intermix Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32FICON and FCP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Virtual Fabrics and FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Supported Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Cloud Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Inter-chassis Links (ICLs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34N_Port ID Virtualization (NPIV) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Benefits of Switched FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Minimize Mainframe Channel Card Costs with Switched FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Remote Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Synchronous Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Asynchronous Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

FICON Emulation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Bandwidth Allocation and Restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3 — Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Domain ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Switch ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Fill Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Traffic Isolation Zoning Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41TI Zone Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Illegal ETIZ Configuration: Separate Paths from a Port to Devices on Same Domain . . . . . . . . . . . . . . . . . . 42

256-area Addressing Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Port Swapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Geographically Dispersed Parallel Sysplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Troubleshooting Trunking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Native Connectivity (M-EOS interoperability) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44ISL Decommission. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4 — Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Command Line Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Diagnostic Port. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Pathinfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Blade Swapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Web Tools - Performance Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46SAN Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

IOCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Types of Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

vi ©2013 Brocade Communications


FICON CUP Zoning and PDCM Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Fabric Watch Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Reasons to Customize Fabric Watch Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Fabric Monitoring Setting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Advanced Performance Monitoring Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49RMF 74-7 Records. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

RMF FICON Director Activity Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51IFCCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52Saving or Copying Allow/Prohibit Matrix Configurations to Another Device . . . . . . . . . . . . . 52Cascaded FICON fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Firmware Downloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

FCIP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5 — Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Top Talker Monitors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55General Information to Gather for all Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Taking the Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. vii


viii ©2013 Brocade Communications


List of FiguresIBM System z10 Mainframe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2I/O definition comparison (FCP to FICON and ESCON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Channel Subsystem Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6FICON Connection Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Parallel Sysplex connectivity with mixed Coordinated Timing Network (CTN) . . . . . . . . . . . . . . . . . . . . . . . .11IBM and Brocade Integrated TS7700 Grid Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13CUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17Single vs. Cascaded FICON Director Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .212-Byte Link Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24Sample B-Series Trunk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25FCIP tunnel and circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27FC long distance cascading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31Brocade DCX with ICLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34Illegal ETIZ configuration: two paths from one port to two devices on the same remote domain . . . . . . .42SAN Health Output with IOCP Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47FICON Director Activity Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50FICON Director Activity Report for an M6140 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. ix


x © 2013 Brocade Communications Systems, Inc. All Rights Reserved.


List of TablesBrocade enterprise-class platform blade terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. xi


xii © 2013 Brocade Communications Systems, Inc. All Rights Reserved.


1 — ArchitectureAfter completing this section you should be able to read a scenario and demonstrate your knowledge of the following:

• Describe System z host, I/O channel subsystem, and FICON protocol architecture.

• Demonstrate knowledge of System z channel I/O system configuration.

• Describe System z I/O, channel, and CUP interdependencies.

System z Host, I/O Channel Subsystem and FICON Protocol Architecture

Mainframe Overview• A mainframe is one machine that is a central electronics complex (CEC)

• A CEC will have many Central Processing Units (CPUs) in it

• A CEC can support logical partitions (LPARs)

• System z can support up to 60 LPARs

• CPUs are allocated to LPARs dynamically

• Special CPUs are called Service Assist Processors (SAPs)

• Each LPAR can run a different Operating System such as z/OS and Linux

Mainframe is simply an industry term for a large computer. The majority of mainframes today are manufactured by IBM. The mainframe has historically been associated with centralized rather than distributed computing but IBM refers to its mainframes as a large server. These mainframes can be used to serve distributed users and smaller servers in a computing environment. This is made possible with capabilities of these to run thousands of virtual Linux images connected via Fibre Channel to storage.

Mainframe Models: zSeries to System z• In 2000, IBM launched the zSeries 900 line of servers which were referred to as the zSeries and included

models 800, 890, 900, and 990

• In 2000, IBM renamed the System/390 to IBM eServer zSeries

• In 2005, IBM launched the z9 series which included IBM z9 Enterprise Class and Business Class

• In 2008, the z10 Enterprise Class was launched

• Since FICON has been available since the late 1990s, the 9672 G5/G6 was the first mainframe to provide support for FICON

© 2013 Brocade Communications Systems, Inc. All Rights Reserved. 1


• zSeries, System z including z9 and z10

- 800, 890 (midrange enterprise)

- 900, 990 (enterprise, large enterprise; business class UNIX)

- z9, z10 (enterprise, large enterprise; business class UNIX)

- z Operating System (z/OS)

- z Virtual Machines (z/VM)

- z Transaction Processing Facility (zTPF)

- Linux for System z

• Elements that affect unrepeated distance

- The type of fiber used

- The port transceiver type

FIGURE 1 IBM System z10 Mainframe

2 © 2013 Brocade Communications Systems, Inc. All Rights Reserved.


Terminology• Link Incident Record Registration (LIRR)

The LIRR Extended Link Service (ELS) requests that the recipient add the requesting port to its list of ports that are to receive a Registered Link Incident Report (RLIR).

• Registered Link Incident Report (RLIR)

RLIR ELS provides a way for a node port to send an incident record to another node port.

• Request Node Identification Data (RNID)

RNID ELS acquires the associated node’s identification data, which provides configuration discovery and management purpose information.

• Missing Interrupt Handler Primary Timeout (MIHPTO)

A value and which determines how long the channel will wait before timing out an expected response from the CUP.

What is FICON?• FIbre CONnection (FICON) is:

- An I/O protocol based on Fibre Channel (FC-SB-2 and FC-SB-3) used with IBM (and compatible) mainframes and storage

- A layered protocol based on industry standards for Fibre Channel (FC) architecture using FC layer 4.

- The successor to ESCON

• Introduced in the late 1990s, the original FICON adapter card which enabled FICON was developed for the 9672 G5 and G6 mainframes

FICON Frame EncodingFICON frame encoding can be either FC standard 8b/10b used for 1, 2, 4 and 8 Gbps or 64b/66b with 10 Gbps ports. There are two types of frame encoding:

• 8b/10b encoding

• 64b/66b encoding

To use 64b/66b encoding, you must use a 10 Gbps ISL which cannot auto-negotiate with 1, 2, 4 and 8 Gbps ports.



FICON Operating ModesThere are two FICON operating modes: FCV and FC. For System zSeries and 9672 G5 and G6 servers, there were two supported modes:

• FICON Bridge – referred to as FCV which is a FICON channel mode designed to enable access to ESCON interfaces using the FICON Bridge Adapter in the ESCON Director (9032-5)

• FICON Native – referred to as FC which enables access to the FICON channel mode from native FICON control units in a point-to-point mode through a FICON Director (No cascaded FICON)

For zSeries and System z including z9 and z10, support was added for:

• FICON Native (FC), but not FICON Bridge

• Supported topologies: point-to-point, switched point-to-point, and cascaded FICON Directors

• Fibre Channel Protocol (FCP) which provided support for open systems using industry standard SCSI devices

FCP and FICON Mode CharacteristicsProbably the single largest difference between the FICON channel (FC and FCV) and FCP channel mode types is the treatment of data access control and security. FICON and ESCON channels rely on MIF to address concerns regarding shared channels and devices. MIF provides ultra-high access control and security of data so that one operating system image and its data requests cannot interfere with another operating system’s data requests. With the introduction of the System z, MIF continues this ultra-high level of access control and security across channel subsystems (CSSs).

Linux guest operating systems under z/VM can have access to an FCP channel defined to the z/VM operating system. Using MIF, an FCP channel can also be shared between Linux Logical Partitions and z/VM Logical Partitions with Linux guests.

The FCP industry-standard architecture does not exploit the data access control and security functions of MIF. As a result, FCP has the following limitations:

• Channel sharing. When N_Port ID Virtualization (NPIV) is not implemented, and if multiple Linux images share an FCP channel, and all of the Linux images have connectivity to all of the devices connected to the FCP fabric, then all of the Linux images use the same worldwide port name to enter the fabric and, thus, are indistinguishable from each other within the fabric. Hence, the use of zoning in switches and LUN masking in controllers cannot be effective in creating appropriate access controls among the Linux images.

By using NPIV, each operating system sharing an FCP channel is assigned a unique WWPN. The WWPN can be used for device-level access control in storage controllers (LUN masking) and in switch-level access control (zoning).

• Device sharing. Without using NPIV, an FCP channel prevents logical units from being opened by more than one Linux image at a time. Access is on a first come, first served basis. This is done to prevent problems with concurrent access from Linux images that are sharing the same FCP channel and thus the same worldwide port name. This usage serialization means that one Linux image can block other Linux images from accessing the data on one or more logical units, unless the sharing images (z/VM guests) are all well-behaved and not in contention.



FICON channels in FCP mode use the Queued Direct Input/Output (QDIO) architecture for communication with the operating system. The QDIO architecture for FCP channels derives from the QDIO architecture that initially was defined for the OSA-Express features and for HiperSockets communications. FCP channels do not use control devices. Instead, data devices that represent QDIO queue pairs are defined, consisting of a request queue and a response queue. Each queue pair represents a communication path between an operating system and the FCP channel. It allows an operating system to send FCP requests to the FCP channel via the request queue. The FCP channel uses the response queue to pass completion indications and unsolicited status indications to the operating system. Figure 2 on page 5 illustrates this process.

FIGURE 2 I/O definition comparison (FCP to FICON and ESCON)



Channel SubsystemThe key to moving data into and out of a mainframe is the channel subsystem (CSS). When an I/O operation is required, the CSS is passed the request from the main processor to manage. While awaiting completion of an I/O request from the CSS, the main processor can continue processing other data. This process supports concurrent transactions because of the interaction between the host processor and the CSS.

FIGURE 3 Channel Subsystem Overview

Multiple CSS ConceptThe design of System z systems offers considerable processing power, memory size, and I/O connectivity. In support of the larger I/O capability, the CSS concept has been scaled up correspondingly to provide relief for the number of supported logical partitions, channels, and devices available to the system.

A single channel subsystem allows the definition of up to 256 channel paths. To overcome this limit, the multiple channel subsystems concept was introduced. The architecture provides for up to four channel subsystems. The structure of the multiple CSSs provides channel connectivity to the defined logical partitions in a manner that is transparent to subsystems and application programs, enabling the definition of a balanced configuration for the processor and I/O capabilities.

Each CSS can have from 1 to 256 channels and be configured to 1 to 15 logical partitions. Therefore, four CSSs support a maximum of 60 logical partitions. CSSs are numbered from 0 to 3 and are sometimes referred to as the CSS image ID (CSSID 0, 1, 2 or 3). These CSSs are also referred to as logical channel subsystems (LCSSs).



Logical Channel SubsystemThe z/OS operating system is limited to a maximum of 256 channel paths (CHPIDs) for use within a logical partition (LPAR). To facilitate the usage of more CHPIDs, the mainframe architecture supports a logical channel subsystem (LCSS). The LCSS is functionally identical to the channel subsystem but up to four LCSSs can be defined within a central processor complex. CHPIDs are unique within the LCSS only; consequently, the 256 CHPID limitation can be overcome. Usage can be maximized with NPIV and good fan-in/fan-out ratios.

Channel SpanningChannel spanning extends the MIF concept of sharing channels across logical partitions to sharing physical channels across logical partitions and channel subsystems.

Spanning is the ability for a physical channel (PCHID) to be mapped to CHPIDs defined in multiple channel subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the channel subsystem to which the logical partition is configured.

A channel is considered a spanned channel if the same CHPID number in different CSSs is assigned to the same PCHID in IOCP, or is defined as spanned in HCD.

In the case of internal channels (for example, IC links and HiperSockets), the same applies, but with no PCHID association. They are defined with the same CHPID number in multiple CSSs.

Open ExchangeAn open exchange is part of FICON (and FC) terminology. Many I/O operations can be in progress over FICON channels at any one time. For example, a disk I/O operation might temporarily disconnect from the channel while performing a seek operation or while waiting for a disk rotation. During this disconnect time, other I/O operations can be managed, as explained here:

• Command mode open exchanges

In command mode, the number of open exchanges is limited by the FICON Express feature. FICON Express8, FICON Express4, FICON Express2 allow up to 64 open exchanges. One open exchange (actually, this is an exchange pair) in command mode is the same as one I/O operation in progress.

• Transport mode open exchanges

In transport mode, one exchange is sent from the channel to the control unit. Then the same exchange ID is sent back from the control unit to the channel to complete the I/O operation. The maximum number of simultaneous exchanges the channel can have open with the CU is up to 750 exchanges. The CU sets the maximum number of exchanges in the status area of the transport mode response IU (the default number is 64 and can be increased or decreased).



FICON Connection Components

FIGURE 4 FICON Connection Components

FICON Express Cards• Original FICON card:

- 100 MB/sec full duplex

- LX: supports 9 micron SM fiber

- SX supports 50, 62.5 MM fiber

- FICON Express2 (LX):


- Supports 9 micron SM fiber

• FICON Express2 (SX):


- Supports 50, 62.5 micron MM fiber

- Supported on all z9 processors

• FICON Express4 (LX and SX):

- 400 MB/sec full duplex (the IBM certified maximum data rate capability for FICON is 350 MB/sec)

- Supports 9, 50, 62.5 micron SM and MM fiber

- Supported on System z9 and z10 servers

- FICON Express4-2C adapters are supported on System z9 BC servers only, and go 550m or 4km

• FICON Express2, FICON Express4, FICON Express8, and FICON Express8S features do not support FCV mode. FCV mode is available with FICON Express LX feature 2319 (carry forward only on System z10). FCV was used with the FICON Bridge solution.



• FICON Express8 (LX and SX):

- 800 MB/sec full duplex.

- Supports 9 micron SM fiber and 50, 62.5 micron MM fiber.

- Supported only on System z10 servers.

- Support for FC and FCP modes.

- Direct memory access using the QDIO architecture.

• FICON Express8S is the most current FICON card

- Allows for the consolidation of existing FICON Express, FICON Express2, FICON Express4, and FICON Express8 channels onto fewer FICON Express8S channels while maintaining and enhancing performance.

- Offers performance improvements for High Performance FICON for System z (zHPF), and Fibre Channel Protocol (FCP) to ensure the zEnterprise EC12, zEnterprise 196 and zEnterprise 114 servers continue to allow your bandwidth to increase to meet the demands of your business applications.

- Operates at 2, 4, or 8 Gbps autonegotiated.

- Has a feature supporting single mode fiber optic cabling (9 micron) terminated with an LC Duplex connector - FICON Express8S 10KM LX.

- Has a feature supporting multimode fiber optic cabling (50 or 62.5 micron) terminated with an LC Duplex connector - FICON Express8S SX.

• FICON Express8S channels

- Each zEC12, z196, or z114 FICON Express8S channel has two channels (ports) per feature.

- Each FICON Express8S channel has two possible modes of operation designed for connectivity to servers, switches/directors, disk, tape, and printers:

• CHPID type FC – FICON, zHPF, and channel-to-channel (CTC) traffic for the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments

• CHPID type FCP - Fibre Channel Protocol (FCP) for attachment to SCSI devices for the z/VM, z/VSE, and Linux on System z environments

The maximum distance for a single director is 10km from the host mainframe. If you are trying to determine the maximum distance of a device attached to the director to the mainframe, you need to add in that distance as well. For example, a director could be attached to a mainframe that is 5km away and a storage device is attached to the Director which is also 5km away. The total distance then that the storage device can be from the mainframe is 10km.

Brocade MLX Series Multiservice Ethernet routersThe Brocade MLX routers provide a rich set of high performance IP version 4, IP version 6, and Multiprotocol Label Switching (MPLS) and advanced Layer 2 switching capabilities. As a result, these routers address the diverse needs in environments that include service provider backbones, data centers, and distributed enterprises.



The Brocade MLX routers are highly optimized for IP Ethernet deployments, providing symmetric scaling with chassis options that include 4-, 8-, 16-, and 32-slot systems. These routers offer industry-leading wire-speed port capacity without compromising the performance of advanced forwarding capabilities. For example, the Brocade MLXe-32 delivers data forwarding performance in excess of 6 Tbps today and scales to 15.36 Tbps. This capacity is enough to future proof networks for years to come. In addition, the robust control plane is proven in thousands of mission-critical deployments around the globe.

The Brocade MLX routers enable reliable converged infrastructures and support key applications. These routers feature advanced redundant switch fabric architecture for high availability. The architecture helps ensure that the system continues to operate at peak performance even if a switch fabric card failure happens. If additional fabric failures occur (although unlikely), with the advanced architecture, the system continues operating in a graceful degradation mode. In this mode, the system tunes its performance to the remaining fabric capacity.

The Ethernet line modules for the Brocade MLX routers are available in two different types:

• The 1-GbE line modules are available in 24- and 48-port models

• 10 GbE line modules are available in 4-port and 8-port models

The Brocade MLX routers use Brocade Network Advisor application, which offers comprehensive unified network management for all Brocade products. Brocade Network Advisor provides an easy-to-use MPLS Manager. MPLS Manager can help to configure, monitor, and manage services across networks. Brocade Network Advisor also uses sFlow-based technology that provides the following capabilities:

• Proactive monitoring

• Traffic analysis

• Reporting

Brocade Network Advisor helps to reduce network downtime. In addition, Brocade Network Advisor offers administrators end-to-end network visibility from a single dashboard.

z/OS Discovery and Auto-configurationz/OS discovery and auto-configuration (zDAC) is a function, exclusive to zEC12, z196 and z114, that automatically performs a number of I/O configuration definition tasks for new and changed disk and tape controllers that are connected to a FICON Director. It is designed to help simplify I/O configuration of zEnterprise CPCs running z/OS, and to help reduce complexity and setup time.

The zDAC function is integrated into the existing Hardware Configuration Definition (HCD) tool. A policy can be defined in HCD according to the availability and bandwidth requirements, including parallel access volume (PAV) definitions, control unit numbers, and device number ranges. The zDAC proposed configurations are created as work I/O definition files (IODF) that can be converted to production IODF and activated.

zDAC provides real time discovery for the FICON fabric, subsystem, and I/O device resource changes from z/OS. By exploring the discovered control units for defined LCUs and devices, zDAC compares the discovered controller information with the current system configuration to determine delta changes to the configuration for a proposed configuration. All new added or changed logical control units and devices will be added into the proposed configuration with proposed control unit and device numbers and channel paths based on the defined policy. zDAC uses a channel path chosen algorithm to minimize single point of failure. zDAC applies to all FICON features supported on zEC12, z196 and z114 when configured as CHPID type FC.



Parallel Sysplex and Coupling LinksThe Parallel Sysplex technology represents a synergy between hardware and software and comprises Parallel Sysplex-capable servers, Coupling Facility, coupling links (IFB, IC, ICB, and ISC-3), Server Time Protocol (STP), optionally Sysplex Timers2, shared DASD, and software, both system and subsystem, designed for parallel processing as shown in Figure 5.

FIGURE 5 Parallel Sysplex connectivity with mixed Coordinated Timing Network (CTN)

Parallel Sysplex technology is a highly advanced, clustered, commercial processing system. It supports high-performance, multisystem, read/write data sharing, enabling the aggregate capacity of multiple z/OS systems to be applied against common workloads.

The systems in a Parallel Sysplex configuration are linked and may fully share the same devices and run the same applications. This allows users to harness the power of multiple zEC12 and System z systems as if they were a single logical computing system.

The architecture is centered around the implementation of a Coupling Facility running the Coupling Facility Control Code (CFCC) and high-speed coupling connections for intersystem and intrasystem communications. The Coupling Facility is responsible for providing high-speed data sharing with data integrity across multiple zEC12 and System z servers.



Parallel Sysplex is designed to provide high availability for business critical applications. The design is further enhanced with the introduction of System-Managed Coupling Facility Structure Duplexing which provides the following additional benefits:

• Availability: Structures do not need to be rebuilt in the event of a coupling facility failure

• Manageability and usability: A consistent procedure is established to manage structure recovery across exploiters

• Configuration benefits: A sysplex can be configured with internal CFs only

A parallel Sysplex supports connectivity between systems differing by up to two generations. For instance, an IBM System z9 cannot participate in a Sysplex with a zEC12, but zEnterprise systems and z10 systems can.

zHPF - High Performance FICONWith the introduction of High performance FICON (zHPF), the FICON architecture has been streamlined by removing significant overhead to the storage subsystem and the microprocessor within the FICON channel. A command block is created to chain commands into significantly fewer IUs. The overhead required to convert individual commands into FICON format is removed as multiple System z I/O commands are packaged together and passed directly over the fibre optic link. For zHPF operations, we lose the concept of the Command Control Word (CCW) and it is replaced with the Transport Control Word (TCW). The TCW allows multiple channel commands to be sent to the control unit as a single entity instead of a stream of individual commands. Compatibility is established such that existing FICON operations (CCWs) coexist with zHPF operations (TCWs).

Essentially, zHPF reduces the number of information units (IUs) processed. Enhancements have been made to the z/Architecture and the FICON interface architecture to provide optimizations for OnLine Transaction Processing (OLTP) workloads.

On the zEC12 and z196 the multitrack operations extension applies exclusively to the FICON Express8S, FICON Express8, and FICON Express41, when configured as CHPID type FC, and connecting to z/OS. zHPF requires matching support by the DS8000 series, otherwise the extended multitrack support is transparent to the control unit.

From the z/OS point of view, the existing FICON architecture is called command mode and zHPF architecture is called transport mode. During link initialization, the channel node and the control unit node indicate whether they support zHPF.

ATTENTIONAll FICON channel paths (CHPIDs) defined to the same Logical Control Unit (LCU) must support zHPF. The inclusion of any non-compliant zHPF features in the path group will cause the entire path group to support command mode only.

1. FICON Express4 LX 4KM is not support on zEC12.



The mode used for an I/O operation depends on the control unit supporting zHPF and settings in the z/OS operating system. For z/OS exploitation there is a parameter in the IECIOSxx member of SYS1.PARMLIB (ZHPF=YES or NO) and in the SETIOS system command to control whether zHPF is enabled or disabled. The default is ZHPF=NO.

Support is also added for the D IOS,ZHPF system command to indicate whether zHPF is enabled, disabled, or not supported on the server.

Brocade Network AdvisorMost IT organizations that use IBM zEnterprise and TS7700 Grid networks face similar challenges in managing their networks. These challenges include minimizing downtime and managing application service-level agreements (SLAs). At the same time, they must find ways to reduce operational expenses. As networks become increasingly complex and critical to business operations, many IT organizations rely on multiple point solutions that address only parts of the network. This network complexity leads to cumbersome, error prone operations. For example, you might have separate management tools (and servers) for each of the components that are illustrated in Figure 6.

FIGURE 6 IBM and Brocade Integrated TS7700 Grid Network



Brocade Network Advisor is the industry’s first unified network management solution for data, storage, and converged networks. Brocade Network Advisor supports various networks, including the following examples:

• Fibre Channel SANs including 16 Gbps platforms

• Fibre Channel over Ethernet networks

• Layer 2/3 IP networks, application delivery networks

• MPLS networks in service provider environments

Brocade Network Advisor is an ideal tool to use for managing a TS7700 Grid network. Brocade Network Advisor gives the user the ability to manage all of the TS7700 Grid network hardware components from a single user interface by using a single server. With Brocade Network Advisor, organizations have a single management framework for their entire network infrastructure. Brocade Network Advisor has intelligently designed interfaces and full role-based access control (RBAC) to support the needs of different network teams.

NOTEFMS must be enabled on the local switch for the remote CUP to work.

Brocade Network Advisor can be used to manage storage and data networks separately or through a single instance of the product. This capability enables better coordination between storage and data networking administrators that perform provisioning, troubleshooting, and reporting. Brocade Network Advisor also helps organizations reduce network downtime through the following capabilities, among other types:

• Real-time and historical performance monitoring

• Traffic analysis

• Change management

• Policy driven remedial actions



Proactive alerts with real-time logging, diagnostic, and fault isolation capabilities help resolve issues before they affect SLAs. For more granular performance analysis, Brocade Network Advisor uses the hardware-based sFlow technology. The sFlow technology is available on all Brocade IP switches and routers. The sFlow technology provides real-time network monitoring and accounting capabilities without affecting network performance. The Traffic Analyzer feature provides trend analysis, management, and monitoring tools for sFlow reporting, accounting, and presentation for all IP devices. Brocade Network Advisor also offers open, standards-based interfaces. It also provides deep integration with a wide range of third-party network monitoring, virtualization, wireless, and security solutions. It integrates with leading server and storage automation solutions to bridge operational gaps across server, network, and storage administrators. This integration provides end-to-end network visibility through frameworks such as the Storage Management Initiative-Specification (SMI-S). Of substantial benefit to the TS7700 Grid user, Brocade Network Advisor provides support for leading data center orchestration frameworks, including IBM Tivoli® Storage Productivity Center (TPC). Brocade Network Advisor and IBM TPC integration is focused on administrators within the same domain, in this case, storage and SAN management. The IBM TPC solution is an industry-leading storage resource management solution that helps customers with the management of storage environments. It simplifies various tasks that are related to the following capabilities:

• Logical unit number provisioning

• Capacity planning

• Replication

• Reporting

IBM TPC also performs basic SAN management activities for Brocade Fibre Channel infrastructure by using the Brocade standards-based SMI-S interface.

Event NotificationThe Management application records the SAN and IP events in the Master Log. You can configure the application to send event notifications to e-mail addresses at certain time intervals. This is a convenient way to keep track of events that occur on the SAN and IP networks. You can also configure products to “call home” for certain events, notifying the service center of product problems.

Filtering Event NotificationsThe application provides notification of many different types of SAN events. If a user wants to receive notification of certain events, you can filter the events specifically for that user.



Reasons to Migrate from ESCON• FICON is an improved protocol over the ESCON protocol

- ESCON provides a single I/O per channel whereas FICON uses multiple, simultaneous ESCON-type I/Os

- ESCON I/O is half duplex; FICON I/O is full duplex

- FICON provides improved support over distance decreased “data droop”

- FICON offer channel path consolidation

- FICON provides increased addressing

- FICON offers a bandwidth increase

- Buffer credit control

- FICON adapters can support Open Systems and LINUX (FICON/FCP intermix)

- zHPF support

• FICON uses Open Systems components

- Fibre Channel Directors

- Fibre Channel cables & connectors

- Fibre Channel adapters

• Migration Considerations:

- Buffer credits

- SX/LX requirements as applicable

FICON CUP SupportControl Unit Port (CUP) is an embedded and not a physical port. It allows in-band management of directors from the management applications on a mainframe and is u.sed for configuration, monitoring, and error handling.

• CUP support is provided by all Brocade certified FICON switches and Directors

- Optional licensed feature for both B-Series and M-Series

- FICON Management Server license required to activate CUP

- Supported in single or cascaded FICON environments

- Utilizes embedded port – FE

• CUP is typically used by customers with:

- SA z/OS (System Automation software) configuration tool

- RMF (Resource Measurement Facility) monitoring tool

• If a mainframe administrator wants to be able to do in-band management from their Hardware Management Consoles – (HMC), they need CUP



• If it is important to them to get the Service Information Messages (SIM) to the z/OS console for FICON device hardware failures

- Interpret these messages using the IBM Mainframe Systems Messages and Codes manual

- If there is a need to do RMF reports for reporting on FICON Director activity statistics in the Record Type 74 Subtype 7, CUP is mandatory

- RMF option FCD/NOFCD in ERBRMFxx enables or disables FICON CUP Director

• Although IBM has developed a new machine type, 2027, for FICON Directors, all Directors and switches, regardless of model or manufacture should be configured as 2032

• Active=Saved will prevent a loss of connectivity on a Director using CUP

FICON Directors have the embedded port “FE” for the CUP port. On FICON Directors that have 256 or more ports, this has caused a slight dilemma. This logical “FE” overlaps the physical “FE” port, so the physical “FE” and “FF” ports cannot be used on these Directors for FICON connectivity. They may still be used for port swaps or for FICON/FCP intermix, however, they cannot be “genned” in HCD for FICON connectivity.

In a FICON environment, only one RMF LPAR should attempt to access the CUP port at any one time, and it is still best practice to have two or more CHPIDs with access to the CUP. However, too much activity to the FICON CUP can cause missing interrupts, leading to the potential for having a boxed device.

FIGURE 7 CUP2

2. Footnote 1 from Figure 7 on page 17: Safe switching is the ability to manipulate ports and adjust path status non-disruptively. In order for System Automation I/O operations to provide safe switching, the System Automation I/O must have access to all switches. All the switches must be online as I/O devices on all the systems where System Automation is running in order to verify that a path can be removed before the path is actually removed.



This is a simplified representation of an FICON environment and how CUP fits into that environment. Best practice is to have two or more CHPIDs with access to the CUP and implement safe switching. Safe switching describes the System Automation process of path removal. Safe switching is a process that involves contacting all systems in a sysplex and verifying a path can be removed before it is actually removed.

Features of CUP• Remote switch functions via z/OS System Automation

• Reporting of failed FRUs to z/OS

• Needed to add Directors to the I/O sysgen for error reporting purposes

• May require the CUP device be taken offline before a firmware upgrade

• Prohibits communication between port pairs

• Allows the host to set the switch to an offline state

Prohibit Dynamic Connectivity Mask (PDCM)• CUP also provides PDCM as a management tool. The PDCM is a mechanism to define port connectivity

such as prohibits, allows, and blocking/unblocking of ports.

• The FICON Prohibit Dynamic Connectivity Mask (PDCM) controls whether or not communication between a pair of ports in the switch is prohibited or allowed

• Block versus Prohibit:

- Blocking causes the firmware to send a continuous “offline” sequence to the port

• Useful to report the link as inactive after varying a device off on the mainframe

- Prohibit causes the firmware to “prevent” connectivity between the ports

• Useful to force FICON traffic over specific ISLs

• PDCM controls whether or not communication between a pair of ports in the switch is prohibited or allowed

• If there are any differences in restrictions set up with Brocade Advanced Zoning and PDCM, the most restrictive rules are automatically applied

Mainframe Reporting with RMFAs mainframes have grown, monitoring resources to provide peak performance has become increasingly complex. Resource Measurement Facility (RMF) provide tools for performance measurement and management for a single z/OS system or a system complex of z/OS systems:

• RMF is used to gather data, report data, and access data across a sysplex from a central administration point.

• RMF is used to track problems back to their sources, no matter where the problem occurred within the enterprise to enable administrators to respond before the problems become critical.



• RMF when used with CUP, can gather data about FICON Directors and make that data available to the mainframe system administrator thus providing a single point of administration.

CUP, or Control Unit Port is a holdover from ESCON Directors. In a FICON environment, CUP allows for in-band management and device statistics reporting. This opens the door to FICON director performance metrics reported out via the RMF 74-7 record, more commonly known as the FICON Director Activity Report.

When you install FICON Management Server (FMS) on a FICON switching device, and then enable CUP on that device, you provide yourself with a lot of valuable tools:

• RMF reports on FICON fabric statistics such as Frame Pacing Delay (the only way to get this information into RMF) and SA z/OS IO/OPs can have in-band access to the FICON switching devices and can manage and monitor them. This is how to get switch related hardware errors to be reported to z/OS against a device (domain ID) number. If the switch is not defined as an I/O device through the use of CUP, and/or that I/O device is not online to z/OS, then switch related errors cannot be surfaced up to z/OS so that it can take appropriate actions.

• Systems Automation can monitor and manage fabric components

• Date and Time can be set by z/OS for log and event synchronization

• The Allow/Prohibit Addressing Matrix becomes available

• The Active=Saved condition becomes user configurable

• At FOS 7.0+, the Port De-Commissioning Feature will be able to function

Creating RMF 74-7 RecordsEnabling RMF 74 subtype 7 (RMF 74-7) records, used for switches and directors, yields an RMF report called the “FICON Director Activity Report”. RMF 74-7 records are collected for each RMF interval if:

• The FMS license is installed on the switching device

• The FMS indicator has been enabled on the switch

• IOCP has been updated to include a CNTLUNIT macro for the FICON switch

• IECIOSnn parmlib member is updated to include “FICON STATS=YES”

• FCD is specified in the ERBRMFnn parmlib member

- FCD/NOFCD can also be modified through an operator command

Only a single LPAR in the sysplex should be gathering these RMF statistics:

• The data contained in a FICON switch is systems level data

• The master RMF LPAR will distribute this data to other LPARs

• Relieves congestion due to having all LPARs asking directly for this data

To use CUP with Systems Automation, the IO Operations (IOOPs) module must be licensed and implemented.



FICON CUP and FOS Firmware UpgradesDuring the FICON Director firmware download process a failover to the backup processor occurs. Any outstanding CUP commands are lost and result in a MIHPTO timeout (usually 3 minutes). Inefficient usage of MIHPTO can result in device boxing. If the CUP falls behind in honoring requests it begins getting strikes against it at the host programming level and after three strikes, the CUPs switch will be boxed. This occurs when upgrading firmware on a FICON switching device. To resolve the boxing issue, vary the CUP port 0xFE offline before starting the firmware upgrade and when complete vary the CUP port back online.

The ficoncupset mihpto command can be used to increase, decrease, or display the MIHPTO value that the CUP returns to the host when configuration data is read. Changing the MIHPTO value requires that the CUP control device be varied offline from the host, as the host will not automatically be notified when the value is changed. The host will normally read the new value, using the Read Configuration Data command, when the CUP is varied back online.



2 — Design ConsiderationsAfter completing this section you should be able to describe and demonstrate your knowledge of the following:

• Single vs. cascaded director design requirements

• Scalability considerations

• Knowledge of Virtual Fabrics

• Factors that influence FICON performance

• High Availability (HA) and disaster recovery solutions

• FICON and FCP intermix configurations

• Extension solutions

Single Director vs. Cascaded Director Design Requirements

FIGURE 8 Single vs. Cascaded FICON Director Environments

A hop is defined as one ISL connection. More than one ISL connection is not supported between directors. Therefore, you can have two directors connected together through an ISL but not more than two directors. In this example, the cascaded configuration shows how an open system server can also be included in the configuration as the open system server is attached to one director that is attached to another director.



• In open systems, connecting two Directors together is achieved by creating an Inter-Switch Links (ISLs) between the machines. In the FICON environment, this concept of connecting two Directors together is called cascading. M-Series Directors use preferred path.

• The standard FC routing mechanism for cascaded links is called Fabric Shortest Path First (FSPF).

• The FSPF protocol compares the cost of various paths between a source Director and a destination Director by adding the costs of the ISLs along each path. FSPF chooses the path with minimum cost. If multiple paths exist with the same minimum cost, FSPF distributes the load among these paths.

- Database maintained of ISL and ICL link states- Reliable flooding is not the method of synchronization between two switches

• Required Fabric OS settings are dlsreset and iodset

- iodset causes a delay to make sure frames have left the fabric• On M-Series Directors. the rerouting delay can lead to I/O timeouts

TABLE 1 Brocade enterprise-class platform blade terminology

ATTENTIONThe 10 GbE SFPs used in the FX8-24 blade and the 10 Gbps FC SFPs used in the FC16-32/48 blades are NOT interchangeable.

Term Abbreviation Blade ID Definition

32-port 16-Gbps port blade FC16-32 97 A 32-port Brocade platform port blade supporting 2, 4, 8, 10, and 16 Gbps port speeds.With Virtual Fabrics (VF) enabled, you can create up to eight 32-port domains.

NOTE: 10 Gbps speed for FC16-xx blades requires the 10G license.

DCX Extension blade FX8-24 75 A 24-port Fibre Channel routing and FCIP blade that also has 10 1-GbE and two 10-GbE ports and is compatible with the Brocade DCX, DCX-4S, and DCX 8510 family CP blades.With VF enabled, you can create up to eight 48-port domainsUp to 4 FX8-24 blades are supported in the DCX 8510 chassis using FOS v7.0.



Connectivity Blades• 48-port blades require Interop Mode 0 (IM=0), so you cannot use a mix of B-Series and M-Series in a

fabric:

- Eight slots filled with 48-port blades would put 384 ports into a single domain ID

- IBM z/OS only allows up to 256 ports to be within a single domain ID

- This creates a requirement for the use of zero-based addressing and Brocade Virtual Fabrics on a Brocade DCX

- Zero-based addressing is required, but it is not supported in IM=2 mode, only in IM=0 mode

- Therefore, on a Brocade DCX, zero-based addressing requires the use of Brocade Virtual Fabrics (VF) thereby making VF a requirement when using 48-port blades on a Brocade DCX

- On a Brocade DCX-4S, the chassis has only four slots; filled with 48-port blades, it would put 192 ports into a single Domain ID

- Since z/OS allows up to 256 ports to be within a single Domain ID, the Brocade DCX-4S does not have to be run using Brocade Virtual Fabrics (although VF can still be deployed if desired)

- On the Brocade DCX-4S, IM=2 was never qualified and, therefore, 48-port blade use is limited to IM=0 only

2-byte Link Addressing• ISLs are not used nor defined

• The first byte is the Domain ID (CHPID) and the second byte is the switch port (egress port)

• All CHPIDs with 2-byte link addresses require all links accessing the CHPID to be a 2-byte link

• 2-byte addressing enables cascaded switch support; the second byte enables a Domain ID to be specified in the IOCP definition

• Remember that intermixed IOCP definitions (1-byte and 2-byte link addresses) on the same channel path are not allowed

• When you create 2-byte FICON addresses in a mainframe configuration:

- The mainframe channels will issue the Query Security Attributes CCW at initialization time (after the FLOGI) to see if the entry FICON Director supports what IBM commonly calls a high integrity fabric (2-byte addressing not 1-byte addressing)

- The channel expects the QSA Response to contain two bits indicating fabric binding and insistent Domain IDs (IDID) are configured and established throughout the entire fabric

- If these bits come back false (either one), then the channel stops and terminates the login process

- The QSA response will be affirmative:

• IF and ONLY IF security is implemented on each Director in this fabric

• Fabric Binding is enabled

• Insistent Domain_ID is enabled



- If any of this is NOT true, then the QSA command will indicate a "high integrity fabric" is not available, and the channel will not initialize

• The only way to recover the channel at this point is to either use single byte addresses or turn on Fabric Binding and Insistent domain ID for every switch in the fabric

The QSA command will not be sent if 1-byte addressing is used for a FICON Director and is in the CNTLUNIT macro in the IOCDS. QSA is only used for cascaded-FICON or high integrity FICON. Anytime you see a 1-byte address for a Director, you know that Director is not part of a cascaded environment.

FIGURE 9 2-Byte Link Addressing

Configuring CHPIDs Online

Two settings must be implemented to configure the CHPIDs online, fabric binding and insistent domain IDs. After you successfully activate the new I/O definitions, you can configure the channel path online and try to communicate to the devices.

Scalability Considerations

TrunkingThe trunking feature optimizes the use of bandwidth by allowing a group of links to merge into a single logical link, called a trunk group. Traffic is distributed dynamically and in order over this trunk group, achieving greater performance with fewer links. Within the trunk group, multiple physical ports appear as a single port, thus simplifying management. Trunking also improves system reliability by maintaining in-order delivery of data and avoiding I/O retries if one link within the trunk group fails.



Trunking is frame-based instead of exchange-based. Since a frame is much smaller than an exchange, this means that frame-based trunks are more granular and better balanced than exchange-based trunks and provide maximum utilization of links.

• Trunking combines two or more physical ISLs into a single logical link, while limiting FSPF rerouting

• Trunk group characteristics:

- Frames are multiplexed across ISLs in the trunk group

- One port in the trunk group represents the link in the routing database

• Trunking license required for all switches participating in trunking

• Available when the license is installed and ports are reinitialize

• Trunk ports must operate at a common port speed

• Trunk ports must originate and end in valid port groups (i.e. 0-7, 8-15, etc.)

• When trunking criteria is met the trunk forms automatically

• Long distance trunks require Extended Fabric licenses on both ends of the link

FIGURE 10 Sample B-Series Trunk Group

High Availability

Buffer-to-Buffer Credits• Define the maximum amount of data that can be sent prior to an acknowledgement

• Buffer credits are physical ASIC port or card memory resources and are finite in number as a function of cost

• Within a fabric, each port may have a different number of buffer credits

• The number of available buffer credits is communicated at fabric logon (FLOGI)

• One buffer credit allows a device to send a fixed 2112 byte frame of data (2K usable for z/OS data)

• Assuming that each credit is completely full, you need one credit for every 1km of link length over a 2 Gbps fiber

• Unfortunately, z/OS disk workload rarely produces full credits. For a 4K transfer, the average frame size for a 4K transfer is 819 bytes.

• Five credits would be required per km over a 4 Gbps fiber



• The “optimal” amount of buffer credits is determined by the distance (frame delivery time), the processing time at the receiving port, link signaling rate, and the size of the frames being transmitted:

• Credit = (round-trip-time + receiving-port-processing time) / frame-transmission-time

• Information required for calculation:

- Link speed (1, 2, 4, 10 Gbps)

- Actual fiber distance in km that the frame must traverse

- The size of the frame – can be difficult to obtain

• Formula for assigning buffer credits using 2148 frame size

- 1 Gbps where distance (in km) / 2 + 20%

- 2 Gbps where distance (in km) + 20%

- 4 Gbps where distance (in km) x 2 + 20%

- 10 Gbps where distance (in km) x 6 + 20%

• If you always assume a frame size of 2148 bytes, in the vast majority of cases, it will produce too few buffer credits

• For extended links, RTT and the average frame payload size must also be considered

Long Distance Link ModesUse the portcfglongdistance command to support long distance links and to allocate sufficient numbers of full size frame buffers on a particular port. Changes made by this command are persistent across switch reboots and power cycles. This command supports the following long-distance link modes:

• Static Mode (LO) - L0 is the normal (default) mode for a port. It configures the port as a regular port. A total of 20 full-size frame buffers are reserved for data traffic, regardless of the port operating speed; therefore, the maximum supported link distance is up to 5 km at 2 Gbps, up to 2 km at 4 Gbps, and up to 1 km at 8, 10, and 16 Gbps.

• Static Mode (LE) - LE configures an E_Ports distance greater than 5 km and up to 10 km. LE does not require an Extended Fabrics license. The baseline for the buffer credit calculation is one buffer credit per km at 2 Gbps. This yields the following values for 10 km:

- 10 buffer credits per port at 2 Gbps.



- 50 buffer credits per port at 10 Gbps

- 80 buffer credits per port at 16 Gbps

• Dynamic Mode (LD) - LD calculates BB credits based on the distance measured during port initialization. Brocade switches use a proprietary algorithm to estimate distance across an ISL. The estimated distance is used to determine the BB credits required in LD (Dynamic) extended link mode based on a maximum Fibre Channel payload size of 2,112. You can place an upper limit on the calculation by providing a desired_distance value. Fabric OS confines user entries to no larger than what it has estimated the distance to be. When the measured distance is more than desired_distance, the desired_distance (the smaller value) is used in the calculation.



• Static Long-Distance Mode (LS) - LS calculates a static number of BB credits based only on a user-defined desired_distance value. LS mode also assumes that all FC payloads are 2112 bytes. Specify LS mode to configure a static long distance link with a fixed buffer allocation greater than 10 km. Up to a total of 1452 full-size frame buffers are reserved for data traffic, depending on the specified desired_distance value.

FICON Emulation Requirement for a Determinate PathFICON emulation processing creates FICON commands and responses on extended CHPIDs, and intercepts all exchanges between a channel and a CU. For FICON Emulation processing to function correctly, all the exchanges between a channel and CU must take a single tunnel path.

There are two ways to ensure a determinate path for FICON commands and responses:

• Define only one FCIP tunnel between sites

• Use a Traffic Isolation zone (TI Zone) to assign a specific tunnel to Channel and control unit ports. In the cases where multiple tunnels are required between a pair of FICON switches (or Logical Switches), use TIZ to define ports that should use specific tunnels.

FCIP Tunnel Between SitesThe Brocade 7800 and FX8-24 uses FCIP Trunking features to overcome the limitation of one Ethernet interface, one IP address and one FCIP tunnel. In Fabric OS v6.3 and later an FCIP Tunnel is created with multiple FCIP circuits over different IP interfaces to provide WAN load balancing and failover recovery in the event of a limited WAN outage. This provides a highly redundant WAN configuration for all FICON or FCP emulation technologies with Fabric OS. Figure 11 shows that a tunnel can have up to 4 circuits per GbE port.

FIGURE 11 FCIP tunnel and circuits



Brocade FCIP ConsiderationsAPTpolicy must be supported for FICON if more than one VE_Port must be used. Multiple FX blades will require more than one VE_Port. Only certain APT policies are supported for FICON, such as PBR (Port Based Routing), however, with FCIP, multiple VE_Ports between the same two domains is not supported because when there are routing changes it can cause an IFCC.

Data in-flight will be lost (IFCC) under these circumstances:

• Modifying or deleting a circuit or tunnel

The Lossless feature with PBR does not extended beyond the C2/C3 ASICs internal to the 7800/FX blade, therefore, routing changes that occur will result in data loss across the FCIP ISLs thus causing an IFCC. If multiple FCIP complexes are required, such as obtaining 20 Gbps on a single blade or using multiple FX blades in a single 8510 chassis, this is not supported because FCIP Trunking cannot be configured across FCIP complex and across FX blades. A recommended alternative architecture would be to utilize multiple VF LS with a single VE_Port in each. Two LSs with one VE_Port in each can fully utilize both FCIP complexes on an FX blade. Channels will have to be distributed across the two LSs.

• FOS FW upgrade/downgrade

Firmware upgrades/downgrades and modifying or deleting a circuit or tunnel in which circuits belong to, can result in an IFCC because data being transmitted will be lost during those operations.

Data will be delivered out of order (IFCC) under these circumstances:

• Protocol optimization with more than one VE_Port between the same domains

Protocol optimization is only supported when there is a single VE_Port connecting two domains. This establishes a deterministic path for data that must pass through the same VE_Ports outbound as it does on the return. The path is not deterministic if two fabrics are connected with multiple links as the fabric may choose to send data out on one path and return on a different path, which must have different VE_Ports. If the VE_Ports are not consistent the result will be breaking the protocol optimization state machine. FCIP Trunking solves this problem by allowing multiple connections while using a single VE_Port and always delivering data in-order.

• Wrong APTpolicy with more than one VE_Port between domains

The following are considerations when connecting Virtual Fabric logical switches between sites:

• The FCIP tunnel should be located in its own logical switch

• The XGE interface must be in the default switch

• There must be a VE_Port in each environment

• VLAN tagging is done per FCIP circuit

High Performance FCIPPP-TCP-QoS is a special QoS function exclusive to Brocade extension and developed especially for high-performance FCIP. For QoS to function properly across an IP network, it is essential that each priority have its own flow. This is not possible using only a single TCP session, because there would only be a single merged flow. PP-TCP-QoS provides a TCP session for each priority flow, so that the IP network can perform according to the QoS settings. FCIP Trunking is fully compatible with PP-TCP-QoS.



7800/FX support three levels of priority (H/M/L). The default amount of bandwidth (BW) that the scheduler will apportion during times of contention is 50/30/20%. QoS portioning of BW only occurs during times of contention, otherwise, the BW is shared equally across all priorities. It is possible to change the default portions to any values you wish as long as H>M>L and the aggregate of all the priorities equals 100%.

There are four TCP sessions per FCIP circuit H, M, L, and F-class. F-Class uses a strict queuing which means if there is any F-Class to send, it all gets sent first. There is very little F-Class traffic and it does not interfere with data traffic. Each TCP session is autonomous and does not rely on other TCP sessions or settings. Each TCP session can be configured with its own DSCP, VLAN tagging and 802.1P values. This permits that TCP session (priority) to be treated independently in the IP network from site-to-site based on the SLA for that QoS priority.

Traffic Isolation ZoningThe Traffic Isolation Routing feature may be used to control the flow of interswitch traffic through the Brocade extension switch or blade VE_Port or E_Port connections. Traffic Isolation Routing uses a special zone, called a Traffic Isolation zone (TI zone), to create dedicated paths for specific traffic.

When setting up TI Zones in your FICON environment, it is not recommended to have failover enabled for the following reasons:

• FICON Emulation will not occur on a failed over path

• FICON devices are identified at one time during channel path activation (ELP/LPE exchange) — if the paths do not fail, the MVS system will not re-issue this sequence over the recovery path

• Multiple error periods will be perceived at the connected LPARs (first link failure, movement of traffic to recovery path and then again when the primary path is restored)

• There is no good way to perform a forced or controlled fallback

• Keep the following best practices in mind when using TI zoning:

• If TI Zones are used to provide routes for Emulated traffic then TI Zone Fail-over cannot be enabled

• Keep the configurations as simple as possible, this will make the systems more supportable

• Include virtual E_Ports in the Traffic Isolation zone

• Use TI zoning for route selection on a port-to-port basis.

• Use port zoning to restrict data flows

• Look at TI zoning, link costs, and zoning definitions to understand the routing paths used in your data center.

• Understand bandwidth requirements, available network resources, etc

• Collect information on all new installations or after network changes by running the portCmd --ipPerf command

• Separate disaster recovery networks from production networks



Enhanced TI ZoningEnhanced Traffic Isolation Zones allow you to specify a preferred and exclusive path through a cascaded fabric for a particular flow by defining a set of devices or ports to appear in more than one Enhanced TI Zone. A preferred path is one that has failover enabled and an exclusive path has failover disabled. An exclusive path in the following topologies is beneficial as it will limit the number of hops between directors:

• Triangular

• System Data Mover

• Emulation

Enhanced TI Zoning for the topologies listed above is available only on the following platforms running Fabric OS v6.4.0 or later:

• Brocade 5100

• Brocade 5300

• Brocade 7800

• Brocade DCX

• Brocade DCX-4S

• Brocade 8510

FC Long Distance CascadingOutside of a local data center, FICON cascading is for connectivity. Connecting FICON attachments between data centers allows for the traffic flow of data between those sites especially for business continuance and/or disaster recovery. This is accomplished using ISL links. Cascaded FICON introduces the open systems SAN concept of the Inter-Switch Links (ISLs). IBM now supports the flow of traffic from the processor through two FICON directors connected through an ISL and on to the peripheral devices such as DASD and tape.

The majority of long distance connectivity is between two sites but it is becoming more prevalent to create a number of remote locations to geographically distribute the computer processing elements to overcome a disaster overtaking one of the data centers.

Below is an example of Brocade Blades with Long Wave Optics and Dark Fiber Connectivity for Applications and RDR. The connectivity requirements are as follows:

• Dark Fiber connections between two data centers

• Dark Fiber can be either owned or leased by the customer



FIGURE 12 FC long distance cascading

The solution involves the following:

• Use the proper Long Wave Length SFP

• Use CWDM/DWDM to optimize fibers

• Provision a proper number of buffer credits

• Disaster Recover and Business Continuance reduces RPO and RTO

FCIP Trunking FCIP Trunking is a Brocade innovation and exclusive technology that was originally developed for the mainframe. Its primary purpose is to provide scalability and resiliency across multiple FCIP connections while preventing IFCC. FCIP Trunking aggregates BW across multiple FCIP circuits forming a single logical ISL between two VE_Ports.FCIP Trunking performs load balancing with failover/failback if a circuit goes offline. If data in-flight is lost when a circuit goes offline that data is retransmitted across remaining links, and put back into order prior to being sent to the Upper Layer Protocols (ULP, i.e. FICON or FCP). All data is always delivered in-order. There is no requirement to enable IOD to get in-order delivery with FCIP Trunking.

FCIP trunking characteristics:

• FCIP Trunking is an FCIP Tunnel with more than 1 circuit

• Commonly known as a Link Aggregation Group (LAG)

FCIP Trunking provides the following benefits:

• Single logical ISL in routing table as a single link, ULP sees a single link

• Bandwidth Aggregation of each circuit

• Load Balancing per batch across the available circuits



• Failover to remaining circuits when a circuit is lost

• Lossless Link Loss (LLL)

- Data in-flight is not lost when a link goes down, it will be retransmitted

- In-Order-Delivery (IOD) Does not require switch “IOD” to be enabled

- Data in-flight will be delivered in the correct order, even after a data loss in-flight

• Works with both FICON and FC

- Supports FastWrite, OSTP, and FICON emulation over multiple circuits

Protocol Intermix ModeProtocol Intermix Mode (PIM), which is also referred to as FICON/FCP intermix, is supported by IBM on mainframes to enable FICON and open systems Fibre Channel Protocol (FCP) traffic to co-exist on the same physical storage network.

When operating in a Protocol Intermix Mode (PIM) environment it is a good practice to not only zone FICON ports away from FCP ports but also to control any currently unused mainframe ports by “prohibiting” them using PDCMs or port disables. Ports that are prohibited will still have light but will not allow connectivity to any other port. You could also block the port, turning off the light, which also disallows connectivity. When you want to connect up an unconnected FICON port then you can go to the Allow/Prohibit Matrix and “allow” that port to have connectivity (or unblock it).

Linux on System z and Storage Array replication through FICON switching devices are arguably the two top reasons a customer might deploy a PIM environment.

FICON/FCP intermix can provide cost reductions such as the consolidation and management of both switching infrastructure and cabling plants as well as access to the latest advances in technology, such as the N_Port ID Virtualization (NPIV).

• Examples for intermix:

- Small FICON and open systems environments with a common storage network

- z/OS or System z servers accessing remote FICON storage through FICON cascading

- Linux on a mainframe running z/OS using FICON to access local storage

- Hardware-based remote DASD mirroring between two locations using FCP as a transport, for example, PPRC

- A mainframe processor that accesses storage via FICON and storage devices via FCP to perform remote mirroring between devices (instead of ESCON)

- Open systems servers accessing storage on the same storage network using FCP

- Linux on the zSeries located at either of two sites using FCP to access storage

NOTEOpen systems use discovery; FICON uses IOCP



FICON and FCP• A FICON channel in Fibre Channel Protocol mode (which is CHPID type FCP) can access FCP devices as

follows:

- From a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a SCSI device

- From a FICON channel in FCP mode through a single Fibre Channel switch or multiple switches to a Fibre Channel-to-SCSI bridge

• FCP support enables Linux on System z to access industry-standard SCSI devices

• FICON Express4, FICON Express2, and FICON Express channels in FCP mode provide full fabric attachment of SCSI devices to the operating system images, using the Fibre Channel Protocol, and provide point-to-point attachment of SCSI devices

• For disk applications, these FCP storage devices use Fixed Block (512-byte) sectors instead of Extended Count Key Data (ECKD) format used with FICON storage devices

If you are considering creating a disaster recovery site by synchronously replicating data between DASD at the two sites, you would need to use FCP protocols between the DASD but still use cascaded FICON Directors to manage the data flow. In this case, you need an intermix environment of FICON and FCP.

• The FCP channel full fabric support enables switches and Directors to be supported between the System z server and SCSI device, which means many “hops” through a storage area network

• FICON channels in FCP mode can use the Queued Direct Input/Output (QDIO) architecture for communication with the operating system

FCP channels do not use control devices but instead, data devices that represent QDIO queue pairs are defined. These queue pairs consist of a request queue and a response queue. Each queue pair forms a communication path between an operating system and the FCP channel. The enables an operating system to send FCP requests to the FCP channel via the request queue. The FCP channel then uses the response queue to pass a completion message and unsolicited status messages to the operating system.

HCD/IOCP is used to define the FCP channel type and QDIO data devices. There is no definition requirement for the Fibre Channel storage controllers and devices in IOCP, nor the Fibre Channel devices such as switches and Directors because of QDIO.

The FCP industry-standard architecture specifies that the Fibre Channel devices (end nodes) in a Fibre Channel network use using World Wide Names (WWNs), Fibre Channel Identifiers (IDs), and Logical Unit Numbers (LUNs) for addressing. These addresses are configured at the operating system level, and passed to the FCP channel together with the corresponding Fibre Channel I/O via a logical QDIO device (queue).



Virtual Fabrics and FICON

Supported Features• Multiple Logical Switches (up to 8 on a DCX)

• Can utilize 384 ports on a DCX chassis

• Utilizes 48-port blades

• Two instances of CUP (one CUP license per logical switch)

• Not on base switch

• Only supported in Native and Interopmode 2

Cloud Environments

Inter-chassis Links (ICLs)If thought is being given to using some sort of cloud infrastructure, there are two technologies that could be of assistance in a FICON environment. The first is the use of ICLs on a Brocade DCX.

FIGURE 13 Brocade DCX with ICLs

• Each ICL is equivalent to 16 standard ISL ports (at 8 Gbps each) – but take up no user ports

• ICL cables can be a max of 2m in length

• ICLs are “silent” hops; IBM does not “count” them as an “official” hop for FICON cascading. By using ICLs, you can use four DCXs in a cascaded environment that does not “violate” the one hop rule.



• Cascading of Directors and switches is limited to one hop for a FICON environment with the following exceptions:

- The DCX Backbone with ICLs consists of 2 or 3 domains, but the hop across the ICLs can be disregarded

- Because of this DCX<->ICL<->DCX<->ISL<->DCX<->ICL<->DCX is considered one hop

- FICON is supported for 16, 32 and 48 port blades

- 48-port blades on a DCX require the use of Virtual Fabrics

ICL Trunking on the Brocade DCX 8510-8 and 8510-4The Brocade DCX 8510-8 has 4 port groups on the CR16-8 core blade. The Brocade DCX 8510-4 has 2 port groups on the CR16-4 core blade. Each port group has 4 QSFP connectors, and each QSFP connector maps to 4 user ports.

Each of the 4 user ports in a QSFP terminates on a different ASIC, so a trunk cannot be formed among these ports.

To establish ICL trunking between platforms in the Brocade DCX 8510 family, follow these configuration rules:

• You need at least 2 ICLs between the platforms. A single ICL does not enable trunking

Each QSFP has four ports. However, these ports cannot form a trunk with each other, but can form trunks only with corresponding ports on another QSFP.

• You can have a maximum of 4 ports in an ICL trunk

• You can have a maximum of 8 4-port trunks to a neighboring domain. Each core blade can have a maximum of 4 ICLs to a neighboring domain

• The QSFP cables must be connected to the same trunk group on each platform

N_Port ID Virtualization (NPIV)NPIV is designed to allow the sharing of a single physical FCP channel among operating system images, whether in logical partitions or as z/VM guests in virtual machines. This is achieved by assigning a unique World Wide Port Name (WWPN) to each operating system connected to the FCP channel. In turn, each operating system appears to have its own distinct WWPN in the SAN environment, thus enabling separation of the associated FCP traffic on the channel.

Access controls based on the assigned WWPN can be applied in the SAN environment using standard mechanisms such as zoning in FC switches and Logical Unit Number (LUN) masking in the storage controllers.

NPIV is exclusive to the System z servers and is the preferred method for sharing SCSI devices among multiple operating system images when using a single physical FCP channel.



Benefits of Switched FICON• Minimize the number of CHPID ports required for connection to storage.

• Switched FICON allows for simpler infrastructure management, lowered infrastructure cost of ownership, and higher data availability.

• Keep port, cable, or HBA failures to a single side of the system rather than allowing a complete connection to be disrupted.

• Can utilize Fan In – Fan Out ratios to continue to scale new or existing mainframe applications even if all the storage ports are in use.

• Distribute workload more evenly and predictably across all of the available storage ports through a well thought out Fan In – Fan Out ratio

• Ability to use NPIV with Linux

• Can provide very robust, dynamic channel path connectivity providing ease of growth and scalability throughout the FICON environment.

• Ability to implement Control Unit Port (CUP) to obtain information such as buffer credit usage and other performance statistics.

• Allow DASD arrays to contain maximum capacity that can actually be utilized due to the Fan In – Fan Out architecture and balanced use of the storage ports.

• Use FICON as a distance extension to support disaster recovery between two sites.

Minimize Mainframe Channel Card Costs with Switched FICON• Each storage port in a point-to-point connection requires its very own physical port connection on the

mainframe

• In a cascaded FICON environment, the fan-in/fan-out ratios solve this problem just like it solves other connectivity and scalability problems

- Fan-in: Attaching servers to storage via connected server/switch ports to a single storage subsystem port

- Fan-out: Attaching multiple storage control units to a single server/switch

• Every storage box has a maximum number of FICON connections:

- Tape might be 1 or 2

- Disk might be 2 to 48

• In a cascaded FICON environment, you can use fan-in/fan-out ratios

- Use all storage ports

- Add more capacity to the DASD

• Use CHPID fan-in to scale mainframe connectivity to the existing storage ports

• Director/switch ports are less expensive than mainframe channel cards

• NPIV can also expand the capabilities

• Can get buffer credit usage into RMF



Remote Data ReplicationRemote data replication is the process of creating replicas of information assets at remote sites (locations). Remote replicas help organizations mitigate the risks associated with regionally driven outages resulting from natural or human-made disasters. Similar to local replicas, they can also be used for other business operations. The infrastructure on which information assets are stored at the primary site is called the source. The infrastructure on which the replica is stored at the remote site is referred to as the target. Hosts that access the source or target are referred to as source hosts or target hosts, respectively.

Two basic modes of remote replications are: Synchronous and Asynchronous replication. Data has to be transferred from the source site to a target site over some network. This can be done over IP networks, over the SAN, using DWDM (Dense Wave Division Multiplexing) or SONET (Synchronous Optical Network).

The following are types of data replication technologies:

• Host based implies that all the replication is done by using the CPU resources of the host using software that is running on the host.

- Logical Volume Manager (LVM) based

- Supports both synchronous and asynchronous mode

- Log Shipping

• Storage Array based implies that all replication is done between Storage Arrays and is handled by the Array Operating Environment.

- Support both synchronous and asynchronous mode

- Disk Buffered - Consistent Point-in-Time (PIT) copies

- Combination of Local and Remote Replication

Synchronous Data ReplicationData is committed at both the source site and the target site before the write is acknowledged to the host. Any write to the source must be transmitted to and acknowledged by the target before signaling a write complete to the host. Additional writes cannot occur until each preceding write has been completed and acknowledged. It ensures that data at both sites are identical at all times.

Asynchronous Data ReplicationIn asynchronous data replication a write is committed to the source and immediately acknowledged to the host. Data is buffered at the source and transmitted to the remote site later. Data at the remote site will be behind the source by at least the size of the buffer. Hence, asynchronous remote replication provides a finite (nonzero) RPO disaster recovery solution. RPO depends on the size of the buffer, available network bandwidth, and the write workload to the source. There is no impact on application response time, as the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over extended distances. Asynchronous remote replication can be deployed over distances ranging from several hundred to several thousand kilometers between two sites.



FICON Emulation OverviewFICON emulation supports FICON traffic over IP WANs using FCIP as the underlying protocol. FICON emulation can be extended to support performance enhancements for specific applications through use of the following licensed features:

• IBM z/OS Global Mirror (formerly eXtended Remote Copy or XRC)

• FICON Tape Emulation (tape read and write pipelining)

The 4Gbps FR4-18i blade uses the following licenses:

• High-Performance Extension over FCIP/FC

Allows creation of tunnels.

• FICON Tape

Allows FICON Tape emulation.

• IBM z/OS Global Mirror

Allows FICON IBM z/OS Global Mirror emulation.

• Brocade Accelerator for FICON

Enables both FICON tape emulation and IBM z/OS Global Mirror emulation. One license must be used per unit or blade.

Bandwidth Allocation and Restrictions• Multiple FCIP tunnels are not supported between pairs of 7800 switches or FX8-24 blades when any of

the FICON emulation/acceleration features or FCP acceleration features are enabled on the tunnel, unless TI Zones or LS/LF configurations are used to provide deterministic flows between the switches.

• VE_Port 12 has two 10 Gbps circuits. Circuit 0 has a metric of 0 on xge1 and circuit 1 has a metric 1 on xge0. With this configuration, no other tunnels or circuits would be allowed on this blade because both XGE ports have 10 Gbps of configured bandwidth.

• A single VE_Port is required on each end.



3 — ImplementationAfter completing this section you should be able to perform the following:

• Describe basic FICON installation best practices

• Demonstrate the ability to implement a cascaded FICON solution in a pure Fabric OS environment

Best PracticesHere are some reasons why you should always enable FMS on your FICON switching devices. As stated, it is a best practice to also implement CUP on your FICON switching devices:

• RMF can have in-band access to the FICON switching.

• Systems Automation for z/OS, with the I/O Operations IOOPs module implemented, can have in-band access to the FICON switching devices.

• FICON Dynamic Channel Management (DCM) can dynamically add and remove channel resources at the Workload Manager’s discretion.

• When CUP is enabled, z/OS will use Coordinated Universal Time (UTC) to update the time/date on all of the CUP-enabled switching devices. z/OS keeps everything in sync and all logs are also synchronized.

• The Allow/Prohibit Addressing Matrix becomes available

• The Active=Saved condition becomes user configurable

Zoning• In a single director fabric with just FICON, it is a common practice to place all ports in a single specific

zone.

• If supporting multiple operating systems from LPARs on the same director, such as z/OS and z/TPF, create separate zones for each operating system from the LPAR.

• In intermixed environments, place the FICON ports in a specific FICON port zone. Then use standard open systems best practices for creating World Wide Name (WWN) zones for the FCP traffic on that same chassis.

• It is a bad practice to mix FICON and FCP ports within the same zone.

• In cascaded environments, paths using 2-byte addressing and their remote devices should be placed in zones that are separate from non-cascaded paths.

• Do not put the E_Ports (ISL ports) in a zone. This can have either a default zone or specific zones, but not both on the same chassis.

• Do not use the Default Zoning Mode (all access) for FICON, but rather define port zones (domain/index) for FICON and WWN zones for FCP if using protocol intermix.

• Brocade generally recommends that customers simply place all FICON devices into one large zone, which achieves the same behavior as “all access,” but provides the added protection that any new device is not allowed to talk to any other device until it is explicitly added to the zone.



• By disabling the default zone (no access) and activating port zones, you ensure that if you utilize Prohibit Dynamic Connectivity Matrix (PDCM) entries that they are honored and enforced.

• Zoning RSCN cannot be suppressed or ignored on B-Series chassis.

• FOS v7.0 and higher allows a switch using a Default Zoning Mode of (no access), and with no zoning configuration, to merge with a fabric that has an active zone configuration

Domain ID• Always make domain IDs unique and insistent. Insistent domain IDs are required for 2-byte addressing

(cascading).

• There is no need for a domain ID to ever change in a FICON environment; fabrics come up faster after recovering from a failure if the domain ID is insistent. Each switch needs its own Switch ID in the IOCP so, assuming you are making the Switch ID = Switch Address, each domain ID must be unique. Note that the switch address is based on the domain ID.

• This allows the directors to be cascaded in the future without having to take any director off line.

• Make sure that the domain offset (normally x60) is the same on all of the devices in all of your FICON fabrics.

Switch ID• Set the switch ID in the IOCP to be the same as the switch address (on Brocade DCX switches this is also

the Domain ID) of the FICON switching device.

• In Interop Mode 0 (this is the typical mode and is used for fabrics with B-type switches only) the switch address and the domain ID are the same thing.

• In Interop Mode 2 (this is the mode for intermixing M-Series and B-Series directors in the same fabric), the M6140 uses an offset of 96 (0x60). For example, in Interop Mode 2, a domain ID of 6 would have a switch address of 0x66; therefore, the switch ID should be 66. Please keep this in mind when configuring any future domain IDs on a B-Series switch.

• Note that channel statements in the IOCP are associated with a switch ID; however, the link statements, when using 2-byte addressing, are the switch address. This avoids any confusion as to which switch you are using.

Fill Words• Prior to 8 Gbps, the fill word was always Idle.

• With 8 Gbps, copper wire EMI emissions were too high; using ARBs instead of Idles helps to lower emissions. – This really had no effect on fiber cables—just copper cables.

• The FC specifications were maturing at the time that Brocade, and other vendors, implemented the fill word. The original FC-PI (Physical Interface) specification did not articulate what to do during initialization:

- As a result, different vendors implemented different solutions.

- IBM ships 8 Gbps defaulted to mode 1 (ARBs).

- There are other OEMs that ship 8 Gbps defaulted to mode 0 (Idles).



• At Brocade FOS 6.3.1 and above, Brocade offers four mode settings.

• Use the portCfgFillWord command to set the mode:

- Mode 0: Idle/Idle

- Mode 1: Arb/Arb

- Mode 2: Idle/Arb (compliant with current spec)

- Mode 3: Try mode 1 first, then try mode 2

• Example: The portcfgfillword 1/8, 0 command changes the fill word for Slot1, Port8 to Idles.

• Example: The portcfgfillword 1/8, 1 command changes the fill word for Slot1, Port8 to ARB(FF).

• HDS storage supports only mode 0 or 2. HDS recommends mode 2 for their 8 Gbps devices.

• Obviously, if a vendor asks for a specific mode, you should try that mode.

• It is important to note that the portcfgfillword command is only for 8 Gbps switching devices and is not supported on any 16 Gbps port card, regardless of the SFP installed (8 or 16 Gbps). The fill word still needs to be set on 8 Gbps products (Condor2 ASIC). This includes ports on the Brocade FX8-24 Extension Blade even if it is installed in a Brocade DCX 8510.

Traffic Isolation Zoning OverviewThe Traffic Isolation Zoning feature allows you to control the flow of interswitch traffic by creating a dedicated path for traffic flowing from a specific set of source ports (N_Ports). For example, you might use Traffic Isolation Zoning for the following scenarios:

• To dedicate an ISL to high priority, host-to-target traffic.

• To force high volume, low priority traffic onto a given ISL to limit the effect on the fabric of this high traffic pattern.

• To ensure that requests and responses of FCIP-based applications such as tape pipelining use the same VE_Port tunnel across a metaSAN.

TI Zone FailoverA TI zone can have failover enabled or disabled.

Disable failover if you want to guarantee that TI zone traffic uses only the dedicated path, and that no other traffic can use the dedicated path.

Enable failover if you want traffic to have alternate routes if either the dedicated or non-dedicated paths cannot be used.



Illegal ETIZ Configuration: Separate Paths from a Port to Devices on Same DomainFigure 41 shows two enhanced TI zones that are configured incorrectly because there are two paths from a local port (port 8 on Domain 3) to two or more devices on the same remote domain (ports 1 and 4 on Domain 1).

The TI zones are enhanced TI zones because they have an overlapping member (3,8). Each zone describes a different path from the Target to Domain 1. Traffic is routed correctly from Host 1 and Host 2 to the Target; however, traffic from the Target to the Hosts might not be.

Traffic from (3,8) destined for Domain 1 cannot go through both port 6 and port 7, so only one port is chosen. If port 6 is chosen, frames destined for (1,4) will be dropped at Domain 1. If port 7 is chosen, frames destined for (1,1) will be dropped.

FIGURE 14 Illegal ETIZ configuration: two paths from one port to two devices on the same remote domain

256-area Addressing ModeThis configurable addressing mode is available only in a logical switch on the Brocade DCX, DCX-4S, and Brocade DCX 8510 family enterprise-class platforms. In this mode, only 256 ports are supported and each port receives a unique 8-bit area address. This mode can be used in FICON environments, which have strict requirements for 8-bit area FC addresses.

There are two types of area assignment modes in the 256-area addressing mode: zero-based and port-based.

• Zero-based mode, which assigns areas as ports, are added to the partition, beginning at area 0x00. This mode allows FICON customers to make use of the upper ports of a 48-port or 64-port blade. Zero-based mode is also supported on the default switch.

• Port-based mode does not support the upper 16 ports of a 48-port or 64-port blade in a logical switch. Port-based mode is not supported on the default switch.



Port SwappingIf a port malfunctions, or if you want to connect to different devices without having to re-wire your infrastructure, you can move a port’s traffic to another port (swap ports) without changing the I/O Configuration Data Set (IOCDS) on the mainframe computer.

The following are port swapping considerations:

• Ports with area OxFE or OxFF addresses cannot be swapped when FMS mode is enabled.

• GbE ports cannot be swapped

• Shared area ports cannot be swapped.

• Ports that are part of a trunk group cannot be swapped.

• Swapping ports between different logical switches is not supported. The ports on the source

• and destination blades need to be in the same logical switch.

Geographically Dispersed Parallel SysplexGeographically Dispersed Parallel Sysplex (GDPS) Manages System z application and data availability at both local and remote sites. It has the following capabilities:

• Monitors systems, disk and tape subsystems

• Manages planned and unplanned activities

• System/disk maintenance/failure

• Site maintenance/failure

• Clustering

• Remote copy (disk and tape)

• Automation

• Easy to use interface

• Intuitive dashboard

• Policy based commands

• Established and experienced

• Over 10 years of enhancement

• Implementation success

• Supported worldwide

There are different vehicles that constitutes the GDPS service such as HyperSwap Manager, the manager for Metro Global Mirroring, as well as z/OS Global Mirror, formerly known as XRC. These are supported as well as two and three site solutions.



Troubleshooting Trunking The trunkDebug command displays the possible reason that two ports cannot be trunked. Possible reasons are:

• The switch does not support trunking.

• A trunking license is required.

• Trunking is not supported in switch interoperability mode.

• Port trunking is disabled.

• The port is not an E_Port.

• The port is not 2 Gbps, 4 Gbps, or 8 Gbps.

• The port connects to a switch other than the one you want it to.

To correct this issue, connect additional ISLs to the switch with which you want to communicate.

• • The ports are not the same speed or they are not set to an invalid speed.

Manually set port speeds to a speed supported on both sides of the trunk.

• The ports are not set to the same long distance mode.

Set the long distance mode to the same setting on all ports on both sides of the trunk.

• Local or remote ports are not in the same port group.

Move all ISLs to the same port group. The port groups begin at port 0 and are in groups of 4 or 8, depending on the switch model. Until this is done, the ISLs do not trunk.

• The difference in the cable length among trunked links is greater than the allowed difference.

Native Connectivity (M-EOS interoperability)A switch running FOS v7.0 cannot form E-port connectivity with any M-EOS platform. A switch running FOS v7.0 can only operate in Brocade native mode (interopmode 0). Connectivity between M-EOS platforms and a switch running FOS v7.0 is supported via FCR.

ISL DecommissionThe Port Decommission feature provides users with the ability to non-disruptively remove an ISL from service.

When an ISL is selected for decommissioning, the switches communicate with each other to coordinate the movement of flows off of the ISL to alternative paths. Once the flows are moved to alternative paths, the switches block the E_Ports associated with the ISL being decommissioned to complete the decommission process.

At FOS 6.4.0a + (2010), DLS can and should always be utilized with FICON. Lossless DLS pauses ingress ports before moving traffic routes. This capability provides customers with the ability to enable DLS for FICON.

The term “Lossless” describes a behavior that FOS uses when moving I/O traffic between the parallel ISL link routes.



4 — ManagementAfter completing this section you should be able to perform the following:

• Demonstrate knowledge of how to use Brocade tools to manage FICON solutions

• Describe the benefits and uses of FICON CUP

• Demonstrate knowledge of firmware and configuration management tools

Command Line Tools

Diagnostic PortFabric OS v7.0.0 allows you to convert a fibre channel port, including ISLs and loopback ports, into a Diagnostic Port (D_Port). This port lets you isolate the inter-switch link (ISL) to diagnose link level faults. The D_Port does not carry any fabric traffic, and is designated to run only specific diagnostics tests on it. The creation of a D_Port is subject to Virtual Fabric restrictions that may be in place. The ports must be 10G or 16G Brocade-branded SFPs on a Brocade DCX 8510, and running Fabric OS v7.0.0 or later.

Pathinfo

The pathinfo command displays routing and statistical information from a source port index on the local switch to a destination port index on another switch. This routing information describes the full path that a data stream travels between these ports, including all intermediate switches.

Blade SwappingBlade-based port swap is mainly used for FICON and is only applicable for port blades. However, the Management application does not block blade-based port swap for other application blades, including the 8 Gbps 24-port blade.

You can swap all of the ports from one blade to another blade. During this operation all ports in the selected blades are swapped. This operation disrupts the traffic on all ports for the selected blades. If GbE ports are present on the blade, only the non-GbE ports are swapped. To swap blades, you must meet the following requirements:

• The chassis must be running Fabric OS 6.3 or later.

• The chassis must have at least two blades of same type present.

• To perform the Swap Blades function you must have Read and Write access for the Product Administration privilege.



Web Tools - Performance Monitoring• Web Tools predefines basic graph types, to simplify performance monitoring. A wide range of end-to-end

fabric, LUN, device, and port metrics graphs are included.

• The advanced monitoring graphs give more detailed performance information to help you manage your fabric

• You can access the basic monitoring graphs on all switches

• Advanced Monitoring graphs are available only on switches that have a Brocade Advanced Performance Monitoring license activated

• Graphs:

- Port Throughput

The performance of a port, in bytes per second, for frames received and transmitted

- Switch Aggregate Throughput

The aggregate performance of all ports on a switch

- Blade Aggregate Throughput

The aggregate performance of all ports on a port card. This graph is available only for the Brocade 48000 and Brocade DCX Backbone.

- Switch Throughput Utilization

The port throughput, in Gbps, at the time the sample is taken

- Port Error

CRC errors for a given port

- Switch Percent Utilization

The percentage utilization for each port in a switch

- Port Snapshot Error

The CRC error count between sampling periods for all the ports on a switch

SAN Health

IOCP• SAN Health is compatible with FICON switches and can be used to review IOCP data with the SAN

environment.

• The IOCP file contains the path configuration statements for a given mainframe system.

- SAN Health will match these path statements with the physical switch ports found in the SAN Health audit

• While SAN Health is run, all interaction with the switches is displayed and can be stopped at any time.

• The diagnostics capture completes in approximately two minutes.



• SAN Health will remain open and capture performance data for the duration that you have set the performance capture

FIGURE 15 SAN Health Output with IOCP Data

To the standard SAN Health Report and Visio diagram, Brocade adds FICON information including the Serial/Sequence number. This information appears anywhere where there is port specific information in the report. AN EXcel spreadsheet is also produced with CHPIDs and CNTLUNITs.

Types of BottlenecksThe bottleneck detection feature detects two types of bottlenecks:

• Latency bottleneck

• Congestion bottleneck

A latency bottleneck is a port where the offered load exceeds the rate at which the other end of the link can continuously accept traffic, but does not exceed the physical capacity of the link. This condition can be caused by a device attached to the fabric that is slow to process received frames and send back credit returns. A latency bottleneck due to such a device can spread through the fabric and can slow down unrelated flows that share links with the slow flow.

By default, bottleneck detection detects latency bottlenecks that are severe enough that they cause 98% loss of throughput. This default value can be modified to a different percentage.

A congestion bottleneck is a port that is unable to transmit frames at the offered rate because the offered rate is greater than the physical data rate of the line. For example, this condition can be caused by trying to transfer data at 8 Gbps over a 4 Gbps ISL.

You can use the bottleneckMon command to configure alert thresholds for congestion and latency bottlenecks.



FICON CUP Zoning and PDCM ConsiderationsThe FICON PDCMs control whether or not communication between a pairs ports in the switch is prohibited or allowed. PDCMs are a per-port construct. Each port on the switch has its own PDCM that defines whether communication is allowed between that particular port, and each of the other ports in the switch, including itself. The Allow/Prohibit Matrix presents a matrix that allows you to set and clear the PDCMs for all external ports. If there are any differences between restrictions set up with Brocade Advanced Zoning and PDCMs, the most restrictive rules are automatically applied.

FMS must be enabled for the Allow/Prohibit matrix to be operational, and there must be an active zoning configuration. (D,I) zoning is recommended for FICON, but is not specifically required.

Fabric Watch OverviewFabric Watch is an optional storage area network (SAN) health monitor that allows you to enable each switch to constantly monitor its SAN fabric for potential faults and automatically alerts you to problems long before they become costly failures.

Reasons to Customize Fabric Watch SettingsCustomization is recommended to achieve the following objectives:

• Selecting one or more event settings

• Selecting an appropriate message delivery method for critical and noncritical events

• Selecting appropriate thresholds and alarm levels relevant to each class element

• Defining the appropriate Time Base event triggering based on the class element traits

• Eliminating message delivery that has little or no practical value to the SAN administrator

• Consolidating multiple messages generated from a single event

Fabric Monitoring Setting GuidelinesIt is recommended that you leave the entire Fabric class in its default state (no alerts) for the following reasons:

• Domain ID changes

Plan and use strict change control practices to avoid Domain ID changes.

• Loss of E_Port

Detect if an E_Port is down using the E_Port class areas.

• Fabric logins

In a large environment of numerous devices, this area is of no interest.

• Fabric reconfiguration



Fabric reconfigurations typically occur when new switches are added to a fabric, which is a planned activity, or when an upstream or downstream ISL fails, which is detected through the E_Port class areas. Since fabric reconfigurations are monitored elsewhere, don’t change the default settings for the Fabric class.

• Segmentation changes

Segmentations only occur in the event of an entire switch failure. In this rare case, you can gather multiple reports from all the attached E_Ports of the link failures.

• Zoning changes

Zone changes are captured through the Audit facility in Fabric OS. All zone changes can be configured to be recorded in the RASlog, which is the recommended practice.

Use the portcfgautodisable command to enable or disable the autodisable feature for a specified port or a range of ports and to manage the configuration.

The port autodisable feature minimizes traffic disruption introduced in some instances of automatic port recovery. When the autodisable flag is set, you can specify the conditions that will prevent the port to reinitialize. Such conditions include loss of sync, loss of signal, OLS, NOS, and LIP.

Advanced Performance Monitoring Overview• Based on Brocade Frame Filtering technology and a unique performance counter engine

• Comprehensive tool for monitoring the performance of networked storage resources

• Advanced Performance Monitoring provides the following monitors

• End-to-end monitors measure the traffic between a host/target pair

• Filter-based monitors measure the traffic transmitted through a port with specific values in the first 64 bytes of the frame

• ISL monitors measure the traffic transmitted through an InterSwitch Link (ISL) to different destination domains

• Top Talker monitors measure the flows that are major consumers of bandwidth on a port

• The Top Talker (TT) feature is an enhancement to APM end-to-end monitors

- First available in Fabric OS v6.0

- When enabled, these monitors determine which SID-DID pairs are the major users of switch F_Port bandwidth

- Can be enabled on specific switch F_Ports on any switch in the fabric

RMF 74-7 Records• Enabling RMF 74 subtype 7 (RMF 74-7) records yields an RMF report called the “FICON Director Activity

Report”



• Data is collected for each RMF interval if FCD is specified in the ERBRMFnn parmlib member. The report captures information based on an interval which is set when you create the report.

- The report captures a snapshot of data and the counters based on an time interval, such as 60 seconds. Often, you need to run these reports more than once and change the interval periods for troubleshooting to determine if there is a trend.

- FMS must be enabled on the switch

- IECIOSxx dataset SYS1.PARMLIB must specify STATS=YES

• This RMF report is often overlooked but contains very meaningful data concerning FICON I/O performance - in particular, frame pacing delay

• Frame pacing delay is the only available method to indicate a BB_Credit starvation issue on a given port

FIGURE 16 FICON Director Activity Report



RMF FICON Director Activity Report

FIGURE 17 FICON Director Activity Report for an M6140

• The two values contained in the report that are most often used for troubleshooting are:

- AVG FRAME PACING refers to frame pacing delay. If it is a non-zero number then it reflects the number of times that I/O was delayed for 2.5 microseconds or longer due to buffer credits falling to zero.

- ERROR COUNT which is the number of errors which were encountered during the interval.

• The AVG FRAME PACING column is where frame pacing delay is reported.

• Even if the number is zero there could be a buffer credit shortage problem:

- Switching device is not fast enough to catch all errors (M6140, Mi10K, etc.)

- Buffer credits went to zero but delays to send the next frame were less than 2.5 microseconds

• Sampling times can be an issue



IFCCs• The System z FICON channel’s approach to capturing error information at the time of a FICON I/O

operation failure is to detect an Interface Control Check (IFCC). The FICON purge-path-extended function requests, collects, and transfers link error statistical buffer information from each Fibre Channel port in the FICON channel-to-CU path to the System z host (MVS) console.

• The FICON purge-path-extended function is used by the mainframe to issue a query to determine the end-to-end path information

• Example of an IFCC Error Message: IOS051I

- 2027-20:17:04 (MVS 20:16:54) IOS051I INTERFACE TIMEOUT DETECTED ON D829,D0,E7,**02,PCHID=0520

- A possible MIHPTO issue

• The message help provides this:

- IOS051I INTERFACE TIMEOUT DETECTED ON dev, chp, cmd, stat

- Explanation: The channel subsystem detected a timeout condition during the operation of device. This message is issued as an informational message only for the interface timeouts that occur on native FICON channels

- In the message text:

• dev (D829)is the device number.

• chp (D0) are the channel path identifiers (CHPID), if known, otherwise, this field is set to asterisks

• cmd(E7) is the failing command code, if known; otherwise, this field is set to asterisks

• stat(**02) is the device and subchannel status, if known; otherwise, this field is set to asterisks. In the example, the **02 indicates an IFCC which indicates a problem with the device.

• PCHID=0520 is the physical channel path identifier

Saving or Copying Allow/Prohibit Matrix Configurations to Another DeviceWhen copying or saving a configuration from a small switch (source switch with fewer ports; for example, 64 ports) to a larger switch (destination switch with a larger number of ports; for example, 256 ports) only the port address range of the smaller switch will be affected on the larger switch. All additional port addresses will display the default settings (port state defaults to ‘Allow’ and the Blocked check box defaults to not checked).

Copying or saving a configuration from a larger switch to a smaller device only copies or saves the port address range that matches the smaller switch. Additionally a message displays that the additional port addresses from the larger switch are discarded.

When copying or saving a configuration from or to Logical Switches, the only ports affected are the port addresses defined in the Logical Switch. The FICON CUP daemon retains the full compliment of records regardless of the size of the Logical Switch. Therefore, copying or saving a configuration from or to logical switches should work the same as copying or saving between standard switches.



Cascaded FICON fabric

NOTEYou must have FICON Management privileges to configure a fabric for cascaded FICON.

The FICON Configuration Management application in Brocade Network Advisor enables you to easily configure a fabric for cascaded FICON. Note that configuring a fabric for cascaded FICON may be disruptive to current I/O operations in the fabric, as it needs to disable and enable the switches in the fabric.

FICON configuration performs the following operations on the selected fabric:

• Turns on the insistent domain ID flag (IDID) on all switches.

• Sets High Integrity Fabric Configuration (HIFC) on the seed switch.

NOTEHigh Integrity Fabric (HIF) activates SCC policy, sets Insistent Domain ID and sets Fabric Wide Consistency Policy for SCC in tolerant mode.

- Fabric-wide consistency policy (FWCP) is configured to include SCC in strict mode.

- SCC policy is created or modified to limit connectivity to only the switches in the selected fabric.

• Enables port-based routing on all switches.

• Enables In-Order Delivery (IOD) on all switches.

• Enables Dynamic Load Sharing (DLS) based on user selection and the firmware level.

NOTETo enable DLS, all switches in the fabric must be 8 Gbps or faster and running Fabric OS 6.4 or later.

• (Optional) Turns on FICON Management Server (FMS) mode on all switches.

- If switches are running Fabric OS 7.0 and later, FMS will not be enabled unless the switches have an active CUP license.

- If switches are running Fabric OS earlier than 7.0 and not have a CUP license, after successful configuration, you can access the Port Connectivity (Allow/Prohibit) matrix, but the host system cannot communicate with the FICON Management Server unless you install a CUP license. If a CUP license is later installed on these switches, then FMS mode must be re-enabled on these switches.

Firmware DownloadsThe CUP device must be varied offline to all MVS partitions before starting a code load. The CUP device can be varied back online after the code load completes. Failure to vary off the CUP devices may result in missing interrupt.



FCIPA disabled XGE or GE port will be re-enabled after a code upgrade/downgrade. To prevent XGE/GE ports from being re-enabled after a code load they should be persistently disabled.



5 — MaintenanceAfter completing this section you should be able to demonstrate knowledge of tools used for problem determination.

Top Talker MonitorsTop Talker monitors determine the flows (SID/DID pairs) that are the major users of bandwidth (after initial stabilization). Top Talker monitors measure bandwidth usage data in real-time and relative to the port on which the monitor is installed.

NOTETop Talkers is a feature under the Advanced Performance Monitoring license.

Applications can use the Top Talker data to do the following:

• Re-route the traffic through different ports that are less busy, so as not to overload a given port.

• Alert you to the top-talking flows on a port if the total traffic on the port exceeds the acceptable bandwidth consumption.

You can use Top Talkers to identify the SID/DID pairs that consume the most bandwidth and can then configure them with certain Quality of Service (QoS) attributes so they get proper priority.

General Information to Gather for all CasesThe following information needs to be gathered for all FICON setups:

• The output from the standard support commands (portLogDump, supportSave, supportShow) the Fabric Manager Event Log, EFCM or Brocade Network Advisor logs.

By default, the FICON group in the supportShow output is disabled. To enable the capture of FICON data in the supportShow output, enter the supportshowcfgenable ficon command. After you get confirmation that the configuration has been updated, the following will be collected and appear in the output for the supportShow command:

- ficonCupShow fmsmode

- ficonCupShow modereg

- ficonDbg dump rnid

- ficonDbg log

- ficonShow lirr

- ficonShow rlir

- ficonShow rnid

- ficonShow switchrnid

- ficuCmd dump -A

• Type of mainframe involved. Need make, model, and driver levels in use.



• Type of actual storage array installed. Many arrays will emulate a certain type of IBM array and we need to know the exact make, model, and firmware of the array in use.

• Other detailed information for protocol-specific problems:

- Port data structures, displayed using the ptdatashow command.

- Port registers, displayed using the ptregshow command.

• Read Brocade Release Notes for specific version information regarding the Fabric OS installed.



Taking the TestAfter the Introduction Screen, once you click on Next, you will see the following non-disclosure agreement:

IMPORTANT: PLEASE READ THE FOLLOWING BROCADE NON-DISCLOSURE CONFIDENTIALITY AGREEMENT CAREFULLY BEFORE TAKING THIS EXAM.

The following Non-Disclosure Confidentiality Agreement (the “Agreement”) sets forth the terms and conditions of your use of the exam materials as defined below.

The Disclosure to you of this Exam and any questions, answers, worksheets, computations, drawings, diagrams, or any communications, including verbal communication by any party, regarding or related to the Exam and such Exam Materials and any derivatives thereof is subject to the Terms and Conditions of this Agreement.

You understand, acknowledge and agree:

• That the questions and answers of the Exam are the exclusive and confidential property of Brocade and are protected by Brocade intellectual property rights;

• That you may not disclose the Exam questions or answers or discuss any of the content of the Exam Materials with any person, without prior approval from Brocade;

• Not to copy or attempt to make copies (written, photocopied, or otherwise) of any Exam Material, including, without limitation, any Exam questions or answers;

• Not to sell, license, distribute, or give away the Exam Materials, questions, or answers;

• You have not purchased, solicited or used unauthorized (non-Brocade sanctioned) Exam Materials, questions, or answers in preparation for this exam;

• That your obligations under this Agreement shall continue in effect after the Exam and, if applicable, after termination of your credential, regardless of the reason or reasons for terminations, and whether such termination is voluntary or involuntary.

Brocade reserves the right to take all appropriate actions to remedy or prevent disclosure or misuse, including, without limitation, obtaining an immediate injunction. Brocade reserves the right to validate all results and take any appropriate actions as needed. Brocade also reserves the right to use any technologies and methods for verifying the identity of candidates. Such technology may include, without limitation, personally identifiable information, challenge questions, identification numbers, photographic information, and other measures to protect against fraud and abuse.

Neither this Agreement nor any right granted hereunder shall be assignable or otherwise transferable by you.

By clicking on the "A" button (“YES, I AGREE”), you are consenting to be bound by the terms and conditions of this agreement and state that you have read this agreement carefully and you understand and accept the obligations which it imposes without reservation. You further state that no promises or representations have been made to induce agreement and that you accept this agreement voluntarily and freely.

A. YES, I AGREE

B. NO, I DO NOT AGREE




Documents

Bcaf Nutshell