Presented by: Quinn GaumerCPS 221

16,384 Processing Nodes (32 MHz)

30 m x 30 m Teraflop 1992

With 16,384 processors the interconnect plays a large role

3 Types of Networks◦ Data◦ Control◦ Diagnostic

Easily Attainable High Performance Scaling Data Parallel Programming High Reliability and Availability Space/Time Shared Fast Time to Market Modular

Include Control Processor Processing Nodes Slices of Data and Control Networks

◦ Privileged vs. Non-Privileged Program Isolation Time Sharing

Provide Simple View of Network to Processors

Sharing and Fault Tolerance Decouple Network/Processor by Providing

Contract◦ Software -> ISA -> Hardware

“The data network promises to eventually accept and deliver all messages injected into the network by the processors as long as the processors promise to eventually eject all messages from the network when they are delivered to the processors. ”

Collection of Memory Mapped FIFOs◦ Outgoing/Ingoing

Restricted Operations◦ Implemented with protected pages

Physical/Relative(Virtual) Address◦ Programs use only relative addresses

Network Independent of User◦ Delivery guaranteed by network not processing

node◦ Requires network diagnostics

Fat Tree Structure◦ Closer to the root, thicker the

tree◦ Ensures no bottlenecks at root

User Partitions and I/O are Sub-trees◦ Guarantees network

independence◦ Messages in partition stay within

partition Many Optimal Node to Node

Paths◦ Choose randomly among open

links

Data can be only 1-5 Words Wormhole Routing CRC Checking done at every Link

◦ Additional !CRC sent when error first found Primary Errors allow Diagnostic Network to

Determine location

Message Counters at every Link Kirchoff’s Law to Determine Missing

Messages What to do with a Bad Chip or Link?

◦ Route Messages Away from Failure◦ Map Out Nearby Processors◦ Which is better?

Both.

Solution: Virtual Channels◦ One channel for request and response◦ 4 channels per chip (Incoming and Outgoing)

Deadlock still possible!◦ User sends but never attempts to receive

messages◦ Higher level languages to implement

communication protocol

Objectives◦ Clear all messages for new

user◦ Allow all messages in transit to

eventually finish “All Fall Down” Method

◦ Evenly misroute all messages in transit to nodes

◦ Message saved at node◦ Resent when swapped in

Control Processor broadcasts program◦ Not instructions(SIMD)

Each Processor runs program on data set Inter-Processor Communication

◦ Hardware Barriers allow for processes to communicate without shared semaphores

Program smaller than instructions◦ Easier to deliver

Local fetch allows commodity processors◦ Fast new RISC processors, less R & D.

Control system useful for other problems Execution of generic MIMD code

◦ Message passing

Broadcasting◦ User/Supervisor◦ Interrupt◦ Utility

Combining◦ Reduction◦ Forward/Backward Scan◦ Router Done

Global Operations◦ Synchronous/Asynchronous OR

Binary Tree Four Types of Packets

◦ Single Source : Broadcasting◦ Multiple Source: Combining◦ Idle: Filler◦ Abstain: Allow control node to skip waiting

Collisions on Network◦ Multiple/Multiple: Buffering based on arrival time◦ Multiple/Single: Single Source Packets Prioritized◦ Single/Single: Error

Control Processor for each Partition◦ Executes scalar code while processing nodes

execute parallel code Connect any Control Processor to any

Partition◦ Problems can occur in control networks too◦ Diagnostics may show part of control network

must be mapped out

Binary Network◦ Pods(physical subsystem)

are leaves JTAG

◦ Designed for Multichip…but serial

Do JTAG for each Pod Combine Responses with

OR/AND