Chapters 8 Storage, Networks, and Other Peripherals

12004 Morgan Kaufmann Publishers

Chapters 8

Storage, Networks, and Other Peripherals



Outline

• 8.1 Introduction

• 8.2 Disk Storage and Dependability

• 8.3 Networks

• 8.4 Buses and Other Connections between Processors, Memory, and I/O Devices

• 8.5 Interfacing I/O Devices to the Processor, Memory, and Operating System

• 8.6 I/O Performance Measures: Examples from Disk and File Systems

• 8.7 Designing an I/O System

• 8.8 Real Stuff: A Digital Camera

• 8.9 Fallacies and Pitfalls

• 8.10 Concluding Remarks

• 8.11 Historical Perspective and Further Reading


8.1 Introduction


Keywords

• I/O requests Reads or writes to I/O devices


Interfacing Processors and Peripherals

• I/O Design affected by many factors (expandability, resilience)

• Performance:— access latency — throughput— connection between devices and the system— the memory hierarchy— the operating system

• A variety of different users (e.g., banks, supercomputers, engineers)

Disk Disk

Processor

Cache

Memory- I/O bus

Mainmemory

I/Ocontroller

I/Ocontroller

I/Ocontroller

Graphicsoutput

Network

Interrupts


I/O

• Important but neglected

“The difficulties in assessing and designing I/O systems haveoften relegated I/O to second class status”

“courses in every aspect of computing, from programming tocomputer architecture often ignore I/O or give it scanty coverage”

“textbooks leave the subject to near the end, making it easierfor students and instructors to skip it!”

• GUILTY!

— we won’t be looking at I/O in much detail

— be sure and read Chapter 8 in its entirety.

— you should probably take a networking class!


I/O Devices

• Very diverse devices— behavior (i.e., input vs. output)— partner (who is at the other end?)— data rate


Figure 8.2 The diversity of I/O devices.

Device Behavior Partner Data rate (Mbit/sec)

Keyboard Input Human 0.0001

Mouse Input Human 0.0038

Voice input Input Human 0.2640

Sound input Input machine 3.0000

Scanner Input Human 3.2000

Voice output Output Human 0.2640

Sound output Output Human 8.0000

Laser printer Output Human 3.2000

Graphics display Output Human 800.0000-8000.0000

Modem Input or output Machine 0.0160-0.0640

Network/LAN Input or output Machine 100.0000-1000.0000

Network/wireless LAN Input or output Machine 11.0000-54.0000

Optical disk Storage Machine 80.0000

Magnetic tape Storage Machine 32.0000

Magnetic disk storage machine 240.0000-2560.0000


8.2 Disk Storage and Dependability


Keywords

• Nonvolatile Storage device where data retains its value even when power is removed.

• Track One of thousands of concentric circles that makes up the surface of a magnetic disk.

• Sector One of the segments that make up a track on a magnetic disk; a sector is the smallest amount of information that is read or written on a disk.

• Seek The process of positioning a read/write head over the proper track on a disk.

• Rotation latency Also called delay. The time required for the desired sector of a disk to rotate under the read/write head; usually assumed to be half the rotation time.

• Small computer systems interface (SCSI) A bus used as a standard for I/O devices.


Keywords

• Redundant arrays of inexpensive disks (RAID) An organization of disks that uses an array of small and inexpensive disks so as to increase both performance and reliability.

• Striping Allocation of logically sequential blocks to separate disk to allow higher performance than a single disk can deliver.

• Mirroring Writing the identical data to multiple disks to increase data availability.

• Protection group The group of data disks or blocks that share a common check disk or block.

• Hot swapping Replacing a hardware component while the system is running.

• Standby spares Reserve hardware resources that can immediately take the place of a failed component.


I/O Example: Disk Drives

• To access data:— seek: position head over the proper track (3 to 14 ms. avg.)— rotational latency: wait for desired sector (.5 / RPM)— transfer: grab the data (one or more sectors) 30 to 80 MB/sec

Platter

Track

Platters

Sectors

Tracks


Disk Read Time

• Q ： What is the average time to read or write a 512-byte sector for a typical disk rotating at 10,000 RPM? The advertised average seek time is 6 ms, the transfer rate is 50 MB/sec, and the controller overhead is 0.2 ms. Assume that the disk is idle so that there is no waiting time.

• A：

ms2.92.001.00.30.6ms2.0MB/sec 50

KB 5.0

RPM 000,10

rotation 5.0ms0.6

If the measured average seek time is 25% of the advertised average time. The answer is

1.5ms + 3.0ms + 0.01ms + 0.2ms = 4.7ms


Figure 8.3 Six magnetic disks, varying in diameter from 14 inches down to 1.8 inches.


Figure 8.4 Characteristics of three magnetic disks by a single manufacturer in 2004.

Characteristics Seagate ST373453 Seagate ST3200822

Seagate ST94811A

Disk diameter (inches) 3.50 3.50 2.50

Formatted data capacity (GB) 73.4 200.0 40.0

Number of disk surfaces (heads) 8 4 2

Rotation speed (PRM) 15,000 7200 5400

Internal disk cache size (MB) 8 8 8

External interface, bandwidth (MB/sec)

Ultra320 SCSI, 320 Serial ATA, 150 ATA, 100

Sustained transfer rate (MB/sec) 57-86 32-58 34

Minimum seek (read/write) (ms) 0.2/0.4 1.0/1.2 1.5/2.0

Average seek read/write (ms) 3.6/3.9 8.5/9.5 12.0/14.0


Mean time to failure (MTTF) (hours) 1,200,000@25℃ 600,000@25℃ 330,000@25℃

Warranty (years) 5 3 —

Nonrecoverable read errors per bits read <1 per <1 per <1 per

Temperature, vibration limits (operating) 5-55 , ℃[email protected]

0-60 , ℃[email protected]

5-55 , ℃400Hz@1G

Size: dimensions (in.), weight (pounds) 1.0”X4.0”X5.8”, 1.9 lbs

1.0”X4.0”X5.8”, 1.4 lbs

0.4”X2.7”X3.9”, 0.2 lbs

Power: operating/idle/standby (watts) 20?/12/ — 12/8/1 2.4/1.0/0.4

GB/cu. In., GB/watt 3GB/cu.in., 4GB/W 9GB/cu.in., 16GB/W 10GB/cu.in., 17GB/W

Price in 2004, $/GB $400,

$5/GB

$100,

$0.5/GB

$100,

$2.50/GB

1510 1410 1410


Figure 8.5 Summary of studies of reasons for failures.

Operator Software Hardware System Year data collected

42% 25% 18% Data center (Tandem) 1985

15% 55% 14% Data center (Tandem) 1989

18% 44% 39% Data center (DEC VAX) 1985

50% 20% 30% Data center (DEC VAX) 1993

50% 14% 19% U.S. public telephone network 1996

54% 7% 30% U.S. public telephone network 2000

60% 25% 15% Internet services 2002


Figure 8.6 RAID for an example of four data disks showing extra check disks per RAID level and companies that use each level.


Figure 8.7 Small write update on RAID 3 versus RAID4


Figure 8.8 Block-interleaved parity (RAID 4) versus distributed block-interleaved parity (RAID 5)


8.3 Networks


Networks are growing in popularity over time, and unlike other I/O devices, there are many books and courses on them. For readers who have not taken courses or read books on networking, Section 8.3 on the CD gives a quick overview of the topics and terminology, including internetworking, the OSI model, protocol families such as TCP/IP, long-haul networks such as ATM, local area networks such as Ethernet, and wireless networks such as IEEE 802.11.


8.4 Buses and Other Connections between Processors, Memory, and I/O Devices


Keywords

• Bus transaction A sequence of bus operations that includes a request and may includes a request and may include a response, either of which may carry data. A transaction is initiated by a single request and may take many individual bus operations.

• Processor-memory bus A bus that connects processor and memory and that is short, generally high speed, and matched to the memory system so as to maximize memory-processor bandwidth.

• Backplane bus A bus that is designed to allow processors, memory. And I/O devices to coexist on a single bus.

• Synchronous bus A bus that includes a clock in the control lines and a fixed protocol for communicating that is relative to the clock.

• Asynchronous bus A bus that uses a handshaking protocol for coordinating usage rather than a clock; can accommodate a wide variety of devices of differing speeds.


Keywords

• Handshaking protocol A series of steps used to coordinate asynchronous bus transfers in which the sender and receiver proceed to the next step only when both parties agree that the current step has been completed.

• Split transaction protocol A protocol in which the bus is released during a bus transaction while the requester is waiting for the data to be transmitted, which frees the bus for access by another requester.


I/O Example: Buses

• Shared communication link (one or more wires)• Difficult design:

— may be bottleneck— length of the bus— number of devices— tradeoffs (buffers for higher bandwidth increases latency)— support for many different devices— cost

• Types of buses:— processor-memory (short high speed, custom design)— backplane (high speed, often standardized, e.g., PCI)— I/O (lengthy, different devices, e.g., USB, Firewire)

• Synchronous vs. Asynchronous— use a clock and a synchronous protocol, fast and small

but every device must operate at same rate andclock skew requires the bus to be short

— don’t use a clock and instead use handshaking


I/O Bus Standards

• Today we have two dominant bus standards:


Figure 8.10 The asynchronous handshaking protocol consists of seven steps to read a word from memory and receive it in an I/O device.



Figure 8.11 Organization of the I/O system on a Pentium 4 PC using the intel 875 chip

Parallel ATA(100 MB/sec)

Parallel ATA(100 MB/sec)

(20 MB/sec)

PCI bus(132 MB/sec)

CSA(0.266 GB/sec)

AGP 8X(2.1 GB/sec)

Serial ATA(150 MB/sec)

Disk

Pentium 4processor

1 Gbit Ethernet

Memorycontroller

hub(north bridge)

82875P

MainmemoryDIMMs

DDR 400(3.2 GB/sec)

DDR 400(3.2 GB/sec)

Serial ATA(150 MB/sec)

Disk

AC/97(1 MB/sec)

Stereo(surround-

sound) USB 2.0(60 MB/sec)

. . .

I/Ocontroller

hub(south bridge)

82801EB

Graphicsoutput

(266 MB/sec)

System bus (800 MHz, 604 GB/sec)

CD/DVD

Tape

10/100 Mbit Ethernet


Figure 8.12 Two Pentium 4 I/O chip sets from Intel.


8.5 Interfacing I/O Devices to the Processor, Memory, and Operating System


Keywords

• Memory-mapped I/O An I/O scheme in which portions of address space are assigned to I/O devices and reads and writes to those addresses are interpreted as commands to the I/O device.

• I/O instructions A dedicated instruction that is used to give a command to an I/O device and that specifies both the device number and the command word (or the location of the command word in memory).

• Polling The process of periodically checking the status of an I/O device to determine the need to service the device.

• Interrupted-driven I/O An I/O scheme that employs interrupts to indicate to the processor that an I/O device needs attention.

• Direct memory access (DMA) A mechanism that provides a device controller the ability to transfer data directly to or from the memory without involving the processor.

• Bus master A unit on the bus that can initiate bus requests.


Other important issues

• Bus Arbitration:

— daisy chain arbitration (not very fair)

— centralized arbitration (requires an arbiter), e.g., PCI

— collision detection, e.g., Ethernet

• Operating system:

— polling

— interrupts

— direct memory access (DMA)

• Performance Analysis techniques:

— queuing theory

— simulation

— analysis, i.e., find the weakest link (see “I/O System Design”)

• Many new developments


Figure 8.13 The Cause and Status registers.


8.6 I/O Performance Measures ： Examples from Disk and File Systems


Keywords

• Transaction processing A type of application that involves handling small short operations (called transactions) that typically require both I/O and computation. Transaction processing applications typically have both response time requirements and a performance measurement based on the throughput of transactions.

• I/O rate Performance measure of I/Os per unit time. Such as reads per second.

• Data rate Performance measure of bytes per unit time, such as GB/second.


Impact of I/O on system performance

• Q ： Suppose we have a benchmark that executes in 100 seconds of elapsed time, where 90 seconds is CPU time and the rest is I/O time. If CPU time improves by 50% per year for the next five years but I/O time doesn’t improve, how much faster will our program run at the end of five years?

A：• We know that

Elapsed time = CPU time + I/O time

100 = 90 + I/O time

I/O time = 10 seconds

The new CPU times and the resulting elapsed times are computed in the following table：

<next page>


After n years CPU time I/O time Elapsed time % I/O time

0 90 seconds 10 seconds 100 seconds 10 %






5.1

90

5.1

60

5.1

40

5.1

27

5.1

18

The improvement in CPU performance over five years is

However, the improvement in elapsed time is only

and the I/O time has increased from 10% to 45% of the elapsed time.

5.712

90

5.422

100


8.7 Designing an I/O System


I/O System Design

• Q： Consider the following computer system：– A CPU that sustain 3 billion instructions per second and averages

100,000 instruction in the operating system per I/O operation

– A memory backplane bus capable of sustaining a transfer rate of 1000 MB/sec

– SCSI Ultra320 controllers with a transfer rate of 320 MB/sec and accommodating up to 7 disks

– Disk drives with a read/write bandwidth of 75 MB/sec and an average seek plus rotational latency of 6 ms.

If the workload consists of 64 KB reads (where the block is sequential on a track) and the user program needs 200,000 instructions per I/O operation, find the maximum sustainable I/O rate and the number of disks and SCSI controllers required. Assume that the reads can always be done on an idle disk if one exists (i.e., ignore disk conflicts).


• A：

second

I/Os625,15

1064

101000

I/Oper Bytes

bandwidth Busbus of rate I/O Maximum

so KB, 64 transfer I/OEach

second

I/Os000,10

10)100200(

103

I/Oper nsInstructio

rateexecution n Instrucito

CPU of rate I/O Maximum

3

6

3

9

MB/sec56.9ms9.6

KB64

imeTransfer t

sizeTransfer rateTransfer

ms9.6MB/sec75

KB64ms6

imeTransfer t imerotation t Seek disk at I/Oper Time

The maximum number of disks per SCSI bus is 7, which won’t saturate this bus. This means we will need 69/7, or 10 SCSI buses and controllers.


8.8 Real Stuff ： A Digital Camera


Figure 8.14 The Sanyo VPC-SX500 with Flash memory card and IBM Microdrive.


Figure 8.15 Characteristics of three storage alternatives for digital cameras.

Characteristics Sandisk Type I compactFlash SDCFB-128-768

Sandisk Type II compactFlash SDCFB-1000-768

Hitachi 4 GB Microdrive

DSCM-10340

Formatted data capacity (MB) 128 1000 4000

Bytes per sector 512 512 512

Data transfer rate (MB/sec) 4 (burst) 4 (burst) 4 – 7

Link speed to buffer (MB/sec) 6 6 33

Power standby/operating (W) 0.15/0.66 0.15/0.66 0.07/0.83

Size: height X width X depth (inches) 1.43 X 1.68 X 0.13 1.43 X 1.68 X 0.13 1.43 X 1.68 X 0.16

Weight in grams (454 grams/pound) 11.4 13.5 16

Write cycles before sector wear-out 300,000 300,000 Not applicable

Mean time between failures (hours) > 1,000,000 > 1,000,000 (see caption)

Best price (2004) $40 $200 $480


Figure 8.16 The system on a chip (SOC) found in Sanyo digital cameras.


8.9 Fallacies and Pitfalls


Fallacies and Pitfalls

• Fallacy: the rated mean time to failure of disks is 1,200,000 hours, so disks

practically never fail.

• Fallacy: magnetic disk storage is on its last legs, will be replaced.

• Fallacy: A 100 MB/sec bus can transfer 100 MB/sec.

• Pitfall: Moving functions from the CPU to the I/O processor, expecting to

improve performance without analysis.


Multiprocessors

• Idea: create powerful computers by connecting many smaller ones

good news: works for timesharing (better than supercomputer)

bad news: its really hard to write good concurrent programs many commercial failures

Cache

Processor

Cache

Processor

Cache

Processor

Single bus

Memory I/ONetwork

Cache

Processor

Cache

Processor

Cache

Processor

Memory Memory Memory


Questions

• How do parallel processors share data?— single address space (SMP vs. NUMA)— message passing

• How do parallel processors coordinate? — synchronization (locks, semaphores)— built into send / receive primitives— operating system protocols

• How are they implemented?— connected by a single bus — connected by a network


Supercomputers

93 93 94 94 95 95 96 96 97 97 98 98 99 99 00

500

400

300

200

100

0

Single Instruction multiple data (SIMD)

Cluster(network ofworkstations)

Cluster(network ofSMPs)

Massivelyparallelprocessors(MPPs)

Shared-memorymultiprocessors(SMPs)

Uniprocessors

Plot of top 500 supercomputer sites over a decade:


Using multiple processors an old idea

• Some SIMD designs:

• Costs for the the Illiac IV escalated from $8 million in 1966 to $32 million in 1972 despite completion of only ¼ of the machine. It took three more years before it was operational!

“For better or worse, computer architects are not easily discouraged”

Lots of interesting designs and ideas, lots of failures, few successes


Topologies

a. Crossbar

P0

P1

P2

P3

P4

P5

P6

P7b. Omega network

P0

P1

P2

P3

P4

P5

P6

P7

a. 2-D grid or mesh of 16 nodes

b. n-cube tree of 8 nodes (8 = 23 so n = 3)


Clusters

• Constructed from whole computers

• Independent, scalable networks

• Strengths:

– Many applications amenable to loosely coupled machines

– Exploit local area networks

– Cost effective / Easy to expand

• Weaknesses:

– Administration costs not necessarily lower

– Connected using I/O bus

• Highly available due to separation of memories

• In theory, we should be able to do better


Google

• Serve an average of 1000 queries per second

• Google uses 6,000 processors and 12,000 disks

• Two sites in silicon valley, two in Virginia

• Each site connected to internet using OC48 (2488 Mbit/sec)

• Reliability:

– On an average day, 20 machines need rebooted (software error)

– 2% of the machines replaced each year

In some sense, simple ideas well executed. Better (and cheaper) than other approaches involving increased complexity


Concluding Remarks

• Evolution vs. Revolution

“More often the expense of innovation comes from being too disruptive to computer users”

“Acceptance of hardware ideas requires acceptance by software people; therefore hardware people should learn about software. And if software people want good machines, they must learn more about hardware to be able to communicate with and thereby influence hardware engineers.”

Documents

Chapters 8 Storage, Networks, and Other Peripherals