FIOS: A Fair, Efficient Flash I/O SchedulerStan Park and Kai Shenpresented by Jason Gevargizian

Flash devices

● NAND Flash devices are used for storage○ aka Solid-state Drives (SSDs)

● much higher I/O performance than mechanical disks

● this increased performance has potential to alleviate I/O bottlenecks in data-intensive applications

I/O Schedulers & Flash

● conventional I/O schedulers were built for mechanical storage devices

● they fail to recognize the unique flash characteristics○ read-blocked-by-write○ I/O anticipation○ suppression of parallelism

● This results in poor fairness and performance on Flash devices

Write limitations of Flash

● erase-before-write○ write target locations must be erased before writing

● large erasure granularity○ 64-256x the size of the read

● consequently, read-write speeds are highly asymmetric○ reads are one to two orders of magnitude faster○ mechanical disks r&w operations are fairly

symmetric

read-blocked-by-write

● write limitations lead to read interference on these devices

● when reads are blocked by writes, reads suffer a substantial slowdown

● scheduling reads and writes with equal preference creates unfairness

I/O Anticipation

● anticipation is the idling of a device to prepare for soon-arriving I/O requests○ necessary for mechanical devices with high seek

times○ some anticipation is necessary for fairness

■ to avoid premature switching/advancing in quanta and fair queuing approaches

● Flash devices have low access times and are hurt by the liberal use of anticipation

I/O Parallelism

● Flash devices have built-in parallelism○ Multiple channels, often each with multiple parallel

planes● conventional fairness oriented I/O

schedulers suppress parallelism○ simplifies accounting/allocation of device for queues○ this does not maximize the benefit of Flash devices

I/O Schedulers & Flash

● conventional I/O schedulers were built for mechanical storage devices

● they fail to recognize the unique flash characteristics○ read-blocked-by-write○ I/O anticipation○ suppression of parallelism

● This results in poor fairness and performance on Flash devices

FIOS Design - 4 Techniques

1. Fair timeslice management with timeslice fragmentation and concurrent request issuance

2. support read preference to combat read-blocked-by-write

3. enable concurrent issuance to utilize device-level parallelism

4. use I/O anticipation judiciously to maintain fairness at minimal idling cost

1) Fair Timeslice Management

● each task is allotted one full timeslice within an Epoch

● timeslices can be fragmented○ remaining timeslice is calculated after each I/O

request completion● epochs end when either:

○ (1) no task has a non-zero timeslice○ (2) no task with a positive timeslice has I/O requests

2) Read/Write Interference

● reads suffer dramatic write interference● read preference policy is used

○ all writes are blocked until all reads complete■ writes have a large service time, so additional

queuing time is small○ Epoch policy governs read preference (to ensure

writes don’t starve)● read preference is optional (some devices

like vertex don’t have high asymmetry)

3) I/O Parallelism

● many Flash drives have internal parallelism● after issuing I/O requests, FIOS searches for

additional requests to run in parallel● I/O Cost accounting

○ outstanding requests can be delayed slightly by parallel request issuance

○ tasks must not be billed for the additional time (spent on other tasks’ requests)

3) I/O Parallelism

● I/O Cost accounting○ FIOS attempts to not bill tasks for this additional

device time○ FIOS calibrates (for the given device once) read and

write request times at different sizes, then uses interpolation for other sizes

○ When calibration unavailable, FIOS assumes that outstanding requests take an equal time on device

4) I/O Anticipation

● anticipation (idling) is used for mechanical disks to combat long seek times

● Flash devices are fast and are hurt by I/O anticipation

● some anticipation is still necessary to ensure fairness...

4) I/O Anticipation

● FIOS uses anticipation minimally● anticipation is employed to delay epoch

ending for anticipating tasks that have remaining timeslice left (but no requests)

● anticipation is also employed after read completion to mitigate read-blocked-by-write○ anticipates instead of immediately issuing write

FIOS Design - 4 Techniques

1. Fair timeslice management with timeslice fragmentation and concurrent request issuance

2. support read preference to combat read-blocked-by-write

3. enable concurrent issuance to utilize device-level parallelism

4. use I/O anticipation judiciously to maintain fairness at minimal idling cost

Evaluation - Schedulers

FIOS is compared against three fairness- oriented I/O schedulers:1. Linux CFQ scheduler (Completely Fair

Queuing)2. SFQ (Start-time Fair Queuing with a

concurrency depth)3. a quanta-based I/O scheduler similar to one

employed in Argon

Evaluation - Flash Devices

Evaluation was performed on the following drives:● Intel X225-M Flash-based SSD (2009)● Mtron Pro 7500 Flash-based SSD (2008)● OCZ Vertex 3 Flash-based SSD (2011)● SanDisk CompactFlash drive on 6-Watts

“wimpy” node

Evaluation - Workloads

● Set of synthetic benchmarks with varying I/O concurrency

● realistic applications○ SPCweb workload on Apache○ TPC-C workload on MySQL database

● CompactFlash drive in a low-power wimpy node using FAWN Data Store workload

Synthetic Benchmarks

● 1-reader 1-writer that continuously issue 4KB reads/writes respectively

● 4-reader 4-writer of 4KB reads/writes● 4-reader 4-writer with thinktime

○ exponentially distributed thinktime between I/O requests such that total thinktime is equal to I/O device time

● 4KB-reader and 128KB reader

Synthetic Benchmarks● slowdown is latency

ratio to running-alone● raw, CFQ, & SFQ

suffer read interference

● quanta performs fairer due to aggressive per-task quantum

● quanta suffers in performance due to excessive anticipation and lack of parallelism

Synthetic Benchmarks● on vertex, all achieve

decent fairness due to low read-write asymmetry

Synthetic Benchmarks (continued)● the 4KB-reader and 128KB-reader benchmark reveals a case

where even on the vertex, only FIOS and quanta achieve fairness

Synthetic Benchmarks● overall concurrent

efficiency (relative throughput of concurrent execution compared to running-alone throughput)

● quanta suffers for its aggressive methods for fairness

SPECweb and TPC-C

● read-only SPECweb99 workload running on an Apache 2.2.3 webserver

● write intensive TPC-C running on MySQL 5.5.13 database

● each application is driven by a closed-loop load generator with 4 concurrent clients

○ each client issuing requests continuously

SPECweb and TPC-C● write intensive TPC-C

experiences more slowdown on Flash

● again, quanta suffers from excessive anticipation

● FIOS performs the best on these benchmarks

Low-Power CompactFlash

● low-power wimpy node like the ones used with FAWN

○ node contains an Alix board with single-core 500MHz AMD Geode CPU, 256MB SDRAM

● 16GB SanDisk CompactFlash drive○ does not have parallelism

● FAWN Data Store application workload○ two tasks, one performing hash gets and another

performing hash puts

Low-Power CompactFlash● quanta and FIOS perform

the most fair● quanta does not suffer as

it had from lack of paralleization (since CompactFlash does not support it)

Wrap Up

● Flash storage devices show potential to alleviate I/O bottlenecks

● conventional I/O schedulers do not account for the unique characteristics of Flash

● the authors propose FIOS as a method to better handle the scheduling of I/O for Flash devices with promising results

Discussion

Thank you. Questions?

To the class:Much of the benefit of FIOS relies upon read-write speed discrepancy of Flash. Do you think FIOS and systems like it will be useful for very long?