WSO 2007 – SBC - Rio

A Year in Operating SystemsA Year in Operating Systems

Dilma da Silva

IBM TJ Watson Research Center

2

IBM Research

WSO’07

OSDI’06 – November, Seattle

best papers– “BigTable”, from Google

– Rethinking the “sync”, from Univ of Michigan

Let’s look at BigTable

3

IBM Research

WSO’07

BigTable Data model: a big map•<Row, Column, Timestamp> triple for key - lookup, insert, and delete API

•Rows:

• atomic

• lexicographic order

• Tablet: range of rows

•Arbitrary “columns” on a row-by-row basis

•Column family:qualifier. Family is heavyweight, qualifier lightweight

•Column-oriented physical store- rows are sparse!

•Does not support a relational model

•No table-wide integrity constraints

•No multirow transactions

4

IBM Research

WSO’07

5

IBM Research

WSO’07

API example: writing to Bigtable

// Open the tableTable *T = OpenOrDie(“/bigtable/web/wtable”);// Write a new anchor and delete an old anchorRowMutation r1 (T, “com.cnn.www”);r1.Set(“anchor:www.c-span.org”), “CNN”);r1.Delete(“anchor:www.abc.com”);Operation op;Apply (&op, &r1);

6

IBM Research

WSO’07

API example: Reading from Bigtable

Scanner scanner(T);ScanStream *stream;Stream=scanner.FetchColumnFamily(“anchor”);Stream->SetReturnAllVersions();scanner.Lookup(“com.cnn.www”);for ( ; !stream->Done(); stream->Next() ) {

printf(“…”, scanner.RowName(),stream->ColumName(), steam->MicroTimestamp(), stream->value();

}

7

IBM Research

WSO’07

Bigtable Major compenents

Library linked into every client Tablet servers (dynamically added/removed) Master

– Assigns tablets to tablet servers

– Detects addition/expiration of tablet servers

– Balances tablet-server load

– Garbage collection of files in GFS

– Schema changes (table and column family creation)

– Not involved in read/writers (clients to go tablet servers)

8

IBM Research

WSO’07

HotOS’07 – May, San Diego, CA

104 submissions, 21 accepted Not open to “public”

9

IBM Research

WSO’07

Hot Subject: TM

Transactional Memory– Keynote speech by David Wood

– Parallel programming to the masses (Univ of Wisc)

– Lots of architecture papers; no large concurrent system tried yet (TxLinux in the works … but is it the right target?)

– Semaphores are simple … look at the mess we have

– What TM is going to look like after Microsoft & Intel put their hands in?

– Is programmer going to do: main { atomic{prog}}

– OS Issues: manage TM hardware resources, deal with I/O (they have the notion of “escaping a trans”)

10

IBM Research

WSO’07

Hot Subject: Multicore/manycore

Paper from UT Austin: Is the optimism in optimistic concurrency warranted?

– TxLinux ISCA’07 showed optimistic conc didn’t help much (parallel make benchmark)

– Syncchar: tool to measure data independence of critical sections

– Reorganize some data structures is necessary Thread Scheduling for Manycore Platforms (Intel)

– Goal is to use high-level information supplied by the user to guide thread scheduling

– Threads receive an RTID (Related Thread ID): which ones should run close together

– Offer an API to specify placement

– Paper, presentation not concrete enough Automatic Mutual Exclusion (Microsoft)

– Really ongoing work (no non-trivial program working)

– Code not decorated is a transaction; unsafe code specifies it

11

IBM Research

WSO’07

Hot Topic: Virtualization

Timothy Roscoe “hype & virtue”– Attached the “hype” on the role of the hypervisor– Whe aren’t the cool things done with VM recently (e.g. replay) done

in the OS?– Is VM exciting because it is not POSIX?– People, write new OS! (Comments: reviewers, accept novel systems

even if they can’t run emacs or compare with other systems) Relaxed Determinism (Univ Toronto): use redundant

execution for security Micro-drivers (U Wisc): tool for splitting (at functional

granularity) perf critical part; put that in the kernel and rest in user space– Reaction: why keep part in kernel? Look forward complete paper at

osdi/sosp

12

IBM Research

WSO’07

Perceived as very cool

Web-based Covert File System (Rutgers)– Storing data with guaranteed confidentializty and

plausible deniability

– Put a file system into the images you store on a online service (covertly, using steganographic techniques)

– Challenge is to design file system so that its access pattern for images doesn’t differ much from normal user storing family pictures

13

IBM Research

WSO’07

Perceived as very cool

Hot Comments: how to make program comments more useful? (UIUC) – Idea: use natural language processing to look at

comments in code and identify bugs and bad comments

– Challenge for NLP is that comments are not well-written prose

– Studied Linux comments; got a tool to work on lock comments (now cool is more generic)

– Succeed in finding bugs in Linux and mozilla

14

IBM Research

WSO’07

Panel Discussion: put the science into Computer Science

Brian Noble advocated that if we say an interface or programming model is better, we have to do the “usability” work to prove– reaction: against holding CHI as a model

YY Zhou: hacker + nerd (user math and models) Jeff Modul (for Jay Lepreau): reproducibility

– Reaction: sciences usually don’t do it as one assumes they do; CS is changing the world so building new stuff seems more relevant/exciting than revising/checking the past

Butler Lampson: write specs

15

IBM Research

WSO’07

Hot Topic without Hot Discussion: Web 2.0

Microsoft session Problem: it used to be that every click on a browser would go to a

server … now a lot of stuff is running in clients Mashup OS: OS ideas to manage what is going on in a browser

with AJAX and mashups and inline scripts Live Monitoring: instrumentation and analysis to debug/maintain

web apps– Scary adaptive application rewriting

End to end web app security– Suggests changes to existing browsers so client-side can enforce fine-grained, app-

specific security policies

– Reaction: browsers (IE, firefox) have been resistent to change (opera exception)

16

IBM Research

WSO’07

Miscellaneous

Auditing to keep online storage services honest (HP Labs)– Nice problem; lots of discussion that instead of spending resources on

auditing/insurance model, increase reliability by replication, etc. Optimization paper from HP Labs Clustering for approximate matches Optimizing power consumption in storage

– Revisiting Log-Structured Files Systems Protecting privacy: finding out about you from your

wireless footprint (Intel) Nixos: system configuration using functional languages

ideas (old state is not changed when you install a new tool)

17

IBM Research

WSO’07

Miscellaneous

Processor Hardware Counter Stats as a 1st class Resource (Rochester)

– Make a case for direct OS management of counter stats, to be used on OS decisions

– Security concerns: if user can get stats, it can learn about other processes running in the system (e.g. it can derive an RSA SSL key)

Practical, Verifiable kernel: (seL4) modeling and model checking in Haskell; model to pass to C manual

– Audience (including myself) missed in the talk how they go from proved model to C; it’s a manual proof it seems

– Maintain code and model 4-minute madness

18

IBM Research

WSO’07

USENIX’07 – June, Santa Clara, CA

Hyperion: High Volume Stream Archival for Retrospective Querying, UMass

SafeStore: A Durable and Practical Storage System, UT Austin

19

IBM Research

WSO’07

Eurosys’07 – March, Lisbon

It seems it will establish itself as an important conference

Session on multi-core, multi-processor issues