EnsemBlue: Integrating Distributed Storage and Consumer Electronics Daniel Peek and Jason Flinn...

EnsemBlue: Integrating Distributed

Storage and Consumer Electronics

Daniel Peek and Jason Flinn

University of Michigan

University of Michigan 2

Bringing Distributed Systems Home

• Distributed systems ignore Consumer Electronic Devices (CEDs)

• Much user interaction is through CEDs

• Integrate CEDs into distributed systems

University of Michigan 3

CED Integration Difficult

• Application-specific resources

• Heterogeneous interfaces• Closed platforms

– Cannot execute our code

University of Michigan 4

Today

University of Michigan 5

• Hypothesis: Distributed file systems can help manage CEDs and multimedia

EnsemBlue

My Data

My Devices

University of Michigan 6

EnsemBlue

• Based on BlueFS [Nightingale 04]– Client-server design– Single namespace– Supports mobile clients– Designed for small group of users

FileServer

University of Michigan 7

Outline

• Motivation• Device-Specific Namespaces• Device-Specific Functionality• Disconnected Collections of Devices

• Evaluation• Conclusion

University of Michigan 8

Device-Specific Protocol

Connecting CEDs

• CEDs cannot perform DFS protocol

DFS Protocol?

Distributed File System

Device-Specific Protocol

• Attach: computer speaks for CED• Works with disconnected clients

DFS Protocol

University of Michigan 9

Device-Specific Namespaces

• User prefers one organization• CED requires another organization

Classical

Jazz

F01

F02

F03

F04

University of Michigan 10

• Where to store the mapping?

Device-Specific Namespaces

• Cannot store in DFS due to partial caching

• Make CEDs self-describing

University of Michigan 11

Receipts

• What should mapping look like?

Object 1.123.46F2

~dpeek/classical/song.mp3

/iPod/f27/ABCD.mp3

• Path lookups may fail with partial caching

• Receipts bypass path resolution

University of Michigan 12

Device-Specific Execution

• Updating indexes• Transcoding• Type-specific caching• Organizers

University of Michigan 13

Device-Specific Execution

• Cannot execute on CED• Leverage general computers

– Take action when files change

• Problem: Need notification of file changes

University of Michigan 14

Notification Via Persistent Queries

• Don’t need new mechanism• Leverage cache consistency mechanism

– Structure notifications as file system object

– Robust to crash– Handles disconnected operation

updates

truncate

read

process*.mp3

University of Michigan 15

MP3

M4A

Persistent Queries• Example: M4A to MP3 transcoder

FileServer

M4A to MP3Transcoder *.m4a

M4A

M4A

MP3

*.m4a MP3

University of Michigan 16

Disconnected Devices

• Server gives safety, availability, consistency

• Server mediates communication

FileServer

University of Michigan 17

Disconnected Devices

• Disconnected devices cannot interact!

FileServer

University of Michigan 18

Disconnected Devices

• Peer-to-Peer better supports mobility

• CEDs store and forward updates

FileServer

University of Michigan 19

Ensembles

• Ensembles can avoid these problems

• Select a pseudo file server– Consistent view of all files– Improved data access– Update propagation

Pseudo FileServer

?

?

University of Michigan 20

Outline

• Motivation• Device-Specific Namespaces• Device-Specific Functionality• Disconnected Collections of Devices

• Evaluation• Conclusion

University of Michigan 21

Microbenchmarks

• Persistent query execution is fast– <200 ms for 1,000s of matches, GBs of data

• Persistent queries have low overhead– Negligible even for 10,000 queries

• Ensemble formation as fast as 20 ms– Proportional to data transfer size

University of Michigan 22

Organizing My Photos

PhotoOrganizer

iCal

*.jpg

FileServer

University of Michigan 23

Managing My Music

iPod DBUpdater

MP3

MP3 DB

DB

MP3 DB

MP3 DB

*.mp3

FileServer

MP3

University of Michigan 24

Conclusion

• Added CEDs to distributed systems

• Device-specific data organization

• Device-specific functionality• Supports TiVo, iPod, digital cameras, media players, cell phones, PDAs

University of Michigan 25

Questions

?