A Scalable Distributed Datastore for BioImaging
R. Cai, J. Curnutt, E. Gomez, G. Kaymaz, T. Kleffel, K. Schubert, J. Tafas{jcurnutt, egomez, keith, jtafas}@r2labs.orgRenaissance Research LabsDepartment of Computer ScienceCalifornia State UniversitySan Bernardino, CA 92407 Supported by
NSF ITR #0331697
Background CSUSB Institute for Applied
Supercomputing Low Latency Communications
UCSB Center for BioImage Informatics Retinal images Texture map searches Distributed consortium (UCB, CMU)
Retina Images
Normal (n)
3 month detachment (3m)1 day detachment followed by 6 day reattached with increased
oxygen (1d+6dO2)
3 day detachment (3d)
Laser scanning confocal microscope images of the retina
Environment
UCSB
Raven Cluster
Imageand
metadataserver
search
external internal
BISQUE
features
analysis
Hammer/Nail ClusterLocal
LAN
CSUSB
Imageand
metadataserver
Imageand
metadataserver
WAN
Loca
l
Lustre
Software Open source OME
Postgresql 7 Bisque
Distributed datastore Clustering NFS Lustre
Benchmark: OSDB
Hardware - Raven 5 year old dual processor 1.4 GHz Pentium 3 256MB RAM 60GB SCSI Compaq Proliant DL-360
servers. Raven has been latency
tuned.
Hardware – Hammer/Nail
UCSB
CSUSB
Hammer headnode 5 Nail nodes quad CPUs
3.2 Ghz Xeon 4GB RAM 140GB SCSI Dell servers Bandwidth tuned
(default)
Outline Effect of node configuration Comparison of network file systems Effects of a wide area network (WAN)
Relative LAN Performance
NFS LAN/WAN Performance
Design Effects? A few expert users
Metadata searches• Small results to user
Texture searches• Heavy calculation on cluster• Small results to user
Latency tuning
Outline Effect of node configuration Comparison of network file systems Effects of a wide area network (WAN)
NFS / Luster Performance. NFS
well known standard Configuration problems with OME performance comparison of the Lustre file system
Lustre Journaling Stripe across multiple computers Data redundancy and failover
Relative Performance on LAN NSF/Lustre
Compared to local DB
1GB LAN two significant
differences
Significant Differences
NSF caching bulk deletes and
bulk modifies
Lustre stripes across computers increase the
bandwidth
Outline Effect of node configuration Comparison of network file systems Effects of a wide area network (WAN)
Effect on Wide Area Network WAN Compared three connections
Local Switched, high speed LAN (1 Gb/s) WAN between UCSB and CSUSB (~50 Mb/s)
NFS only UCSB didn’t have Lustre installed Active research prevented reinstalling OS
Local/LAN/WAN Performance
Effect on Wide Area Network WAN Most significant effect
Not bandwidth intensive operations Latency intensive operation Next generation WAN will not solve the problem.
Frequently used data must be kept locally Database cluster Daily sync of remote databases
Conclusions Scientific researchers
Latency tune network Don’t bandwidth tune
Latency of WAN is too large replicate data and update.
Bisque/OME NFS issues Lustre
High bandwidth operations Stripe Lustre across systems
Future directions: Agent based texture search engine
Loosely coupled cluster• WAN connection• Unreliable connection• Fault tollerant• Parallelize Jobs
Open source components• Scilab• Convert NSF funded algorithms in Matlab
Simple interface Superior caching scheme for Lustre
Questions…