View
30
Download
0
Category
Tags:
Preview:
DESCRIPTION
Out-of core Streamline Generation Using Flow-guided File Layout. Chun-Ming Chen 788 Project. Background. Visualize flow fields with streamlines Scientific data is huge Traditional: Compute in clusters Drawbacks: High Equipment Cost Inter-node communication. - PowerPoint PPT Presentation
Citation preview
1
Out-of core Streamline Generation Using Flow-guided File Layout
Chun-Ming Chen
788 Project
2
Background
• Visualize flow fields with streamlines • Scientific data is huge – Traditional: Compute in clusters– Drawbacks: • High Equipment Cost• Inter-node communication
http://vidi.cs.ucdavis.edu/publicationshttp://nuweb2.neu.edu/
3
Background
• Nowadays: multi-core CPU on single machine• May not have enough memory capacity• Out-of-core computation is needed– Out-of-core: data cannot be fully loaded into main
memory
4
Goal
Compute streamlines on a lower-cost multi-core machine with limited memory, given arbitrary seeds
5
Demand Paging Algorithm
• Preparation Stage:– Break flow fields into blocks
• Streamline Generation Stage:– Only load needed blocks during computation– Release least recently used (LRU) block when
memory full
Load data from Disk Compute Release data
(LRU)
Store data in memory
pool
6
Multi-core streamline computation
Threaded Computation
Seeds for block 1
Seeds for block 2
Seeds for block 3
Seeds for block 4
Threaded Computation
New seeds generated from
block 1
Job Queue
7
Problem of Out-of-core Computation
• Earlier tests: 1Gb Data– Environment:• 8-core Intel Machine• Limit 25Mb memory usage
– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s
• IO is the bottle neck
8
More tests
• Read all blocks in a 6Gb data• Unit block size: float 16x16x16 (49152 bytes)
Total 131,072 blocks– Random access: 249.146794 sec– Sequential read: 13.594309
sec– Reverse-Sequential read: 264.262752 sec
• Sequential read can be 20 times faster• Reason: Disk Prefetching
9
File Layout
• Re-arrange data to increase more sequential reads
• Hilbert Curve Layout:
10
Result of Scheduling for Hilbert Curve Layout
• Scheduler: only read forward• Test: 1Gb Data– Environment:
• 8-core Intel Machine• Limit 25Mb memory usage
• Old test:– Time Generating streamlines: 8.436 s– Time Loading flow field : 32.524 s
• Hilbert layout:– Time Generating streamlines: 7.890 s– Time Loading flow field : 0.610 s
11
Layout By Flow Direction
12
Next And Conclusion
• Next:– Better layout?– Re-arrange data based on flow direction– NP-hard Problem
• Conclusion:– If we want to analyze large scientific data in a single
machine, out-of-core computation is required now and also in the future
– Good File layout is important for out-of-core computation
Recommended