Embed Size (px)
Citation preview
Etree — A Database-oriented Method for Generating Large Octree Meshes
David R. O’Hallaron1,2
Tiankai Tu1 Julio C. López2
1School of Computer Science2Electrical and Computer Engineering Department
Carnegie Mellon University
2
MotivationGoal : Make it possible to run large-scale physical simulations on PC’s with limited physical memory
Mesh generation
Mesh
t
Simulation results
Physical model Visuali-
zationSolver
Approach : Index and store the datasets in databases and compute on the databases directly
Requires research at the intersection of computer systems, scientific computing, and databases
3
Octree mesh generation
a b c
d e f g
h
i j k l
m
a compromise between structure and modeling power
balance requirement (2-to-1 constraint)
can be implemented in-core or out-of-core
h1 h2 h3 h4
element/octant
a
b
c
d
e
f
g
h
i
j
k
l
m
h2
h1
h4
h3
master node
slave node
4
Etree mesh generation overview
A general method for manipulating large out-of-core octrees by querying databases
unbalancedoctree
application-specific input
construct
etree library
transform
etree library
balancedoctree
balance
etree library
elementdatabase
nodedatabase
5
Etree library components
Application(e.g.construct, balance)
Etree library
B-Tree
Lin
ea
rq
uad
tre
e
Au
ton
avi
gat
ion
Lo
cal
ba
lan
cin
gEtree API
Etree API — a simple Application Programming Interface
Linear quadtree — an encoding scheme to assign keys to octants
Auto navigation — a mechanism for constructing an octree automatically
Local balancing — a technique that speeds up balancing operation
B-tree — a database index structure to store and access octants on disk
6
Etree API
Unix file I/O style, three classes :
initialization and cleanup; example:
etree_t *etree_open(const char *path, int flag, …);
octant-level operations; example:
int etree_insert(etree_t *ep, location_t loc, void *value);
octree-level operations; example:
int etree_balance(etree_t *ep, decom_t *baldecom);
7
Linear quadtree — how it works
Morton code: maps n-dimensional points to one-dimensional scalars
Locational code: appends an octant’s level to the Morton code of its left-lower corner
a b c
d e f g
h
i j k l
m
x
y
a
b
c
d
e
f
g
h
i
j
k
l
m
32 4 7 85 60 1
0
1
2
3
4
5
6
7
8
a b c d e f g h i j k l m
8
Linear quadtree — how it works (cont’d)
010
00 11 00
010
interleave the bits to obtain Morton code
d’s left-lower corner (2, 2)
binary form (010, 010)
001100_11append the level of d
x
y
a
b
c
d
e
f
g
h
i
j
k
l
m
32 4 7 85 60 1
0
1
2
3
4
5
6
7
8
9
Linear quadtree — applications
an addressing scheme that clusters nearby octants
finding an octant without knowing its locational code
the order imposed by the locational code is the same as the preorder traversal of leafs in octree
a b c
d e f g
h
i j k l
m
a
b
c
d
e
f
g
h
i
j
k
l
x
mm
10
Navigation octree
guided by an application function
an in-memory pointer-based octree
dynamically grows in the depth-first order
leaf octants are pruned and flushed to disk in preorder (increasing locational code order)
appends the octant to the database to avoid database search
Auto navigation — how it works
: octants not yet processed (in memory)
: non-leaf octants being decomposed (in memory): leaf octants (flushed to database)
11
Local balancing — how it works
Operational steps
1. partition the whole domain into equal-size blocks
2. conduct internal balancing to enforce 2-to-1 constraint within each block (in memory a resident blocking array)
3. perform boundary balancing to resolve interactions between adjacent blocks
12
Correctness of local balancing
Claim: Interactions between adjacent blocks are always absorbed by boundary octants and will not be propagated into the blocks
Proof: See the paper
13
Evaluation — questions
Is the etree method feasible?
How does the running time vary with the physical memory size?
What is the impact of auto navigation?
What is the impact of local balancing?
14
Evaluation — methodology We implemented an etree-based mesh generator to generate a family of finite element meshes for San Fernando valley earthquake wave propagation simulations
Mesh Elements Nodes Slave nodes
SF10 7,940 12,118 4,432
SF5 76,330 105,886 34,858
SF2 1,838,524 2,213,035 407,336
SF1 13,579,124 15,097,365 1,649,855
15
Evaluation — setup All experiments are conducted on a PIII 1GHz machine running Linux 2.4.17.
The machine’s physical memory for the experiments ranges from 128MB to 880MB
Before each experiment, two 1.5 GB files are sequential scanned to ensure that the operating system’s buffer cache is flushed
16
Evaluation — etree feasibility
Generating a mesh with 13.6 million elements and of size 4.3GB in 2.6 hours seems reasonable
The overall throughput increases with mesh size
Mesh Elements DB size (MB) Time(sec) Thruput(elem/s)
SF10 7,940 2.5 39.9 199
SF5 76,330 24 186.0 410
SF2 1,838,524 583 1,636.7 1,123
SF1 13,579,124 4,300 9,448.8 1,439
Etree-based mesh generator running time and throughput
All experiments are performed with 128MB physical memory
17
Evaluation — impact of memory size
Memory size does not have a significant impact on the running time
The etree method is not relying on the operating systems internal caching mechanism to achieve its performance
0%
20%
40%
60%
80%
100%
128
256
512
880
128
256
512
880
128
256
512
880
128
256
512
880
Memory size (MB)
Run
ning
tim
e
findslave
query
transform
balance
construct
SF5 SF2 SF1SF10
18
1.0E+00
1.0E+01
1.0E+02
1.0E+03
1.0E+04
1.0E+05
1.0E+06
512 1,000 2,000 4,000 8,000 16,000 32,000 64,000
B-tree buffer size (KB)
Co
nst
ruct
ion
tim
e (m
s)
sf1
sf2
sf5
sf10
Evaluation — impact of auto navigation
Reducing B-tree buffer size does not increase the construction time
Auto navigation is not sensitive to B-tree buffer size
19
1.0E+02
1.0E+03
1.0E+04
1.0E+05
1.0E+06
1.0E+07
1.0E+08
0 262 2,097 16,777
Blocking array size (KB)
Bal
ance
tim
e (m
s) sf1
sf2
sf5
sf10
Evaluation — impact of local balancing
Achieves a speed-up factor ranging from 8 (SF1) to 28 (SF10)
Benefits from the one-time scan of the database and the efficient array-based neighbor finding algorithm
20
Related work General octree algorithms: Samet 90
Octree mesh: Shepard & Geoges 91, Bern et al. 90, Young et al. 91, Wang99
Out-of-core octree method: Salmon 97
Linear quadtree: Gargantini 82, Morton 66
Space filling curve: Orenstein 84, Orenstein 86, Faloutsos & Roseman 89
Large dataset processing: Freitag & Loy 99, Seamons & Winslett 96, Ferreira et al. 99, Kurc et al. 01, Choudhary et al. 99
Parashar & Browne 97
21
Conclusion and future work Experiment results suggest that the etree method can generate large octree meshes on memory-limited machines in a reasonable amount of time
Incorporating existing database techniques (linear quadtree and B-tree) with new algorithms (auto navigation and local balancing) in a unified design scheme (the etree) can deliver new capability
We are porting the etree method to commercial database systems such as IBM DB2
