Deep Learning Meets Flow Visualization

Chaoli Wang

Big Data Challenges for Predictive Modeling of Complex Systems Symposium

University of Hong KongNovember 26, 2018

Outline of talk

• Deep learning + Visualization• Flow visualization• FlowNet• Vector field reconstruction• Future directions

Flow visualization

Flow fields

• A vector field: F(U) = V • U: field domain (x, y) in 2D; (x, y, z) in 3D • V: vector (u, v) in 2D; (u, v, w) in 3D

• Like scalar fields, vectors are defined at discrete points

Flow (vector) field

• Rendering each individual vector in the field• Hedgehogs: oriented

lines spread over the volume, indicating the orientation and magnitude of the flow; do not show directional information

• Glyphs, arrows: icons that show directions, but

Glyph-based methods

• Shading every pixels/voxels in the visualization• Line integral convolution

(LIC)• 3D LIC volume is

problematic due to the dense texture

Texture-based methods

• Rendering primitives built from particle trajectories• 1D: streamlines • Line variations: tubes,

ribbons• 2D: stream surfaces• 3D: flow volumes

Geometry-based methods

Streamlines• A family of curves that are instantaneously tangent to the

velocity vector of the flow• Show the direction a fluid element will travel in at any

point in time• Streaklines, pathlines, timelines: extension streamlines

to unsteady data

FlowVisual https://sites.nd.edu/chaoli-wang/flowvisual/

Examples of flow lines and surfaces

FlowNet

Outline of approach• Goal

– A single deep learning approach for identifying representative flow lines or flow surfaces

• Key ideas– Leverage an autoencoder to automatically

learn line or surface feature descriptors– Apply dimensionality reduction and interactive

clustering for exploration and selection

FlowNet user interface

Video demo

• Encoder-decoder framework • 3D voxel-based binary representation as input• Feature descriptor learning in the latent space

FlowNet architecture

Why voxel-based approach• Manifold-based

– Suitable for 3D mesh manifold (genus zero or higher genus surface)

– Does not work for flow lines or surfaces (non-closed)• Multiview-based

– Represent 3D shape with images rendered from different views

– Flow surfaces could be severely self-occluded• Voxel-based

– No precise line or surface is required for loss function computation and reconstruction quality evaluation

– Currently limited to a low resolution (e.g., 643)

FlowNet details

• The encoder consists of four convolutional (CONV) layers with batch normalization (BN) added in between, one CONV layer w/o BN, followed by two fully-connected layers

• The decoder consists of five CONV layers and four BN layers

• Apply the rectified linear unit (ReLU) at the hidden layers and the sigmoid function at the output layer

• Consider three loss functions: binary cross entropy, mean squared error (MSE), and Dice loss

Dimensionality reduction (DR) and object clustering

• Consider three DR methods: t-SNE (neighborhood-preserving), MDS and Isomap(distance-preserving)

• Consider three clustering methods: DBSCAN (density-based), k-means (partition-based), and agglomerative clustering (hierarchy-based)

• Finally choose t-SNE + DBSCAN• Compare three distance measures: FlowNet

feature Euclidean distance, streamline MCP distance, and streamline Hausdorff distance

Parameter setting and performance

Qualitative evaluation

Training set only Test set only Training set + test set

Quantitative evaluation

• Use representative streamlines to reconstruct the vector field using gradient vector flow (GVF)

FlowNetresults (1)

FlowNetresults (2)

FlowNet results (3)

Vector field reconstruction

Outline of approach• Goal

– Target streamline-based flow field representation and reduction in in-situ visualization setting

– Outperform the de facto method of gradient vector flow (GVF) for vector field reconstruction

– Achieve higher reconstruction quality compared with using compressed vector field

• Key ideas– Compress streamlines as sequence data using

SZ compression– Reconstruct vector field using deep learning

Two-stage vector field reconstruction

Stage 1 (S1) Stage 2 (S2)

S1: Low-resolution initialization• Starting from a randomly initialized vector field

Vlow

• Iteratively update the low-resolution vector field Vlow until it produces the same velocities as the input streamlines

Original S1: Random tracing

S1: Fixed tracing

S2: High-resolution refinement• Borrow the ideas of image super-resolution and

image completion• Build a CNN to upscale Vlow to Vhigh

• Fill the empty voxels through a nonlinear combination of their neighborhoods

Original S1: Random tracing

S2: Random tracing

Verification of two-stage reconstruction

Original

S2: Random tracing

Direct fixed tracingDirect random tracing

S1: Random tracing

S1: Fixed tracing

CNN architecture

The CNN includes seven deconvolutional layers and four residual blocks

An example of a residual block

Reduction rate and reconstruction PSNR (different numbers of lines)

Compression rate and reconstruction PSNR (different error bounds)

Visualization of reconstruction errorsG

VFO

ur m

etho

d

Comparison against direct compression of vector field

Ground truth

SZ compression Our method

• Scientific simulation data– From steady to unsteady data– From single variable to multivariate data

• Deep learning techniques– From CNN & RNN to GNN (graph neural network)– Adversarial learning and transfer learning

• Goals– Generalized 3D feature learning framework– Target scalar and vector fields, time-varying

multivariate data– Data augmentation for scientific visualization

Future directions

• Team members– Jun Han, Hao Zheng, Dr. Jun Tao (now at Sun

Yat-sen University), Martin Imre• Collaborators

– Drs. Hanqi Guo, Danny Chen• Funding

– NSF IIS-1455886, CNS-1629914, and DUE-1833129

– NVIDIA GPU Grant Program

Acknowledgments

chaoli.wang@nd.edu

Thank you!