Upload
joaquin-vanschoren
View
1.347
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Terabyte scale Sensor Network data analysis using MapReduce/ Hadoop
Citation preview
dr. ir. Joaquin Vanschoren
InfraWatchterabyte-scale sensor network analysis
Hollandse Brug
Hollandse BrugBridge deck weakened, heavy traffic bannedComplete deck replaced
Sensor network to monitor bridge behavior
Stress (longitudinal)
Stress (longitudinal)
Stress (longitudinal)
Stress (transverse)
Stress (longitudinal)
Stress (transverse)
Vibration
Stress (longitudinal)
Stress (transverse)
Vibration
Stress (longitudinal)
Stress (transverse)
Vibration
Temperature
Stress (longitudinal)
Stress (transverse)
Vibration
Temperature
Stress (longitudinal)
Stress (transverse)
145 sensors, 100Hz: 5GB/day
Vibration
Temperature
Stress (longitudinal)
Stress (transverse)
InfraWatch project
• Model bridge behavior
• Effects of traffic, weather,... (short term) - warnings
• Effects of decay (long term)
• Interplay between effects (e.g. temp vs stress)
• Optimal sensor network (for new bridges)
• Additional sensors (e.g. chloride)
Enter Hadoop
• Terabyte-scale data analysis (I/O bound)
• Typical MapReduce (Hadoop) case!
• No Hadoop libraries for signal processing?
• Mahout (in progress) general data mining
• Some math libraries (in progress, mostly for calculating Pi(?))
Enter Hadoop
• We need new building blocks:
• Convolution
• Discrete Fourier Transform
• Segmentation
• Modified classification, clustering,... algorithms
• Let’s start coding...
A typical day
• Cars, trucks, traffic jams
• Lots of noise
Smoothing
• Remove noise by convolution with Gaussian response function
Smoothing
• Remove noise by convolution with Gaussian response function
Smoothing
• Remove noise by convolution with Gaussian response function