Experience with IoT, Sensor Fusion and AI for Predictive Analytics

3 IoT Sensor Fusion Projects:ADAC Smart Cam (Alaska)

U. of Colorado ESE ProgramERAU Drone Net (Arizona)

Sam Siewert

https://www.altera.com/https://antmicro.com

https://www.nvidia.com

April 1, 2019

Data Challenges for ICARUS Predictive AnalyticsGoal - Safe, Secure sUAS Class-D FlightVirtuous Simulation Based Prediction

Sensor and Information Fusion Required:1. LWIR – narrow and wide field imaging2. Visible – hemispherical, narrow imaging3. Acoustic – intensity, beamforming4. GPS – localization, time for IoT nodes5. OEM Navigation - OTS UAS Track6. HF Navigation – custom GPS + INS7. ADS-B - GA de-confliction, See-be-Seen8. Tracking RADAR – UAS (Echodyne)9. Weather RADAR – ERAU, NWS, EDD10. LIDAR – Flight Sense-and-avoid,

obstacle avoidance, urban mapping11. UTM planning - B4UFLY, LAANC, Maps12. Flightradar24 - aggregation service

Sam Siewert AFRL - Predictive Analytics Workshop 2

RADAR Track

EO/IR Cue, Classify, Identify

Weather

FAA Facility Maps

Traffic

2014-16– ADAC & ERAU Past Fusion Project

UAA – ADAC, SmartCam

ERAU (Undergraduate Research Team)– Arctic Power Team – Power Team Poster

CU Boulder – Embedded Systems Engineering Graduate Program

Industry Advising/Collaboration Participants– Randall Myers, Mentor Graphics

Sam Siewert 3

Smart Cam Concept – FPGA or GP-GPU

Sam Siewert 4

USB3.0 HD (Panchromatic,

NIR, RGB)

GP-GPU (FPGA)Co-Processing

Cloud Analytics and

Machine Learning

Flash SD Card(local

database)SD Analog(LWIR)

Saliency & Behavioral AssessmentThermal Fusion Assessment

Many multi-spectral focal planes …

2D/3D Spatial Assessment

Smart Cam Vessel Tracking FeasibilityHot-spots (engines, exhaust, cabin, lights) segment wellImprove with Common Intrinsic/Extrinsic Characteristics and Image FusionValdez Harbor, Alaska (2015)

Sam Siewert 5

Exhaust stacks for Tanker at TAP

??

200mm Visible

25mm Visible 25mm LWIR

Add camera systems to USCG Cutters (around, mast)Detect bodies in water, Port trespassing, Complements Aircraft LWIR Automation for Unattended Arctic Camera Systems

Feasibility for SAR Ops & Port Security

Surfers in the WaterHand-held, Cutter Mounted, BuoysComplements Existing Helicopter and C130 FLIR(Field Test – June 2015, Malibu)

Trespassers at Night Shown on JettyHand-held, Port Drop-in-Place, BuoysComplements Existing SecurityOff-Grid Installations(Field Test – June 2015, San Pedro)

Sam Siewert 6

Trespassers at Night(Motion, Audio Cues)

University of Colorado ESE & IoT Program

Founded August 2000

Sam Siewert 7

Founded August 2000 (McClure, Siewert) -CE, SE programs in EE

Certificate Program and MS EE

Professional MS Program in 2014

Coursera Degree in 2019 TBA

Mixture of Traditional and Professor Practitioner Faculty

Strong Connection to Local High-Tech and High-Tech Nationally

Teach Students Embedded, IoT, Sensor Fusion, Intelligent Automation

March 20, 2019 Sam Siewert, ICARUS Group

AFRL - Predictive Analytics Workshop

Sensor Fusion Infrastructure for Urban Drone Traffic Management - Current Project

Urban Aerial Traffic Management ReimaginedInstead of, or in addition to planned corridors and ADS-B

Consider urban infrastructure that collaborates with Drones in a network and has an immune system for non-compliant Drones, where all sensors self-localize with trusted messaging (distributed ledger)

Optical, Acoustic, RADAR, LIDAR Sensor Fusion

Ignoring Spectrum Analysis (Many Current Products)

Sam Siewert AFRL - Predictive Analytics Workshop 9

NASA UPP/IPP

Wide-field Eyes

Narrow-field Eyes

DistributedAcoustic

RADAR

We use 2 roof-tops at ERAU NowThe Campus is our Lab (Labs)– Concept of IoT Smart Cities– Buildings can be situationally

aware

AXFAB - designed for Aerospace Experimentation

AC-1 - has roof-top weather station RADAR (being replaced), Tracking RADAR

Physics Observatory - Take-off location for testing

Most buildings are not designed well for this (e.g. STEM)

Acoustic Sensor go Anywhere

Sam Siewert AFRL - Predictive Analytics Workshop 10

AC-1

Observatory

AXFABEO/IR

AC-1RADARAll-sky

LaunchPoint

STEMAcousticEars

ERAU Prescott LocationClass-D Airspace (less than 5 miles from PRC Ernest A. Love Field) -4Km Campus to Runway

Semi-Rural, but Campus Emulates Office Park (Urban)

Work with Facilities, and Bldg. Mgr., Deans to get Permission

Power Line Hazards

FAA Waiver with Tower Required on Campus for sUAS flights

Sam Siewert AFRL - Predictive Analytics Workshop 11

https://www.google.com/maps

www.Bloomberg.com

Urban Drones - Challenge & SignificanceGrowth of sUAS and UAMSafety/Security ChallengesNASA UAM L4+, UTM and FAA UPP, IPP, ICAO UA SafetyProblem – Sharing AirspaceRegulations Sufficient?

Research and Development– Explore sensor fusion of ground

camera (visible and infrared) and acoustic passive sensing

– Compare to ground RADAR and flight LIDAR

– Address Public Concern, both Perceived and Real

– Enable Safe Urban UAS Operations - Trust, but Verify! Sam Siewert, ICARUS Group AFRL - Predictive Analytics Workshop 12

www.citylab.com/transportation

Phys.org NASA UAM

Drones Collaborate with Ground (Buildings)

© Sam Siewert, ICARUS Group Presented at AIAA SciTech 2019 - Fusion of Networked Sensors 13

All-sky camera (AC-1) coarse AZ, EL estimate narrows search space for EO/IR cameras (AXFAB)Acoustic array (sound intensity probe) in many locations on campusSimple Motion Detect with Classification (Plane, Drone, Bug/Bird, Other)Operates in real-time, wireless connections between acoustic nodes, all-sky, EO/IR

All-Sky Detection For Coarse AZ, ELFeasibility with 12 MP, 6 camera array shownMotion detect pixel registration for coarse AZ, ELCorrespondence to Acoustic coarse AZ, EL

Sam Siewert, ICARUS Group Presented at AIAA SciTech 2019 - Fusion of Networked Sensors 14

Step 1 - Camera that registers drone target (classified) over timeCamera Registering Drone or PlaneAll-sky camera coverage and FoV Model

Specific Camera (C1…C6) Pixel Registration Azimuth: Compute from X,Y registered pixel COM, Xbar and YbarAngle-off-CamAzimuth=( Xbar - [Xres/2]) / Xpixels-per-degreeAzimuth = CamAzimuth + Angle-off-CamAzimuthAzimuth = 41.36 degrees

Elevation: Camtilt + [(Yres-Ybar) / Ypixels-per-degree]Elevation=55.45 degrees

Acoustic Detection For Coarse AZ, ELDistributed processing microphone arraySound intensity probe feasibility for coarse AZ, ELCorrespondence to All-sky visual coarse AZ, EL

Sam Siewert, ICARUS Group Presented at AIAA SciTech 2019 - Fusion of Networked Sensors 15

EO/IR Identification Method – MV/ML

Machine Vision using SoC Linux Built-into each EO/IR Tracker– Salient Object Detection (R-CNN sensor fusion input pre-processing)

Shape, Behavior and Contrast/Color/Texture in Multiple BandsPerformance [ROC, PR, F-measure, confusion matrices]

– Real-Time Detection, Segmentation, Tracking, Classification, Identification

Comparing R-CNN and Deep Learning Methods to Traditional Machine Learning– Expert systems– Bayesian inference, Deep Belief Net– PCA [Principal Component Analysis]– SVM [Support Vector Machines]– Clustering [e.g. K-means]– GPU Accelerated DNN (cuDNN)– Supervised, Unsupervised learning

Leverage Open Source: ROS, OpenCV, PyBrain, PyML, MLpack, cuDNN, Caffe, Tensorflow

© Sam Siewert, ICARUS Group Presented at AUVSI 2018 16

Research References[for IoT, Sensor Fusion and AI]

Sam Siewert 17

References

Sam Siewert 18

All ICARUS Drone Net papers - http://mercury.pr.erau.edu/~siewerts/extra/papers/Drone-Net-Publications.pdf

S. Siewert, M. Andalibi, S. Bruder, J. Buchholz, S. Rizor, R. Fernandez, “Slew-to-Cue Electro-Optical and Infrared Sensor Network for small UAS Detection, Tracking and Identification”, AIAA SciTech [program, presentation], San Diego, January 2019.

S. Siewert, M. Andalibi, S. Bruder, I. Gentilini, A. Dandupally, S. Gavvala, O. Prabhu, J. Buchholz, D. Burklund, “Drone Net, a passive instrument network driven by machine vision and machine learning to automate UAS traffic management”, AUVSI Xponential poster, Denver, Colorado, May 2018.

S. Siewert, M. Andalibi, S. Bruder, I. Gentilini, J. Buchholz, “Drone Net Architecture for UAS Traffic Management Multi-modal Sensor Networking Experiments”, IEEE Aerospace Conference [presentation], Big Sky, Montana, March 2018.

S. Siewert, M. Vis, R. Claus, R. Krishnamurthy, S. B. Singh, A. K. Singh, S. Gunasekaran, “Image and Information Fusion Experiments with a Software-Defined Multi-Spectral Imaging System for Aviation and Marine Sensor Networks”, AIAA SciTech 2017, Grapevine, Texas, January 2017.

S. Siewert, V. Angoth, R. Krishnamurthy, K. Mani, K. Mock, S. B. Singh, S. Srivistava, C. Wagner, R. Claus, M. Demi Vis, “Software Defined Multi-Spectral Imaging for Arctic Sensor Networks”, SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, Baltimore, Maryland, April 2016.

S. Siewert, J. Shihadeh, Randall Myers, Jay Khandhar, Vitaly Ivanov, “Low Cost, High Performance and Efficiency Computational Photometer Design”, SPIE Sensing Technology and Applications, SPIE Proceedings, Volume 9121, Baltimore, Maryland, May 2014.

Piella, G. (2003). A general framework for multiresolution image fusion: from pixels to regions. Information fusion, 4(4), 259-280.

Blum, R. S., & Liu, Z. (Eds.). (2005). Multi-sensor image fusion and its applications. CRC press.

References

Sam Siewert 19

Liu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., & Wu, W. (2012). Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(1), 94-109.

Fiott, Daniel. "Europe and the Pentagon’s third offset strategy." The RUSI journal 161.1 (2016): 26-31.

Simone, G., Farina, A., Morabito, F. C., Serpico, S. B., & Bruzzone, L. (2002). Image fusion techniques for remote sensing applications. Information fusion, 3(1), 3-15.

Mitchell, H. B. (2010). Image fusion: theories, techniques and applications. Springer Science & Business Media.

Szeliski, R. (2010). Computer vision: algorithms and applications. Springer Science & Business Media.

Sharma, G., Jurie, F., & Schmid, C. (2012, June). Discriminative spatial saliency for image classification. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 3506-3513). IEEE.

Toet, A. (2011). Computational versus psychophysical bottom-up image saliency: A comparative evaluation study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11), 2131-2146.

Valenti, R., Sebe, N., & Gevers, T. (2009, September). Image saliency by isocentric curvedness and color. In Computer Vision, 2009 IEEE 12th International Conference on (pp. 2185-2192). IEEE.

Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. (2011, June). Image saliency: From intrinsic to extrinsic context. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 417-424). IEEE.

Liu, F., & Gleicher, M. (2006, July). Region enhanced scale-invariant saliency detection. In Multimedia and Expo, 2006 IEEE International Conference on (pp. 1477-1480). IEEE.

Cheng, M. M., Mitra, N. J., Huang, X., & Hu, S. M. (2014). Salientshape: Group saliency in image collections. The Visual Computer, 30(4), 443-453.

References

Sam Siewert 20

http://global.digitalglobe.com/sites/default/files/DG_WorldView2_DS_PROD.pdf

http://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf

Richards, Mark A., James A. Scheer, and William A. Holm. Principles of modern radar. SciTech Pub., 2010.

Brown, Christopher D., and Herbert T. Davis. "Receiver operating characteristics curves and related decision measures: A tutorial." Chemometrics and Intelligent Laboratory Systems 80.1 (2006): 24-38.

Wang, Bin, and Piotr Dudek. "A fast self-tuning background subtraction algorithm." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014.

Panagiotakis, Costas, et al. "Segmentation and sampling of moving object trajectories based on representativeness." IEEE Transactions on Knowledge and Data Engineering 24.7 (2012): 1328-1343.

Public SDMSI shared data web site for video sequences captured and used in two experiments presented in this paper - http://mercury.pr.erau.edu/~siewerts/extra/papers/AIAA-SDMSI-data-2017/

Perazzi, Federico, et al. "Saliency filters: Contrast based filtering for salient region detection." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.

Achanta, Radhakrishna, et al. "SLIC superpixels compared to state-of-the-art superpixel methods." IEEE transactions on pattern analysis and machine intelligence 34.11 (2012): 2274-2282.24Hou, Xiaodi, and Liqing Zhang. "Saliency detection: A spectral residual approach." 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007.

Global Contrast based Salient Region Detection. Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, Shi-Min Hu. IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI), 37(3), 569-582, 2015.

References

Sam Siewert 21

flightradar24.com, ADS-B, primary/secondary RADAR flight localization and aggregation services.

Birch, Gabriel Carisle, John Clark Griffin, and Matthew Kelly Erdman. UAS Detection Classification and Neutralization: Market Survey 2015. No. SAND2015-6365. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2015.

Drone Detection and Neutralization CompaniesLeading Drone Detection Companies

• Rohde & Schwarz Ardronis - Ardronis I• https://www.blacksagetech.com/• https://www.droneshield.com/• http://www.dedrone.com/en/• https://www.kongsberggeospatial.com/applications/argus-cuas• https://fortemtech.com/ - DroneHunter• https://echodyne.com/products/

List of Drone Detection and Counter UAS Products and Experiments• DJI Aeroscope• Drone Capture with Nets• Test at JFK by FBI• Dynetics - Counter UAS• SRC Gryphon Sensors Counter UAS• SPI Infrared Drone Detection• Industrial Camera Drone Detection• HGH Infrared Drone Detection• http://www.cerbair.com• Israel Defense Counter UAS• AARONIA Drone Detection

© Sam Siewert 22