Developing Computer-Vision Solutions for Sensing Problems ...€¦ · Developing Computer-Vision Solutions for Sensing Problems in Intelligent Systems A research vision on its challenges

Solar Energy Analytics Smart Traffic Analytics Neuroimaging Analytics Conclusion

Developing Computer-Vision Solutions forSensing Problems in Intelligent Systems

A research vision on its challenges and opportunities

Soumyabrata Devhttps://soumyabrata.github.io/

The ADAPT SFI Research CentreTrinity College Dublin, Ireland

University of DerbyApril 27, 2018

1 / 18


Where have I been, and where am I now?

Postdoc at The ADAPT SFI Research Centre, Trinity CollegeDublin, Ireland (2017-present).

Ph.D. in Electrical Engineering, NTU Singapore (2012-2017).

Visiting Doctoral Assistant, EPFL Lausanne (Aug-Dec 2015).

Engineer at Ericsson (2010-2012).

Undergrad in electronics engineering, NIT Silchar, India(2006-2010).

2 / 18


My high-level research interests

Research Vision

Identifying specific computational problems in intelligent systems;and solve them in a collaborative manner with the domain experts.

My research interests are mainly in the field of imageprocessing, computer vision, machine learning and deeplearning.

My early faculty research plan -• Solar Energy Analytics [60%]• Smart Traffic Analytics [30%]• Neuroimaging Analytics [10%]

Firm believer of open science 1 and research reproducibility 2.

1Advisor role at Overleaf, 207 Regent Street, London.

2P. Vandewalle, J. Kovacevic, M. Vetterli, Reproducible Research in Signal Processing – What, why, and how,

IEEE Signal Processing Magazine, vol. 26, no. 3, pp. 3747, 2009.

3 / 18


1 Solar Energy Analytics

2 Smart Traffic Analytics

3 Neuroimaging Analytics

4 Conclusion

4 / 18


How to Solve Nature’s Energy Crisis Using Imaging Data?

Solar (and other renewable) energy is necessary to beharnessed to reduce our dependency on traditional fossil fuels.

Can imaging data assist in solar analytics?• Traditionally done via satellite images.• Can we build intelligent systems to perform better?

Spatial resolution

Tem

por

al r

eso

lutio

nS

hort

term

Long

term

5 m

inut

es1

hour

1 da

y1

wee

k

500 meters 50 kilometers 500 kilometers

Whole Sky Imagers(WSI)

Satellites

5 se

cond

s

50 meters

Machine learning techniques on earth’s imaging data canprovide us better insights in solar analytics.

5 / 18


Intelligent System Design

WAHRSIS: Wide Angle High-Resolution Sky Imaging System

Easy-to-design model 3.Low-cost (2000USD) and high image-resolution (18MP).

(a) WAHRSIS (b) Image

Planning to further miniaturize these sky cameras 4 5foreffective deployment at multiple locations.

3S. Dev, F. M. Savoy, Y. H. Lee and S. Winkler, DIY Sky Imager For Weather Observation: A complete guide

to build a ground-based sky imager using off-the-shelf components with automatic cloud coverage computation,IEEE Signal Processing Society, 2016.

4S. Dev et al. WAHRSIS: A low-cost, high-resolution whole sky imager with near-infrared capabilities, Proc.

IS&T/SPIE Infrared Imaging Systems: Design, Analysis, Modeling, and Testing, May 2014.5

S. Dev et al. Design of low-cost, compact and weather-proof whole sky imagers for high-dynamic-rangecaptures, Proc. IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2015. 6 / 18


Challenges & Opportunities

Imaging data obtained from the ground can help us inunderstanding the solar irradiance fluctuations. 6

08:0009:00

10:0011:00

12:0013:00

14:0015:00

16:0017:00

18:000

200

400

600

800

1000

Sola

r ra

dia

tion [

Watt

/m2]

2015-12-07

Weather station

Clear sky model

10:30 (758 W/m2) 10:32 (283 W/m2) 10:34 (714 W/m2)

Fig: Impact of clouds on direct solar irradiance.

6S. Manandhar and S. Dev et al. Analyzing Solar Irradiance Variation From GPS and Cameras, IEEE AP-S

Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting, 2018 7 / 18



In what ways can the rapid fluctuations of the solar irradiancebe best captured?

Estimating solar energy 7 from captured sky camera images.

Hemispheric sampling Projection on the image.

Techniques from computational geometry 8 + machinelearning 9 can assist in harnessing such insights from thesky/cloud images.

7S. Dev et al. Estimation of solar irradiance using ground-based whole sky imagers, Proc. IEEE International

Geoscience and Remote Sensing Symposium (IGARSS), July 2016.8

S. Dev, A. Ghasemi, M. Vetterli, A. Scholefield, Point localization in Multi-camera system, Doctoralinternship report, Ecole Polytechnique Federale de Lausanne, Switzerland, 2015.

9S. Dev et al., Machine Learning Techniques and Applications For Ground-based Image Analysis, IEEE

Geoscience and Remote Sensing Magazine, 2016.

8 / 18



How can we obtain a reliable short-term solar energygeneration forecasting 10?

Input at t − 2’

Input at t’

Horizontal translation

Vertical translation

Actual at t + 2’

Predicted at t + 2’

Initial contacts with: Center for Energy Research, UC SanDiego and Solar Energy Research Institute of Singapore.

10S. Dev et al. Short-term prediction of localized cloud motion using ground-based sky imagers, Proc.

TENCON 2016 - 2016 IEEE Region 10 Conference, November 2016.

9 / 18





4 Conclusion

10 / 18


How Can Imaging Data Help in Assistive Driving?

Massive interest in the field of autonomous driving usingimage- and video- analytics.

Can we provide an end-to-end system in detecting the traffic-and road- signs from low-resolution car dashboard cameras?

Automatic detection of traffic- light and signage.

Possibility of a multi-modal approach with Near-Infra-Red(NIR) 11 + GPS 12, in addition to RGB camera images.

11S. Dev, F. M. Savoy, Y. H. Lee, S. Winkler, WAHRSIS: A low-cost, high-resolution whole sky imager with

near-infrared capabilities, Proc. IS&T/SPIE Infrared Imaging Systems: Design, Analysis, Modeling, and Testing,May 2014.

12S. Manandhar, Y. H. Lee, S. Dev, GPS Derived PWV for rainfall monitoring, Proc. IEEE International

Geoscience and Remote Sensing Symposium (IGARSS), July 2016.

11 / 18



My current project (at The ADAPT Centre, Dublin) involvesbillboard object detection and localization.

Several challenges in this task -• Deep learning models do not generalize across variouslocations.• Lack of publicly available datasets.• Based on only RGB camera images.

12 / 18



Color-based 13 object detection can help in identifying trafficsigns.

Techniques from set theory 14 + deep learning 15 are helpfulfor object detection.

Initial work with ai Robotics, an Anuva Ventures start-upcompany incubated in Singapore.

13S. Dev et al. Color-based segmentation of sky/cloud images from ground-based cameras, IEEE Journal of

Selected Topics in Applied Earth Observations and Remote Sensing, 2017.14

S. Dev et al., Rough Set Based Color Channel Selection, IEEE Geoscience and Remote Sensing Letters, 2017.15

S. Dev et al. CloudSegNet: A Deep Network for Nychthemeron Cloud Segmentation, Proc. IEEEInternational Conference on Image Processing (ICIP), 2018 (under review).

13 / 18





4 Conclusion

14 / 18


Can Medical Imaging Data Assist in UnderstandingChronic Schizophrenia?

Grey matter volumes in cortical and sub-cortical regions ofbrain 16 can provide insights in chronic schizophrenia. 17

How can we calculate the grey matter volumes in cortical andsub-cortical regions of brain using multimodal neuroimaging?

16From web.stanford.edu/group/hopes/cgi-bin/hopes_test/dementia-in-huntingtons-disease/

17Comparison of grey matter volume and thickness for analysing cortical changes in chronic schizophrenia: a

matter of surface area, grey/white matter intensity contrast, and curvature [Kong et al. Psychiatry Res. 2015].

15 / 18

web.stanford.edu/group/hopes/cgi-bin/hopes_test/dementia-in-huntingtons-disease/



Performed early experiments in this field and providedsuper-pixel imaging solutions for generation of Cortical andSub-cortical masks 18.

Slice 96 of .nii image Oversegmentation Three-level segmentation.

Initial work with National Neuroscience Institute, Singapore.

18S. Dev, Multimodal Neuroimaging: Analyzing MRI images using SPM, Technical report, 2017.

16 / 18


Where am I now, and what’s next?

Finishing my first industry-affiliated project with the ADAPTSFI Research Centre.

Ongoing projects: deep-learning based pipeline for automaticadvert integration in consumer videos; multi-modal approachusing cameras and GPS for intelligent systems design.

What’s farther ahead?: Studying the broad spectrum ofintelligent systems, providing image-assisted solutions, andalso expanding to other research directions.

17 / 18


Times when we were not researching on the clouds, we clickedselfies!

Thank You!

18 / 18

Documents

Developing Computer-Vision Solutions for Sensing Problems ...€¦ · Developing Computer-Vision Solutions for Sensing Problems in Intelligent Systems A research vision on its challenges