Dr. nikos grammalidis (information technologies institute) “research on remote sensing and dete

www.firesense.eu www.biosos.euwww.outland-project.eu

Research on Remote Sensing and Detection/Management of Forest Fires in

the Information Technologies Institute: Research Projects FIRESENSE, BIOSOS

and OUTLAND”

Dr. Nikos Grammalidis and Dr. I. Manakos,Centre for Research and Technology Hellas /

Information Technologies Institute

ArcFuel Final Conference, Thessaloniki, 18 December 2013


Introduction

Firesense project

BIO_SOS project

OUTLAND project

Conclusions

Outline



Introduction: CERTH-ITI

The Information Technologies Institute (ITI) was founded in 1998 under the auspices of the General Secretariat of Research and Technology of the Greek Ministry of Development. Since March 2000, it is part of the Centre for Research and Technology Hellas (CERTH)

It became a European Centre of Excellence in 3D and Stereoscopic Imaging and Multimedia in 2001



Overview

Personnel: 16 affiliated professors, 7 researchers, 35 postdoctoral, more than 160 postgraduate researchers, 8 administration staff

More than 140 R&D projects funded by European Commission Programmes, more than 60 R&D projects funded by National Programmes and 100 Consulting Subcontracts with the Private Sector (I&T Industry)

More than 350 journal papers, 850 conference papers, 120 book chapters

http://www.iclass.info/

http://www.iclass.info/

http://eurongi.enst.fr/en_accueil.html























Research areas

ITI

Image and Signal Processing and Coding

Biomedical and Bioinformatics

Patten Recognition and Machine Learning

Human-Computer Interaction

Computer Vision

Social Network Analysis

Virtual and Augmented Reality

Multimedia analysis

Cultural and Educational Technology

e-Government

Geoscience and Remote sensing

Environment

Databases and Information Systems

Artificial Intelligence

Integrated

Commercial Solutions Communications and

Networking

Security and Surveillance



FIRESENSE projectFIRESENSE (Fire Detection and Management through a Multi-Sensor Network for the Protection of Cultural Heritage Areas from the Risk of Fire and Extreme Weather Conditions)

ENV.2009.3.2.1.2: Technologies for protecting cultural heritage assets from risks and damages resulting from extreme events, especially in the cases of fires and storms

Grand Agreement n°: 244088, STReP Project Project start: December 1st, 2009 Project duration: 36 months Project total cost: 3 609 027 € EC contribution: 2 697 092 €


www.firesense.eu www.biosos.euwww.outland-project.euESSC, Thessaloniki, Greece, 9-14 May 2011

FIRESENSE partners

TunisiaSUPCOM

GreeceCERTH

HMC

TurkeyBILKENT

BOGAZICITITAN

BelgiumXENICS

ItalyCNR

NetherlandsCWI



Why FIRESENSE? Need to protect cultural heritage and archeological

sites Majority of these sites in the Mediterranean region are

covered or surrounded by vegetation and this exposes them to an increased risk of fire.

Ancient Olympia (Aug. 2007), Marathon (Aug. 2009), the ancient Kameiros, Rhodes Island in 2008, the temple of Epikouros Apollo in 1998, three Monasteries of Mount Athos in 1990 etc.



Why FIRESENSE?

Early fire warning is the only way to avoid or minimize damages

Need to combine state of art sensing technologies in an integrated surveillance system



FIRESENSE main objectives Development of an automatic early

warning system to remotely monitor areas of archaeological and cultural interest from the risk of fire.

Taking advantage of recent advances in multi-sensor surveillance technologies using a wireless sensor network, optical and infrared cameras as well as local weather stations on the deployment sites

Improved fire propagation estimation and visualization



System architecture

Sensor Polling

Video-based Fire Detection

Data fusion

Estimation of Fire Propagation

GIS

IR Data ProcessingWeather Data Processing

External Weather Forecast

Area Fuel Model

Alarm Levels 1,2 ,..



Visible Cameras

12

Fire Detection

Smoke Detection



Video-based detection / Software platforms

Offline Software Platform Online Software Platform

Several new algorithms were developed or extended for flame/smoke detection using visible data

Increased detection rates and lower false positive ratios are achieved (compared to the literature)



MULTI-SPECTRAL IMAGING

Optical SWIR LWIR



Fire detection using IR sensors Fire detection algorithms based on several types of IR sensors

were developed:• LWIR image processing• SWIR image processing • Covariance features based IR Video flame detection

A Bimodal approach combining flame detection in LWIR with smoke detection in optical camera also yields promising results.

Other sensors PIR system for flame detection

Seismic system for wildfire detection



Wireless Sensor Network• We also used Wireless (temperature, humidity etc.) sensor

networks for detecting sudden local variations that should raise a fire alarm

• The network architecture is shown below• Each node communicates via a zigbee USB dongle.

• Cluster-heads form an infrastructure WiFi mesh backhaul, and each governs up to 20 end-nodes that are immediately accessed (0 hop) via zigbee.

• Cluster-heads are directly connected to the main gateway which communicates with the Control Center using HTTP



Control Centre

Different alarm types Locations and the status of

sensors on the map Statistical information/history from

the database Selection of one or more cameras

from the main screen (zoom in/out etc).

The optical and IR cameras rotate automatically to the area of interest in case of a fire alarm.

Estimation of the fire’s propagation using the present conditions (ignition point, weather conditions etc)



Fire propagation estimation

To help the forest fire management and identify critical situations, after detecting a wildfire, it is also important to estimate of the propagation direction and speed.

Factors effecting Fire Propagation Ignition point Topology (Slope and Aspect) Fuel Model Meteorological Conditions

• Wind• Fuel moisture – which

may depend on temperature, humidity, time of day, etc.



Fire propagation estimation Estimation of fire propagation (EFP) is based on the

popular BEHAVE fire behavior algorithm implementations (fireLib, Fire Behavior SDK).

A grid of cells is defined and a fire growth modeling algorithm is recursively applied.

Additional EFP extensions have also been implemented. Ignition point(s) may automatically be provided by the

detection software Topography parameters (slope

and aspect) are extracted from a Digital Elevation Model (DEM): Freely available data from CGIAR-CSI SRTM with a resolution of 90m are used.



Fuel ModelsThree sources of fuel data were used: The CORINE Land Cover (CLC) map that records 44 land

cover and land use classes which represent the major surface types across Europe.

Very high resolution satellite images (QuickBird) are used for vegetation classification.

Ground truth (or site survey) is often required for developing and testing satellite image processing algorithms and fuel modeling. Surveys for Kabeirion are (near Thebes) were made from a) G. Xanthopoulos and b) OMIKRON.



Vegetation Estimation Results(based on SVM classification)

Ground truth (Observation) Classified image Post-filtered classified image

Forest Trees Grass Water Shadow Road Built LawnBare soil

1Bare soil

2



A user-friendly 3D visualization software was developed using GIS information from Google EarthTM using C++ and third party libraries (Qt, Google Earth COM API, fireLib, Fire Behaviour SDK). Supports:

Multiple layers Multiresolution Wind interpolation Multiple ign. points Variable weather Prob. of crown fire Physical models etc.

Fire propagation visualization



EFP Evaluation: Field data for the Isthmia fire (near Corinth, 30/7/2008)

Real Burned area Fuel maps (mapped by expert forestry researchers)



Simulation results

Simulation results are seen to be consistent with the real observations/burned area (at the same time instants).



Pilot sites Five demonstrators were developed in four different countries

Thebes, Boeotia, Greece

Monteferrato-Galceti Park, Prato, Italy

Antalya, Turkey

Dodge Hall, Bogazici

University, Istanbul, Turkey

Temple of Water, Djebel

Zaghouan, Tunisia

http://www.firesense.eu/index.php?option=com_content&view=article&id=55&Itemid=103

http://www.firesense.eu/index.php?option=com_content&view=article&id=55&Itemid=103



Conclusions

FIRESENSE developed an automatic early warning system to remotely monitor areas of archaeological and cultural interest from the risk of fire.

The FIRESENSE system is a powerful cost-efficient approach that can be used for the protection of cultural heritage providing:

High reliability: The system utilizes different sensing technologies (CCTV cameras, PTZ, IR, temperature sensors).

Early detection of fire: Automatic detection of flame/smoke/rise in temperature.

Forest fire management: The system provides real-time information about fire’s extent/location through WSN, while it also estimates and visualizes its propagation based on the area’s fuel model, the local weather conditions and ground morphology.



BIO_SOS projectBIOdiversity Multi-Source Monitoring System: from Space

TO Species SPA.2010.1.1-04 - Stimulating the development of

GMES services in specific areas Grand Agreement n°: 263435, CP Project start: December 2010-November 2013 Project total cost: 3 159 510 € EC contribution: 2 476 363 €


Main Objective is

The development of a

pre-operational multi-modular ecological modelling system

suitable for multi-annual monitoring of NATURA 2000 sites and their surroundings. 1Dec 2010 -

15 Sept 2011

Biodiversity Multi-Source Monitoring System: From Space To Species


BIO_SOS working objectives

The development of pre-operational HR and VHR EO data processing and understanding techniques to provide as output- LC/LU and LC Change (LCC) maps as an improvement of GMES/ Copernicus core services

The development of an ecological modelling framework at both habitat and landscape level to combine EO and in-situ data for site monitoring. - Habitat maps as GHCs (Bunce, 2008)- Habitat change maps - Biodiversity indicators and their trends

as an extension of GMES/ Copernicus downstream–services


Drawbacks of crisp classification (need for fuzzification

approach):

– Noisy input data (i.e. satellite imagery, LiDAR, ancillary data,

etc.)

– Inaccurate rule thresholds

– Intolerance to changes in illumination conditions, seasonality

– Restricted transferability to similar sites, especially in

different geographical regions

Advanced fuzzy expert rules are additionally derived by the field surveys, for example: - Adjacency rules & Morphology of the patch area rules

BIO_SOS points of contribution to mapping


Robust classification framework

• Rule-based classification framework based on – Dempster–Shafer theory and – fuzzy logic

• Application fields– Detection of targeted entities

• Burned areas• Flooded areas• Particular forest types• Arable areas• Built or infrastructure areas

– Land cover / use mapping– Habitat mapping– Conservation planning– Site management


Habitat mapping example Use of Dempster–

Shafer theory To handle

uncertainty in expert rules

To handle missing data – Allowing multiple classes

Use of fuzzy logic To handle noisy

data To handle

inaccurate expert rules

The work presented herein was partially supported by the European Union Seventh Framework Programme FP7/2007-2013, SPA. 2010.1.1-04:616 “Stimulating the development of 490 GMES services in specific area”, under grant agreement 263435, project BIO_SOS: BIOdiversity Multi-Source Monitoring System: from Space To Species, coordinated by CNR-ISSIA, Bari-Italy.


Example of fuzzification and the D-S uncertainty handling (1/2)

Fuzzification example:

True area: 0.82ha Measured area: 0.76ha No fuzzification: Non applied rule Fuzzification: Rule applied by 40%

Vegetation adjacent to buildings, but with large area (>0.8ha) will be most likely (80%) within the Natural category (i.e., TRS, or HER)

0.8

1

0.5

0 x0.76

p

0.6 0.8 1

1

0.5

0

p

x0.76

0.4

LCCS to GHC mapping


Example of fuzzification and the D-S uncertainty handling (2/2)

___________D-S theory example_____________- Adj. to buildings: [0.5,0.8]………………………………….

- Area > 0.8ha: 40% (as shown in fuzzification slide)………………

- Rule validity: [0.2,0.32]……………………………………………………….

- Confidence in Natural: 80%………………………………

- Natural: [0.16,0.256] (0,256: 0,16+0,096 Or Natural: [0.16, 0,936] (0,936:0,256+0,68) ……. in the absense of an excluding for the Natural category rule

B15 50% B15 or B16 30%

Area > 0.8ha 40%

Natural Any other80% 20%

B15B15 or B16

B1620%

30%

50%

?

valid20% 12%

maybe valid

68%

invalidrule

Naturalmaybenatural

16% 4% 9.6% 2.4%

Any other maybe any other

68%?

Vegetation adjacent to buildings, but with large area (>0.8ha) will be most likely (80%) within the Natural category (i.e., TRS, or HER)

LCCS to GHC mapping


Improvement of classification results

Accuracy improvement by ≈ 12%

• Rule expressions:– A1: Rules with definite

outcome– A2: Rules with uncertainty

in the outcome

• Methods:– 6 fuzzy approaches (F1–

F6)– Crisp classification

approach (F0)

Reference (“A1” and “A2” are referred to as “B1” and “B2”, respectively):Z. Petrou, V. Kosmidou, I. Manakos, T. Stathaki, M. Adamo, C. Tarantino, V. Tomaselli, P. Blonda, M. Petrou, "A rule-based classification methodology to handle uncertainty in habitat mapping employing evidential reasoning and fuzzy logic", Pattern Recognition Letters, 2013, ISSN 0167-8655, http://dx.doi.org/10.1016/j.patrec.2013.11.002. To appear.



OUTLAND projectOpen protocols and tools for the Education and Training of Voluntary organisations in the field of Civil Protection, against Natural Disasters (forest fires) in Greece and Bulgaria

European Territorial Cooperation Programme Greece-Bulgaria 2007-2013 (2012: INTERREG IV A)

Lead partner: Municipality of Thermi Project duration: February 2012 – February 2014 Project total cost: 1,157,380 € (Funding by EU and

National sources)



The OUTLAND project

The overall objective of the OUTLAND Project is the creation of a complete system / framework for the education and training of the Firest Fire Volunteers Groups of Civil Protection Agencies in Greece and Bulgaria. This framework includes:

educational material, the necessary infrastructure, tools and mechanisms (with emphasis to novel informatics

applications) The aim is to establish an educational and training

framework for Civil Protection volunteers that will be available and useful after the end of the project.



CERTH/ITI role in OUTLAND

CERTH develops novel informatics tools within the project Development of novel user-friendly informatics applications that are based on

the communication between: a mobile application for Android Smartphones and a Control Center with fire simulation capabilities able to support various volunteer training

scenarios. Development Mobile Technology (Android SDK) and Fire Simulation

Techniques An e-learning platform (based on Moodle) for the education and training of

volunteers was also developed. CERTH is also responsible for the OUTLAND web page.



1. Report of Real Fire Incidents:

The volunteer is able to report Fire Incidents (via Internet or SMS) to a Control Center, which estimates the location of the Fire and informs the authorities via email about the Fire Incident.

2. Report of Vegetation Types:The volunteer is able to report (via Internet) the Vegetation Types of an area to the Control Center. A web tool for editing Vegetation Type was also developed.

3. Training Scenarios:The volunteer is able to participate (via Internet) in Training Scenarios organized by the Control Center.

SYSTEM FUNCTIONALITIES



SYSTEM COMPONENTS

A mobile application for Android Smartphones

Unit for receiving Fire Incident Reports Unit for organizing Training Scenarios

A Control Center consisting of 2 main Units :



1. Report Fire

Screen

2. Vegetation Report Screen

3. Training Scenarios

Screen

Main Screen:

The user selects the functionality he wants to use

Login Screen:

The user enters username &

password and sends login

request to the Control Center

Initial Screen

MOBILE APPLICATION



CONTROL CENTER: Fire Report Unit

Receives Fire Incidents, estimates the Fire Location and informs the authorities via email.



CONTROL CENTER: Training Scenarios Unit

Organizes Training Scenarios based on Fire Simulations.



FUNCTIONALITY 1: Fire Report(a) Report via Internet (Mobile Application)

The volunteer takes a picture of the Fire and sends to the Control Center via Internet :

Mobile Application Control Center

The volunteer sends via Internet the data for the Report of a Fire Incident to

the Control Center

1. The picture of the Fire.

2. The location of his device: values of Latitude / Longitude / Altitude.

3. The rotation angles ( = orientation) of the device camera: values of Heading / Tilt / Roll angles.



When the Control Center receives a Fire Report:

1. Shows the view of the device camera in Google Earth, using the location and rotation values it received from the fire report.

2. Shows the location of the volunteer in Google Maps.

3. Shows the Picture of the Fire

FUNCTIONALITY 1: Fire Report(a) Report via Internet (Control Center)



5. Shows the estimated Fire Location in Google Maps.6. Sends email to specific email addresses with the report data.

4. Calculates the Latitude and Longitude of the Center of Google Earth Window. This point is an estimation of the Fire Location.

FUNCTIONALITY 1: Fire Report(a) Report via INTERNET (Control Center)



FUNCTIONALITY 1: Fire Report(b) Report via SMS

1. The volunteer sends an SMS with the data (the picture is not sent) to another smartphone device which is located in the Control Center.

2. This device acts as a «gateway» between the networks of Mobile Telephony and the Local Internet of Control Center.

3. The «gateway» device receives the SMS, reads the data and sends it via Internet to the Server Computer of the Control Center.

Device «Gateway

»

The «gateway»

device sends the report

via Internet to the server

The volunteer sends SMS to the «gateway» device.

Server of the Control CenterVolunteer

device



FUNCTIONALITY 2: Vegetation Report (Server Application “FuelTypes”)

“Fuel Types” Application (developed by OMIKRON) is a plugin in the open source Quantum GIS software.

Vegetation types from EUNIS (2004) habitat classification system were used. Each Level III habitat type was mapped to one Scott-Burgan fuel model.

A rectangular grid with 50m x 50m cells is defined. The areas of interest in Municipality of Thermi are shown within the red boundaries.

The software allows editing the classification/fuel type of a cell.



Selection of Vegetation Type from a list with Scott and Burgan Vegetation

Types.

Send Vegetation Type and location coordinates to the

Control Center

1. The user selects from the Main Screenof the application the functionality of Vegetation Report.

2. The user selects from a list the Vegetation Type which best describes the vegetation of his location.

3. The user sends the Vegetation Typeand the location coordinates of his device (latitude, longitude) to the Control Center.

Selection of Functionality «Vegetation Report»

FUNCTIONALITY 2: Vegetation Report (Mobile Application)



FUNCTIONALITY 2: Vegetation Report(Control Center)

The Control Center receives the Vegetation Report and updates the records of Vegetation Types in the Database.

Thus, Fuel Maps for the areas of interest are created/updated.

The Fuel Maps allow us to estimate the fire behavior within each cell, in case of a fire.

Fuel Maps are used by our System for the estimation of the fire spread and flame length, when we run Fire Simulations to the Control Center.



FUNCTIONALITY 3: TRAINING SCENARIOS (Mobile Application)

During Training Scenarios, the volunteer is able to use the mobile application so as to:

The volunteer sends

periodically his location to the Control Center

1. Periodically report his location to the Control Center. For this purpose, he sends periodically - via Internet - his location coordinates to the Control Center (values of longitude, latitude, altitude).

Control CenterMobile

Application



FUNCTIONALITY 3: TRAINING SCENARIOS (Mobile Application)

2. Receive from the Control Center and see in his device (using Google Maps):

i. The positions of all fellow-volunteers involved in the scenario.

ii. Fire Simulations and Safe Routes, for the safe movement of a volunteer from a point Α to a point Β.

The user receives from the Control Center Fire

Simulations and Safe Routes

Control CenterMobile

Application



During Training Scenarios, the Control Center:

The positions of volunteers are displayed

in Google Maps (blue markers)

List with the volunteers of the scenario

1. Receives and shows the positions of the volunteers in Google Maps:

FUNCTIONALITY 3: TRAINING SCENARIOS (Control Center)



The Fuel Models of the area.

The direction / speed of the Wind.

An Ignition Point, which is the start point of the Fire.

2. Runs Fire Simulations

A Fire Simulation Software is used, which estimates the Fire Spread and shows the Fire Simulations in Google Earth.

Fire Simulation

The Fire Simulation Software accepts various input parameters:




Volunteers’ positions

Fire Simulation in Google Earth

Fire Simulation in Google Maps




3. Calculates Safe Routes for the safe move of the volunteers from a Point A to another Point B and shows the routes in Google Maps.


Start of the Route

Fire Simulation

Volunteers

Route

End of theRoute



3. Sends the Fire Simulation and the Safe Routes to the mobile devices .

Control Center Mobile Application

The Safe Routes are calculated with the use of the Routing Software pgrouting (http://pgrouting.org/).

The calculation of the Routes uses the results of the Fire Simulation in order to reject the routes which are not safe (are threatened by fire).

We use of existing open source data for the road network (OpenStreetMap).


http://pgrouting.org/



Routing Algorithms are used, in order to choose the shortest safe routes for the safe movement of the volunteers.


Statistical measurements can be provided (time for the movement of the volunteers from a point Α to a point Β).

Fire Simulation and Safe Route



Conclusions and Future Work Novel rule-based fuzzy habitat classification algorithms

have been developed within BIOSOS project FIRESENSE and OUTLAND projects allowed CERTH-

ITI to develop powerful fire detection and management tools

We are currently integrating the functionalities of both tools in a common framework/product

We indent to release of the core of the EFP software developed in FIRESENSE (with extensions by Dr. Xanthopoulos) as open source in the future

The interaction and collaboration between forestry and informatics experts was very fruitful and led to interesting results.


Thank youQuestions?