The University of Hong Kong Department of Computer Science

2020/21 COMP4801 Final Year Project

Detailed Project Plan

Body Temperature Measuring System with Smart Patrol Robot

Supervisor Dr. T.W. Chim

Members Lee Ka Fun, Katherine 3035474925

Leung Lok Yi, Harper 3035473359

Tse Man Kit, Jacky 3035477757

Content:

I. Abbreviations p.3

II. List of Figures p.4

1. Introduction p.5

1.1 Current Situation p.5

1.2 Our idea p.5

1.3 Motivation p.6

1.3.1 COVID-19 and fever p.6

1.3.2 Importance of temperature monitoring p.6

1.3.3 Reason on applying mobile thermal camera p.6

2. Objective p.8

3. Methodology p.9

3.1 Equipment set up p.9

3.1.1 Thermal Camera p.9

3.1.2 Normal Camera p.9

3.2 Algorithm and model design p.10

3.2.1 Infrared thermography p.10

3.2.2 Image Segmentation with thresholding methods p.11

3.2.3 Object selection with template matching p.12

4. Risk and challenges p.13

5. Budget p.14

6. Project Management p.15

6.1 Deliverables p.15

6.2 Schedule p.16

6.3 Division of Labour p.18

7. References p.19

I. List of Figures Figure 1. Thermograms of children's faces …………………………………………..……… p. 10

Figure 2. Demonstration of global grey level thresholding ………………….……………… p. 11

Figure 3. Demonstration of object selection for temperature calculation ……....…………… p. 12

II. Abbreviations

COVID-19 Coronavirus Disease 2019

HKAA Hong Kong Airport Authority

HKIA Hong Kong International Airport

IR Infrared radiations

WHO World Health Organization

1. Introduction

1.1 Current Situation In light of COVID-19, HKIA has adopted a series of preventive measures to safeguard public

health, including disinfection channels and autonomous sterilisation robots [1]. In addition,

arriving, departing and transit passengers are required to have their body temperature checked for

fever [2]. Despite strict measures of health screening being implemented in airports around the

globe[3], there are reported cases of passengers cheating thermal screening by taking fever

reducers[4][5]. To combat this situation, we aspire to utilise the use of the smart patrol robot to

enhance the future health measure in HKIA.

1.2 Our idea To ensure the health condition of every passenger before onboarding, the smart patrol robot can

be the second line of defense in the airport. After going through the health screening, passengers’

body temperatures will be monitored by the smart patrol robot that is keeping watch in the

boarding gate areas. When a suspected fever case is spotted, using the existing alert mechanism,

the robot will alert health control staff to handle the case by delivering images of the fevering

passengers and their location details.

1.3 Motivation

1.3.1 COVID-19 and fever

COVID-19 is an infectious disease originated from Wuhan, China, in December 2019 [6]. As of

26th of September, 2020, there have been at least 32.6 millions confirmed global cases of

COVID-19, causing more than 990,000 global deaths [7]. Declared to be a global pandemic by

the WHO [6], the spread of COVID-19 brings challenges to humanity. A study in June 2020

confirmed fever as the most prevalent symptoms among confirmed COVID-19 cases, where 78%

of patients reported fever [8]. Since air travel plays a vital role in the spread of COVID-19 [9],

speedy identification of airport users who are having a fever may help screen COVID-19

patients, which may in turn benefit the containment of the pandemic.

1.3.2 Importance of temperature monitoring

Besides COVID-19, fever is the major symptoms of other communicable diseases such as Ebola

and hand-foot-and-mouth disease. If those showing signs of a fever perform self quarantine and

refrain from having human interactions, control of such pandemic can become easier. However,

not all patients are aware of their fever, and some are reluctant to report themselves to the

authorities [10]. Therefore, it is more accurate and more reliable, to perform constant

temperature monitoring, instead of entrusting self-reporting.

1.3.3 Reason on adding mobile thermal camera

HKIA is currently using stationary thermal cameras for checking the body temperature of

travellers. Fixed cameras are set and airport uses are instructed to walk past the cameras, so that

their temperature can be assessed.

Since there is usually a large number of travellers walking past the stationary cameras at a time,

measurement may be obstructed due to blocking of the camera view. When mobile thermal

cameras are added for assistance, thermal images of airport users can be captured from different

angles. Also, since the robot is patrolling to different locations of the airport, camera blindspots

are avoided. Therefore, adding mobile thermal cameras to the existing stationary ones can

provide the airport with better monitor coverage. This can act as the second line of defense

against fevering passengers

2. Objective The project aims to build a mobile temperature measuring instrument for HKIA with the smart

patrol robot. The objectives of this project are listed below.

1. Incorporate a thermal camera in the robot

2. Develop a thermal image segmentation algorithm using different thresholding methods

3. Compare the result of using different thresholding methods for specific range of human

temperature

4. Develop a object selection algorithm for human faces in processed thermal images

5. Develop an algorithm for calculating the human temperatures from processed thermal

images

3. Methodology

3.1 Equipment set up

3.1.1 Thermal Camera

Thermal Camera converts heat energy released into visible light such that we can collect the data

to make analysis on the temperature of that object. When a thermal camera is used to check

humans, we are able to collect the body temperature of travellers.

By using thermography, the image collected from the camera, also known as thermogram, will

then be analyzed. Moreover, the image collected can be projected on the screen for further

interpretation.

3.1.2 Normal Camera

Other than the thermal camera, a normal camera will also be installed. Due to the fact that

thermal cameras can only produce thermal images as shown in Figure 1, it would be difficult for

the staff to identify the facial features of travellers. With the assistance of a normal camera, we

are able to capture the real appearance of the targeted passengers. The normal images combined

with the thermal images can help airport staff spot the fevering passenger more easily.

3.2 Algorithm and model design

3.2.1 Infrared thermography

According to the black body radiation law, all objects of temperature above absolute zero emits

infrared radiation [10]. The amount of radiation emitted by an object increases with its

temperature. Infrared thermography allows the temperature of human bodies to be captured in

the form of a thermogram (Figure 1), where the surface temperature of living things can be

monitored [11]. Since there is great correlation between thermal physiology and skin

temperature, in the medical field, infrared thermography is used to diagnose breast cancer,

diabetes neuropathy and peripheral vascular disorders, etc. [12]. With COVID-19 going on a

rampage, we are particularly interested in its capability to detect fever.

Figure 1 [13]. Thermograms of children's faces. The left shows normal body temperature, while

the right shows raised temperature around the eyes and nose, indicating a possible fever.

3.2.2 Image Segmentation with thresholding methods

Image segmentation is the process of partitioning an image into different parts with respect to

specific qualities, for instance, textures and pixel gray level[14]. Global grey level thresholding

is one of the most popular thresholding methods for image segmentation [15]. It is used to

differentiate between an object from its background by setting a certain grey level as the

threshold[16]. For example, if the grey level of a pixel goes above the threshold, it is classified as

the object and is changed to a certain colour, such as, white. If the grey level of a pixel goes

below the threshold, it is classified as the object and is changed to another colour, such as, black.

Figure 2 [17]. Demonstration of global grey level thresholding. The left shows an unpressed

image with the blue area as the background and the green area as the object. The middle shows

the histogram with x-axis showing the grey level and the y-axis showing the number of pixels. It

indicates that the background and the object can be differentiated if the grey level threshold is

set at 0.5. The right shows the result of global grey level thresholding.

This thresholding method can be leveraged to process thermal images from the footage recorded

by the thermal camera and obtain the size and positions of human bodies in the images.

3.2.3 Object selection with template matching

After image segmentation, images of human bodies can be differentiated from the background.

As we focus on measuring the temperature of the facial area, further object selection is needed to

sort out facial regions from the image. Template matching can be used to extract the facial area

by choosing elliptical and circular templates whose shapes are similar to typical face contour

[18].

(Figure 3a) Thermal image before processing

(Figure 3b) Thermal image after grey level thresholding

(Figure 3c) Thermal image after facial area is identified

Figure 3 [19]. Demonstration of object selection for temperature calculation.

As the facial area can be located, the average temperature can be calculated with reference to the

original thermal image.

4. Risk and challenges

Description Consequences Mitigation

Practical constraint:

Access control at the HKIA allows

only those taking a flight to enter

the airport

Testing in the airport may be

difficult

Seek assistance from HKAA so

that we can be allowed in the

airport under supervision.

Technical constraint:

Both the camera and measuring

subjects are moving constantly

It may be challenging to

capture the infrared radiation

from the subjects

Use software means (e.g.

programs/ scripts) to perform

adjustments and calibration.

Technical constraint:

There is a very thin margin between

normal body temperature and fever

(for example, forehead temperature

of 35.9°C is considered normal but

36.0°C may be considered as an

indication of fever)

False positive cases, or false

negative cases.

We need to calibrate our

functioning mechanism with the

thermal camera, such as

readjusting our cut-off

temperature, performing multiple

measurements and taking the

average, etc..

5. Budget

Item Description Quantity Price

Meridian Innovation -

MI0801 Camera Module

Evaluation Kit

The kits will be installed to the patrol

robot and used to capture thermal

images

2 ~$1400HKD

Travel expense Travel cost for travelling to airport for

research and data collection

1 $600HKD

Total: $2000HKD

As our project is supported by the Teaching Development and Language Enhancement Grant

(TDLEG), we were offered a maximum grant of HK$8,000 in the form of reimbursement upon

completion of the project. We are confident that our budget can be fully covered by the grant.

6. Project Management

6.1 Deliverables

Date Deliverables

4 October, 2020 Phase 1 (Inception) : • Detailed project plan • Project web page

11-15 January 2021 First presentation

- Progress updates

24 January, 2021 Phase 2 (Elaboration): • Preliminary implementation

- Purchase of cameras - Installation of thermal camera - Installation of normal camera - Implementation of the human face identification function

• Detailed interim report

18 April, 2021 Phase 3 (Construction):

• Finalized tested implementation

- Implementation of fever detection function - Development of the alerting mechanism (when a fever

case is discovered)

• Final report

19-23 April, 2021 Final presentation

4 May, 2021 Project exhibition

6.2 Schedule

Date Milestones

September 2020 Deliverables of Phrase 1: ● Detailed Project Plan ● Project Web Page

Review:

● Manual of the smart patrol robot provided by HKAA Research:

● Existing features and capabilities of the smart patrol robot ● Thermal camera for the project

October 2020 Research: ● Algorithms on image thresholding and object selection

Implementation:

● Basic navigation of the smart patrol robot ● Installation of thermal camera

November 2020 Research: ● Algorithms on image thresholding and object selection

Implementation:

● Data collection and processing ● Preliminary implementation of algorithms of image thresholding

Reviews:

● Evaluation of the result of different algorithms of image thresholding

December 2020 Preparation of Phrase 2: ● Interim report ● First presentation ● Product Demonstration in the first presentation

Implementation:

● The best performing image thresholding algorithms

● Preliminary implementation of algorithms of image thresholding ● Developing algorithms for object selections

January 2021 Deliverables of Phrase 2: ● Interim report ● First presentation ● Demo in the first presentation

Implementation:

● Evaluation of object selections algorithms ● Developing body temperature measuring algorithms

Research:

● Existing APIs of the robot to send out information

February 2021 Implementation: ● Testing of temperature measure algorithms ● Fine tune the image thresholding and object selection algorithms ● Information delivery framework of the robot

March 2021 Preparations of Phrase 3: Final report

● Final presentation ● Product Demonstration the final presentation ● Integration test and debug the whole project

April 2021 Deliverables of Phrase 3: ● Final report ● Final presentation ● Product Demonstration the final presentation ● [if selected] Preparation for the project competition

6.3 Division of Labour

Task Lee Ka Fun Leung Lok Yi Tse Man Kit Estimated Time spent

(days)

Beginning and Basic Research

Detailed Project Plan ✓ ✓ ✓ 10

Project Webpage ✓ 3

Review on technical document about the smart patrol robot provided by HKAA

✓ ✓ ✓ 3

Research on existing features of the smart patrol robot

✓ ✓ ✓ 3

Research on existing algorithm adopted in the smart patrol robot

✓ ✓ ✓ 3

Time Spent 22

Elaboration

Installation of thermal camera ✓ 2

Installation of normal camera ✓ 2

Research on algorithms on image thresholding and object selection

✓ 5

Design temperature measure algorithms ✓ ✓ ✓ 7

Implementation of temperature measure algorithms

✓ 10

Testing of temperature measure algorithms

✓ ✓ 5

Design image thresholding and object selection algorithms

✓ ✓ ✓ 7

Implementation of image thresholding and object selection algorithms

✓ ✓ 10

Testing of image thresholding and object selection algorithms

✓ ✓ 5

Preliminary implementation of algorithms of image thresholding

✓ ✓ ✓ 10

Design object selections algorithms ✓ ✓ ✓ 7

Implementation of object selections algorithms

✓ ✓ 10

Testing of Object selections algorithms ✓ ✓ ✓ 5

Interim report ✓ ✓ ✓ 10

First Presentation ✓ ✓ ✓ 10

Time Spent 95

Finalization

Integration test and debug the whole project

✓ ✓ ✓ 10

7. References [1] "HKIA Applies Advanced Technology to Step Up Disinfection Against COVID-19", Hong

Kong International Airport, 2020. [Online]. Available:

https://www.hongkongairport.com/en/media-centre/press-release/2020/pr_1446. [Accessed: 30-

Sep- 2020].

[2] "Department of Health - Health Control Measures for travellers at airport, seaport and land

boundary control points", Info.gov.hk, 2020. [Online]. Available:

https://www.info.gov.hk/info/sars/en/boundarycontrol.htm. [Accessed: 30- Sep- 2020].

[3] "Coronavirus: airports around the world carry out screenings", the Guardian, 2020. [Online].

Available:

https://www.theguardian.com/science/2020/jan/21/coronavirus-screenings-global-travelling-airp

ort. [Accessed: 30- Sep- 2020].

[4] "Passengers cheat flu scan with fever reducers", Abc.net.au, 2020. [Online]. Available:

https://www.abc.net.au/news/2009-06-15/passengers-cheat-flu-scan-with-fever-reducers/171426

4. [Accessed: 30- Sep- 2020].

[5] "A Chinese embassy in Paris tracked down a woman who gloated on social media about

cheating airport detection with a medicine that lowered her fever", Business Insider, 2020.

[Online]. Available:

https://www.businessinsider.com/wuhan-coronavirus-woman-avoided-airport-tests-travel-france-

2020-1. [Accessed: 30- Sep- 2020].

[6] Q&A on coronaviruses (COVID-19). (n.d.). World Health Organization. [Online].

Available:

https://www.who.int/emergencies/diseases/novel-coronavirus-2019/question-and-answers-hub/q-

a-detail/q-a-coronaviruses#:~:text=symptoms.

[Accessed Sep. 27, 2020].

[7] COVID-19 Dashboard. (n.d.). Center for Systems Science and Engineering at Johns Hopkins

University. [Online]. Available:

https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467

b48e9ecf6. [Accessed Sep. 27, 2020].

[8] M. Grant, L. Geoghegan, M. Arbyn, Z. Mohammed, L. McGuinness, E. Clarke, R. Wade.

(2020, June 23). “The prevalence of symptoms in 24,410 adults infected by the novel coronavirus

(SARS-CoV-2; COVID-19): A systematic review and meta-analysis of 148 studies from 9

countries”. [Online]. Available:

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0234765. [Accessed Sep. 27,

2020].

[9] H. Nakamuraa, S. Managi, “Airport risk of importation and exportation of the COVID-19

pandemic”, Transport Policy, vol. 96, pp. 40-47, Sep 2020.

[10] S. Stewart, R. Johnson. Blackbody Radiation: A History of Thermal Radiation

Computational Aids and Numerical Methods. Cleveland, Ohio: CRC Press, 2016.

[11] G. Gaussorgues, S. Chomet, Infrared Thermography, London, U.K.: Chapman and Hall,

1994.

[12] B. Lahiri, S. Bagavathiappan, T. Jayakumar, J. Philip. “Medical applications of infrared

thermography: A review”, Infrared Physics & Technology, vol. 55, pp. 221-235, July 2012.

[13] F. Ring, “The Herschel Heritage to Medical Thermography”, Journal of Imaging, vol. 2, pp.

13 April 2016.

[14] Zheng, Xiulian, et al. “Image Thresholding Based on Gray Level-Fuzzy Local Entropy

Histogram.” IEEJ Transactions on Electrical and Electronic Engineering, vol. 13, no. 4, 2018,

pp. 627–631.

[15] S. Sarkar and S. Das, "Multilevel Image Thresholding Based on 2D Histogram and

Maximum Tsallis Entropy— A Differential Evolution Approach," in IEEE Transactions on

Image Processing, vol. 22, no. 12, pp. 4788-4797, Dec. 2013, doi: 10.1109/TIP.2013.2277832.

[16] Parker, J.R. “Gray Level Thresholding in Badly Illuminated Images.” IEEE Transactions on

Pattern Analysis and Machine Intelligence, vol. 13, no. 8, 1991, pp. 813–819.

[17] "2.6.8.16. Histogram segmentation — Scipy lecture notes", Scipy-lectures.org, 2020.

[Online]. Available:

http://scipy-lectures.org/advanced/image_processing/auto_examples/plot_histo_segmentation.ht

ml. [Accessed: 30- Sep- 2020].

[18] H. F. Haghmohammadi, D. S. Necsulescu and M. Vahidi, "Remote measurement of body

temperature for an indoor moving crowd," 2018 IEEE International Conference on Automation,

Quality and Testing, Robotics (AQTR), Cluj-Napoca, 2018, pp. 1-6, doi:

10.1109/AQTR.2018.8402698.

[19] "A tutorial on Camera based Thermal Screening using Computer Vision", Medium, 2020.

[Online]. Available:

https://medium.com/@feat7/a-tutorial-on-camera-based-thermal-screening-using-computer-visio

n-d8b3304c08e3. [Accessed: 30- Sep- 2020].