7
Remote respiratory sensing with an infrared camera using the Kinect TM infrared projector Andrew Loblaw, Dr. John Nielsen, Dr. Michal Okoniews Lakhani Ali Mazhar and ki Department of Electrical & Computer Engineering, University of Calgary, Calgary, Alberta, Canada AbstractThe need for an inexpensive and portable remote respiratory monitor is in particular demand in a hospital setting as the respiratory rate provides early warning for cardiorespiratory arrest. This paper proposes an inexpensive infrared (IR) camera combined with the Microsoft Kinect TM IR projector as a low-cost module for accurate measurement of respiratory rate. The IR camera utilizes a background subtraction algorithm to obtain the respiratory information of a patient in a bed. Issues with ill-defined feature points for the background subtraction algorithm are overcome by using the Kinect TM IR projector. The IR camera can easily detect the subtle respiratory motion of a prone or side- sleeping patient, even under covers. The IR camera system is experimentally validated in a home scenario as well as with a respiratory mannequin at the Foothills hospital in Calgary. Keywords: CV Remote Sensing, Background Subtraction 1. Introduction P ATIENTS in a hospital setting often require continuous monitoring of their vital signs as they are at a higher risk for mortality in which cardiorespiratory arrest is a common contributing factor. Early indication of a cardiorespiratory event are often indicated in the vital signs, specifically by an acceleration or slowing of the respiratory rate [1]. In a sleep apnea lab, an intensive care unit, or in an operating theater the respiratory rate of a patient is closely monitored. However, in a post-operative setting the respiratory rate of a patient is seldom monitored even though they are commonly administered narcotics for pain and are usually still under the influence of residual anaesthetic agents [2], [3]. A continuous respiratory monitoring method would offer the ability to recognize a cardiorespiratory event and intervene to prevent the event. While there are many methods of continuous respiratory monitoring currently used in hospitals none of them are utilized for long term monitoring in the post-operative set- ting. The oronasal thermistor, the nasal pressure cannula, and the inductive plethysmography belt which are common in any sleep lab are cumbersome, and inconvenient. These instruments all require contact with the patient, a dedicated technician to set up, and may be prone to detachment from the patient. In addition, the need for contact with delicate or sensitive patients, such as burn victims, is in- feasible. Many non-contact methods have been proposed, from microwave and millimeter-wave continuous wave (CW) Doppler radar [4], [5], [6] and ultra wideband (UWB) pulse [7], [8] to laser vibrometry [9], optical and infrared camera [10], [11], [12], and thermal cameras [13]. While microwaves and millimeter-waves have the potential to pass through bed sheets to measure respiration directly they lack spatial resolution and are therefore more prone to non- respiratory related interference. To obtain spatial information about a scene, cameras present themselves as the obvious alternative. A typical sleeping patient has very little visible light emitted on their person, so there are few well defined feature points making optical techniques infeasible. Thermal cameras are only able to measure exposed body parts, usually the neck and head, and are particularly expensive compared to CMOS camera technology. Infrared cameras can utilize active lighting that does not affect a sleeping sub- ject. An infrared camera using active structured lighting [12] has demonstrated the ability to obtain geometric information. The downside to the technique presented in [12] is the need for pre-calibration to obtain the accurate physical profile. Based on the previous development of respiratory sensing, a simplified respiratory sensing system using active struc- tured lighting without the need for calibration is proposed. A background subtraction algorithm is applied to the raw infrared (IR) camera video to obtain a respiratory signal. Next, a respiratory classification and moving Fourier trans- form algorithm is applied to obtain the respiratory rate. The algorithms and techniques proposed in this paper are validated with several different experiments: two human subjects, male and female in a home setting covered by bed sheets; and a respiratory mannequin in a hospital setting covered by bed sheets. 2. Background Subtraction Algorithm To detect the subtle motion of respiration a background subtraction algorithm is proposed. The Microsoft Kinect TM structured infrared light projector is used to provides feature points in the form of IR dots. A picture of the Kinect TM and the dot pattern is shown in fig. 1 and 2. A separate IR webcam is used to capture the data for processing. The overall setup is illustrated in fig. 3. As the subject inhales and exhales the projected IR dots will translate along the

Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

Remote respiratory sensing with an infrared camera using theKinect

TMinfrared projector

Andrew Loblaw, Dr. John Nielsen, Dr. Michal Okoniews Lakhani Ali Mazhar andki

Department of Electrical & Computer Engineering, University of Calgary, Calgary, Alberta, Canada

Abstract— Theneed for an inexpensiveand portable remoterespiratory monitor is in particular demand in a hospitalsetting as the respiratory rate provides early warning forcardiorespiratory arrest. This paper proposes an inexpensiveinfrared (IR) camera combined with the Microsoft Kinect

TM

IR projector as a low-cost module for accurate measurementof respiratory rate. The IR camera utilizes a backgroundsubtraction algorithm to obtain the respiratory informationof a patient in a bed. Issues with ill-defined feature pointsfor the background subtraction algorithm are overcome byusing the Kinect

TMIR projector. The IR camera can easily

detect the subtle respiratory motion of a prone or side-sleeping patient, even under covers. The IR camera systemis experimentally validated in a home scenario as well aswith a respiratory mannequin at the Foothills hospital inCalgary.

Keywords: CV Remote Sensing, Background Subtraction

1. Introduction

PATIENT’ S in a hospital setting often require continuousmonitoring of their vital signsas they areat ahigher risk

for mortality in which cardiorespiratory arrest is a commoncontributing factor. Early indication of a cardiorespiratoryevent are often indicated in the vital signs, specifically byan acceleration or slowing of the respiratory rate [1]. In asleep apnea lab, an intensive care unit, or in an operatingtheater the respiratory rate of a patient is closely monitored.However, in a post-operative setting the respiratory rate of apatient is seldom monitored even though they are commonlyadministered narcotics for pain and are usually still underthe influence of residual anaesthetic agents [2], [3]. Acontinuous respiratory monitoring method would offer theability to recognize a cardiorespiratory event and interveneto prevent the event.

While there are many methods of continuous respiratorymonitoring currently used in hospitals none of them areutilized for long term monitoring in the post-operative set-ting. The oronasal thermistor, the nasal pressure cannula,and the inductive plethysmography belt which are commonin any sleep lab are cumbersome, and inconvenient. Theseinstruments all require contact with the patient, a dedicatedtechnician to set up, and may be prone to detachmentfrom the patient. In addition, the need for contact with

delicate or sensitive patients, such as burn victims, is in-feasible. Many non-contact methods have been proposed,from microwaveand millimeter-wavecontinuouswave(CW)Doppler radar [4], [5], [6] and ultra wideband (UWB)pulse [7], [8] to laser vibrometry [9], optical and infraredcamera [10], [11], [12], and thermal cameras [13]. Whilemicrowaves and millimeter-waves have the potential to passthrough bed sheets to measure respiration directly they lackspatial resolution and are therefore more prone to non-respiratory related interference. To obtain spatial informationabout a scene, cameras present themselves as the obviousalternative. A typical sleeping patient has very little visiblelight emitted on their person, so there are few well definedfeature points making optical techniques infeasible. Thermalcameras are only able to measure exposed body parts,usually the neck and head, and are particularly expensivecompared to CMOS camera technology. Infrared camerascan utilize active lighting that does not affect a sleeping sub-ject. An infrared camera using active structured lighting [12]hasdemonstrated theability to obtain geometric information.The downside to the technique presented in [12] is the needfor pre-calibration to obtain the accurate physical profile.

Based on the previous development of respiratory sensing,a simplified respiratory sensing system using active struc-tured lighting without the need for calibration is proposed.A background subtraction algorithm is applied to the rawinfrared (IR) camera video to obtain a respiratory signal.Next, a respiratory classification and moving Fourier trans-form algorithm is applied to obtain the respiratory rate.The algorithms and techniques proposed in this paper arevalidated with several different experiments: two humansubjects, male and female in a home setting covered by bedsheets; and a respiratory mannequin in a hospital settingcovered by bed sheets.

2. Background Subtraction Algor ithmTo detect the subtle motion of respiration a background

subtraction algorithm is proposed. The Microsoft KinectTM

structured infrared light projector is used to provides featurepoints in the form of IR dots. A picture of the Kinect

TM

and the dot pattern is shown in fig. 1 and 2. A separateIR webcam is used to capture the data for processing. Theoverall setup is illustrated in fig. 3. As the subject inhalesand exhales the projected IR dots will translate along the

Page 2: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

chest. The motion of the dots, which is enhanced for smallerprojection angles (θ), can be detected by the separate IRcamera. This exaggerated motion of the IR dots is themotivation for separating the IR camera and the projector.

Fig. 1: Microsoft KinectTM

device with the IR projector (leftaperture) used for generating the feature points. The otherKinect

TMfeatures are not used.

Fig. 2: KinectTM

IR projector pattern showing the∼30000IR dots. Photo taken using the Kinect

TMbuilt-in IR camera

while the KinectTM

is facing a smooth flat wall.

Fig. 3: IR Camera with KinectTM

measurement setup.

The first step in the background subtraction algorithm isa spatial pre-filter. A5 × 5 Gaussian kernel is applied toeach image in an effort to remove some of the environment

and electronic noise. The kernel size of the Gaussian blur ischosen due to the physical nature of the IR feature points.The individual IR dots measure approximately 5 pixelssquared, so the Gaussian kernel smooths out noise featureswithout significant smearing of the IR dots.

The goal of the background subtraction algorithm is toclassify all static scenery and non-respiratory related activityas part of thebackground. The foregroundcan be extractedas the difference between the source and the backgroundvideo and will contain the respiratory motion. The back-ground subtraction algorithm implements an infinite impulseresponse (IIR) filter operating separately on each individualpixel. Although the effective motion is a lateral translationof the dots during respiration the change in each individualpixel’s intensity expresses this motion. Thus each individualpixel can be temporally filtered for its varying intensity. Thecombination of all of the pixels’ motion yields the respira-tory motion. The IIR filter impulse response is illustratedin fig. 4. A 0.1–1.5 Hz second order Butterworth band-stopfilter is used in this paper. This filter bandwidth is chosen toprevent any possible respiratory frequencies (6-90 breathsper minute (BPM)) from being classified asbackground.A low pass filter could also have been used, although theband-stop filter provides the benefit of filtering out higherfrequency noise and non-respiratory related motion.

After the background is computed using the IIR filter, it isa simple matter of computing the absolute value of a pixel-by-pixel image difference to generate the foreground. Oncethe foreground is established the respiratory motion is com-puted as the pixel-wise sum of all of the foreground pixels.The overall flow of the background subtraction algorithm isillustrated in fig. 5.

0 0.5 1 1.5 2 2.5 3−20

−15

−10

−5

0

Frequency [Hz]

Mag

nitu

de [d

B]

Fig. 4: Background subtraction algorithm IIR filter impulseresponse:0.1–1.5Hz 2nd order Butterworth band-stop filter.

Fig. 5: Overview of the background subtraction algorithmused to obtain the respiratory motion waveform.

Page 3: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

3. Respiratory Rate Computation Algo-rithm

In order to validate the background subtraction algorithmand obtain a useful metric from the respiratory motion, arespiratory rate computation algorithm is proposed. Thisalgorithm takes the respiratory motion from the backgroundsubtraction algorithm as input and produces the respiratoryrate as output.

The first step is to apply a0.1–1.5 Hz second orderButterworth band-pass pre-filter to the respiratory signal.The pre-filter attenuates the low and high frequency noiseand most importantly removes the large DC offset. Next amoving Fourier transform is computed. The peak frequencyof the Fourier transform for each time step is selected asthe respiratory rate. The parameters of the moving Fouriertransform are described in table 1.

Table 1: Moving Fourier transform properties

Moving Fourier Transform ValueWindow Length 512 samples,∼17 seconds @30

FPS

Window Type Hamming

Time steps between eachFourier transform

6 samples, 0.2 seconds @30FPS

In order to detect periods where the patient may com-pletely stop breathing entirely, an apnea classification algo-rithm is run in parallel with the moving Fourier transform.The apnea classification algorithm computes the average am-plitude of the respiratory signal and determines the minimumbaseline breathing amplitude required fornormal breathing.If the amplitude of breathing falls below this threshold, itis classified as an apnea and the breathing rate is set to 0BPM for as long as the breathing amplitude is below thethreshold. The apnea classification algorithm parameters aredescribed in table 2.

Table 2: Apnea classification algorithm properties

Apnea Classification ValueBaseline Amplitude CalculationWindow Length

Entire time series of respiratorydata

Regional AmplitudeCalculation Window Length

256 samples,∼8.5 seconds@30 FPS

Regional Amplitude ApneaQualification

≤50% of the baseline breathingamplitude

An overview of the respiratory rate calculation algorithmis shown in fig. 6. These parameters were used for all ofthe results in this paper; however, they are flexible andvariation of certain values can be beneficial. For example,if the patient’s breathing rate is highly variable, a shorterFourier transform time window may be advantageous sinceit would allow for faster tracking of the variable rate.

Fig. 6: Overview of the respiratory rate calculation algorithmused to obtain the respiratory rate

4. Results and DiscussionIn the following demonstrations, the performance of the

IR camera remote respiratory sensing system is shown andevaluated. The main metric is the accuracy of the respiratoryrate computed by the respiratory rate calculation algorithm,although visual qualitative analysis of the respiratory signalcomputed by the background subtraction algorithm is alsopresented. The accuracy of the respiratory rate is evaluatedby computing the root mean squared error (RMSE) betweenthe measured and the known respiratory rate.

4.1 Mannequin Trials

Fig. 7: Picture of Stan the respiratory mannequin.

The issue that is commonly encountered with most valida-tion techniques is the difficulty in establishing aground truthto compare against. Usually, the ground truth is establishedby employing a well known and trusted alternative measure-ment modality, such as a respiratory belt or a spirometer.This paper validates the system with measurements on arespiratory mannequin named Stan (Standard Man), picturedin fig. 7. Stan has the ability to very accurately control hisrespiratory rate so that no alternative measurement modalityis needed. In addition, Stan has a highly consistent breathing

Page 4: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

depth which should be reflected in the processed IR output.The IR camera background subtraction algorithm as well

as the respiratory rate calculation algorithm are demonstratedon the mannequin in fig. 8→11. In each of these trials themannequin’s respiratory rate is different:

1) Fig. 8 shows the Mannequin breathing at 11 BPM.2) Fig. 9 shows the Mannequin breathing at 15 BPM.3) Fig. 10 shows the Mannequin breathing at 22 BPM.4) Fig. 11 shows the Mannequin breathing at 40 BPM.

The individual respiratory events can be easily seen infigs. 8a, 9a, 10a, 11a. Interestingly, the mannequin has a dif-ferent breathing behaviour than humans. Stan’s breathsare sharp impulsive inhale-exhales rather than a sinusoidalinhale-exhale pattern more typical of human respiration. An-other interesting characteristic of the mannequin’s breathingis the additional periodic square-wave amplitude modulationwhich is detected by the background subtraction algorithm.This periodicity is most easily seen in figs. 10a and 11a. Thedetection of this periodicity demonstrates the ability of theIR system to detect subtle breathing mechanics.

The computed respiratory rate tracks the known respira-tory rate quite accurately with a maximum average RMSEof 1.5 BPM. The results for all the mannequin trials arepresented in table 3.

0 20 40 60 80 100 1200.5

1

1.5

2

2.5

3

3.5

4

4.5

5x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

14

16

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 8: Two minute trial - mannequin breathing at 11 BPM.Each individual respiratory event is clearly visible.

0 20 40 60 80 100 1202

4

6

8

10

12

14x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

14

16

18

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 9: Two minute trial - mannequin breathing at 15 BPM.

0 20 40 60 80 100 1201

2

3

4

5

6

7

8

9

10x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

5

10

15

20

25

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 10: Two minute trial - mannequin breathing at 22 BPM.

0 20 40 60 80 100 1200

1

2

3

4

5

6

7

8

9x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

5

10

15

20

25

30

35

40

45

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 11: Two minute trial - mannequin breathing at 40 BPM.

Table 3: Mannequin Trial Results

Respiratory Rate Error (BPM) % Error11 BPM 0.39 BPM 3.5%

15 BPM 0.71 BPM 4.7%

22 BPM 1.23 BPM 5.5%

40 BPM 1.55 BPM 3.9%

4.2 Human TrialsTo demonstrate the performance of the system in a more

realistic setting it is also tested on two different humansubjects in several different trials. For all of the humantrials the patient counted the total number of breaths overthe entire trial or specific intervals to compute the averagerespiratory rate. This average respiratory rate is used as theground truth for comparison, although there is expected tobe some amount of variability in the breathing rate sincetheir respiratory rate can vary breath to breath. The setupfor the human subject trials is shown in fig. 12.

The background subtraction algorithm performs well toobtain the respiratory signal of both human subjects. Theindividual respiratory events can easily be seen in all ofthe trials. Contrary to the mannequin trials, the respiratorysignals show a sinusoidal breathing pattern typical of humanbreathing and each inhale and exhale is detected separately.The respiratory rate calculation algorithm demonstrates theability to ignore some amount of non-respiratory relatedinterference as seen in fig. 13. The large disruptions in

Page 5: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

Fig. 12: Measurement Setup for human trial measurements.When the trials are taking place all lights are shut off andthe room is darker.

the respiratory signal of fig. 15 are caused by a handtwitch of the patient. Although these twitches disrupt therespiratory rate calculation the algorithm recovers within∼10 seconds. These trials also demonstrate the ability of theIR camera background subtraction algorithm and respiratoryrate calculation to accurately track the respiratory rate of apatient in both supine and side-sleeping postures.

0 20 40 60 80 100 1200

2

4

6

8

10

12

14x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

14

16

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 13: Female - supine, 24 breaths total = 12 BPM.

0 20 40 60 80 100 1201

2

3

4

5

6

7

8x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 14: Male - supine, 20 breaths total = 10 BPM.

0 20 40 60 80 100 1200

1

2

3

4

5

6

7

8

9x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 15: Female - side, 21 breaths total = 12 BPM.

0 20 40 60 80 100 1201

1.5

2

2.5

3

3.5

4

4.5

5x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 16: Male - side, 23 breaths total = 11.5 BPM.

Table 4: Human Normal Breathing Trial Results

Respiratory Rate Error (BPM) % Error12 BPM 1.09 BPM 9.5%

10 BPM 0.73 BPM 4.7%

12 BPM 1.6 BPM 13.3%

11.5 BPM 0.72 BPM 6.3%

The ability of the background subtraction algorithm andrespiratory rate calculation algorithm to detect apneas ispre-sented in figs. 17→19. An apnea can be defined as a periodwhere respiration decreases significantly for more than 10seconds. All of the apneas are at least partially detected bythe respiratory rate calculation algorithm, although visualinspection of the respiratory signals shows that all of theapneas can be easily qualified. This demonstrates there isroom for improvement of the respiratory rate calculationalgorithm to detect apneas faster.

Figs. 17 and 18 show a male and female subject lyingsupine, simulating apneas at 30-50 seconds and 90-100seconds by holding their breath. The worst error duringregular breathing is 3.25 BPM, while the average is around1.5 BPM. The large errors during the apnea periods aredue to the fact that the respiratory rate algorithm requiresbetween 3-10 seconds to qualify the apnea.

The last trial, shown in fig. 19 is of a male subject lyingsupine. The subject counted the number of respirations ineach 20 second interval and slowly decreased his breathing.

Page 6: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

In the last 20 seconds of the trial the subject held hisbreath to simulate an apnea. While the patients respirationis decreasing, the respiratory rate is measured to less than2BPM of error. When the patient stops breathing, the apneais detected within 3 seconds of starting.

0 20 40 60 80 100 1200.5

1

1.5

2

2.5

3

3.5

4

4.5

5x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

14

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 17: Female - supine, simulating an apnea between 30-50 seconds and 90-100 seconds. Average respiratory rate forperiods of normal breathing = 11.33 BPM.

Table 5: RMSE in BPM between the measured and estimatedrespiratory rate for different respiratory rate regions offig. 17.

Tim

e

8.5-30s 30-50s 50-90s 90-100s 100-111.5s

Rat

e

11.3 BPM 0 BPM 11.3 BPM 0 BPM 11.3 BPM

Err

or 1.40 5.57 1.57 4.31 2.68

12% – 14% – 24%

0 20 40 60 80 100 1200

1

2

3

4

5

6

7

8x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de

(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 18: Male - supine, simulating an apnea between 30-50seconds and 90-100 seconds. Average respiratory rate forperiods of normal breathing = 11.33 BPM.

Table 6: RMSE in BPM between the measured and estimatedrespiratory rate for different respiratory rate regions offig. 18.

Tim

e

8.5-30s 30-50s 50-90s 90-100s 100-111.5s

Rat

e

11.3 BPM 0 BPM 11.3 BPM 0 BPM 11.3 BPM

Err

or 3.25 4.30 0.80 9.13 0.56

29% – 7.1% – 5%

20 40 60 80 100 1201

1.5

2

2.5

3

3.5

4

4.5

5

5.5x 10

4

Time [seconds]

Res

pira

tory

Am

plitu

de(a) Respiratory Signal

20 40 60 80 1000

2

4

6

8

10

12

14

16

Time [seconds]

Res

pira

tory

Rat

e [B

PM

]

Calculated Breathing RateActual Breathing Rate (avg.)

(b) Respiratory Rate

Fig. 19: Male - supine, decreasing respiratory rate through-out trial until simulating an apnea from 100-120 seconds.

Table 7: RMSE in BPM between the measured and estimatedrespiratory rate for different respiratory rate regions offig. 19.

Tim

e

8.5-20s 20-40s 40-60s 60-80s 80-100s 100-111.5s

Rat

e

15 BPM 12 BPM 9 BPM 6 BPM 6 BPM 0 BPM

Err

or 1.47 1.19 1.82 0.79 0.64 2.98

10% 10% 20% 13% 11% –

5. ConclusionThe remote respiratory rate tracking system using an IR

camera with the Microsoft KinectTM

IR projector has beensuccessfully demonstrated. Several trials of a mannequinwith varying respiratory rates were shown where the av-erage error is less than 1.5 BPM. The system was alsodemonstrated on two human subjects for several differentarrangements. Normal respiration is accurately detected witherrors between 0.7 and 1.6 BPM. Scenarios in which thesubjects simulate an apnea are also demonstrated and thesystem never fails to detect at least part of the apnea. Theerror measurements for the human subjects are somewhatmisleading since the reference respiratory rate is really justthe overall average. The time domain plots of the respiratorysignal attest to the accuracy of the system as not a singlebreath was missed during normal operation. In terms ofinstantaneous breathing rate, the kernel of the system is

Page 7: Remote respiratory sensing with an infrared camera using the …worldcomp-proceedings.com/proc/p2013/IPC3072.pdf · 2014-04-16 · structured infrared light projector is used to provides

100% accurate. It is the respiratory rate calculation whichgives the perception of inaccuracy. The system is capable ofmeasuring subjects’ respiratory rate in a variety of lightingconditions and the posture of the patient is not criticalto obtaining an accurate measure of respiratory rate. Thetime domain results for the mannequin and human subjectsdemonstrate the ability of the system to differentiate be-tween different breathing mechanics. The IR camera systemrequires no calibration and is very practical for sleepingpatients since it does not emit or require visible light. Inaddition, it is cheap, requiring only an IR webcam and theKinect

TM. While this system is intended for easy deployment

in a post-operative hospital setting it can be easily designedfor home use, perhaps for pre-screening of sleep apnea.

The respiratory rate calculation algorithm performs wellfor normal respiration and can detect apneas, but it doesnot detect these apneas very fast and there is potentialto reduce the error. Beyond improving the respiratory ratecalculation algorithm, future work includes detection ofthe respiratory region and detecting non-respiratory relatedmotion. A hidden Markov model could be designed forclassification of patient breathing states and postures forimproved respiratory detection. Although not demonstratedin this paper, there is the potential for this technique tomeasure tidal volume if properly calibrated.

References

[1] M. DeVita, G. Smith, and S. Adam, “"Identifying thehospitalised patient in crisisâAI - A consensus conference onthe afferent limb of Rapid Response Systems,”Resuscitation,vol. 81, no. 4, pp. 375–82, Apr. 2010. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/20149516

[2] R. Robinson, C. Zwillich, E. Bixler, R. Cadieux, A. Kales, andD. White, “Effects of oral narcotics on sleep-disordered breathing inhealthy adults.”CHEST . . ., pp. 197–203, 1987. [Online]. Available:http://journal.publications.chestnet.org/article.aspx?articleid=1060018

[3] K. Pattinson, “Opioids and the control of respiration,”British journalof anaesthesia, vol. 100, no. 6, pp. 747–58, June 2008. [Online].Available: http://www.ncbi.nlm.nih.gov/pubmed/18456641

[4] A. Droitcour, O. Boric-Lubecke, V. Lubecke, J. Lin, andG. Kovacs, “Range correlation and I/Q performance benefitsin single-chip silicon Doppler radars for noncontact cardiopulmonarymonitoring,” Microwave Theory and Techniques, IEEE Transactionson, vol. 52, no. 3, pp. 838–848, 2004. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1273725

[5] C. Li, J. Ling, J. Li, and J. Lin, “Accurate DopplerRadar Noncontact Vital Sign Detection Using the RELAXAlgorithm,” Instrumentation and Measurement, IEEE Transactionson, vol. 59, no. 3, pp. 687–695, Mar. 2010. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5247107

[6] I. Mikhelson, S. Bakhtiari, T. Elmer, and A. Sahakian,“Remote Sensing of Heart Rate and Patterns of Respirationon a Stationary Subject Using 94 GHz Millimeter WaveInterferometry,” Biomedical Engineering, IEEE Transactionson, no. 99, pp. 1–7, Feb. 2011. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5709976

[7] I. Immoreev and T. Tao, “UWB radar for patientmonitoring,” Aerospace and Electronic Systems Magazine, IEEE,vol. 23, no. 11, pp. 11–18, Nov. 2008. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4693985

[8] J. Lai, Y. Xu, E. Gunawan, E. Chua, A. Maskooki, Y. Guan,K. Low, C. Soh, and C. Poh, “Wireless Sensing of HumanRespiratory Parameters by Low-Power Ultrawideband ImpulseRadio Radar,”Instrumentation and Measurement, IEEE Transactionson, vol. 60, no. 99, pp. 1–11, 2011. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5618561

[9] L. Scalise, I. Ercoli, and P. Marchionni, “Optical methodfor measurement of respiration rate,” inMedical Measurementsand Applications Proceedings (MeMeA), 2010 IEEE InternationalWorkshop on. IEEE, 2010, pp. 19–22. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5480208

[10] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancementsin noncontact, multiparameter physiological measurements usinga webcam.” IEEE transactions on bio-medical engineering,vol. 58, no. 1, pp. 7–11, Jan. 2011. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/20952328

[11] Y. Kuo, J. Lee, and P. Chung, “A visual context-awareness-based sleeping-respiration measurement system,”InformationTechnology in Biomedicine, IEEE Transactions on, vol. 14,no. 2, pp. 255–265, Mar. 2010. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5325889

[12] H. Aoki, K. Koshiji, H. Nakamura, Y. Takemura, andM. Nakajima, “Study on respiration monitoring methodusing near-infrared multiple slit-lights projection,” inMicro-NanoMechatronics and Human Science, 2005 IEEE InternationalSymposium on. IEEE, 2005, pp. 291–296. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1590006

[13] M. Yang, Q. Liu, T. Turner, and Y. Wu, “Vital sign estimationfrom passive thermal video,”IEEE Computer Society Conference onComputer Vision and Pattern Recognition, 2008. [Online]. Available:http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4587826