Machine Vision and Applications (2019) 30:867–874https://doi.org/10.1007/s00138-018-0980-5

SPEC IAL ISSUE PAPER

Amotion-based waveform for the detection of breathing difficultiesduring sleep

Samaher Lashkar1 · Heyfa Ammar1,2

Received: 6 February 2018 / Revised: 11 August 2018 / Accepted: 13 September 2018 / Published online: 27 September 2018© Springer-Verlag GmbH Germany, part of Springer Nature 2018

AbstractIndividuals who suffer from different sleep breathing disorders suffer from a wide range of serious health problems. Unfor-tunately, the rate of diagnosis is very low, and the existing breathing monitoring techniques are expensive, uncomfortableand time- and labor-intensive. The gold standard PSG is invasive, costly, technically complex and time-consuming. Towarddeveloping a non-contact sleep breathing monitoring system, this study presents a motion-based computer vision approachthat aims to detect breathing movements of the sleeping patient from infrared videos and map them into a waveform. The pro-posed waveform illustrates that each type of breathing difficulty has a specific pattern and hence can be easily distinguished.This facilitates identifying only suspicious periods during which physiological signals will be scored, instead of analyzingthe whole signals of 8h of sleep.

Keywords Breathing monitoring · Waveform · Motion estimation · Apnea · IR video recordings

1 Introduction

Adequate sleep plays an important role in human life. Recog-nizing the several effects of sleep disorders and poor sleep isgrowing. In that regard, sleep disorders or sleep deprivationhas a broad range of serious health consequences includ-ing raising the risks of diabetes [1], cardiovascular diseasesand hypertension [2], affecting daily work performances [3],decreasing brain activity, low heart rate, immune systeminsufficiency and memory loss [4]. The variety of symptomsalong with their broad consequences require classifying thedistinct types of sleep disorders with their respective crite-ria to get a proper diagnosis. According to the InternationalClassification of Sleep Disorders (ICSD-3) [5], sleep-relatedbreathing disorders which are the interest work of the presentstudy exist among 90 other distinct sleep disorders [3,5].The most common sleep disorder related to breathing isthe obstructive sleep apnea (OSA) [4], where the airflow

B Heyfa Ammarhammar@kau.edu.sa; heyfa.amar@gmail.com

Samaher Lashkarsamaherlashkar@gmail.com

1 FCIT, King Abdulaziz University, Jeddah, Kingdom of SaudiArabia

2 University of Tunis El Manar, Tunis, Tunisia

severs but the chest wall (rib cage and abdomen) move-ment perseveres, which implies a respiratory effort to facea closed upper airway. A less commonly form of apneais the central sleep apnea (CSA), where both airflow andchest movement sever, due to the breathing cessation. Inaddition, no effort is required to breathe again [4]. CSA iscentralized in the nervous system, as it fails in initiating sig-nals to arouse breathing. Thus, treating CSA is harder thanOSA [4]. Another form of breathing difficulty is the hypop-noea. It consists of an airflow reduction of 10s or more,related to arousal or desaturation of oxyhemoglobin [6]. Agold standard to diagnose sleep disorders is polysomnog-raphy (PSG) [7]. It measures large variables of the bodyfunctioning during an eight-hour-period of sleep. These vari-ables include respiratory effort, airflowmeasured by thoracicabdominal bands, bloodoxygen saturation bypulse oximetry,electromyography (EMG) used to measure muscle changes,electrooculography (EOG) that measures eyes movements,electrocardiography (ECG) used to measure heart electricsignals and electroencephalography (EEG) that measuresthe brain activity. However, PSG is technically complex,requires costly hardware and is time-consuming. Further-more, it is invasive and thus may disturb the patient’s sleepand compromise the diagnosis. A common less invasive andcost-effective alternative for diagnosing breathing difficultiescombines infrared (IR) video monitoring with pulse oxime-

123

868 S. Lashkar, H. Ammar

try [8]. It works by tracing the pulse oximetry, as the clinicianidentifies periods with decreased level of oxygen saturationmore than 4%. Then, the video data need to be analyzedand matched to these periods in order to reach a diagno-sis. An appropriate and well suitable alternative to reducethe intensive work of the clinicians and to ensure the com-fort of the patient is the automatic analysis of the overnightIR video recordings of the patient’s behavior. This has theadvantage of using low-cost hardware and being noninva-sive. Only fewworks investigated the use of video recordingsfor the purpose of diagnosing the sleep disorders [9–11]. Theapproach proposed in [9,10] aims to distinguish between fourevents: normal breathing, deep breathing, body movementsand OSA. The idea is to build two types of activity maps:a template T (t) and a map A(t). A template T (t) is built toidentify the spatial positions of moving pixels during a nor-mal breathing episode. Then, during each episode, an activitymap A(t) of moving pixels is constructed and compared toT (t) in order to recognize the corresponding event. The recog-nition of the current event as deep breathing, OSA or bodymovement is performed according to a matching score O(t)

defined by:

O(t) = w(t)1 /w

(t)2

w(t)1 = |T (t) ⋂

A(t)||A(t)|

w(t)2 = |T (t) ⋂

A(t)||T (t)| (1)

where |·| denotes the number of ones contained in the consid-ered sets, w(t)

1 represents the ratio of the moving pixels that

are absent in the template andw(t)2 is the ratio of common pix-

els to both T (t) and A(t). The presence of a specific event at atime t is then deducedwith respect to two empirically definedthresholds. However, the template T (t) highly depends onthe pose of the patient as it needs to be reconstructed withevery pose change. In addition, the level of noise is problem-atic in the constructed adaptive motion template and manyparameters need to be adjusted. Another method proposedin [11] aims to automatically detect CSA in newborn infantsby detecting breathing absence. First, a region of interest ismanually selected in the first image of the video, targeting thechest and abdomen. After that, the subtle breathing motion ismagnified by adopting a video magnification technique pro-posed in [12] that aims to magnify each pixel movementsand reconstruct each image of the video. Then, the sum ofpixel intensities in each image over the selected region ofinterest is calculated. At a given instant, if this sum doesnot exceed a predefined threshold, then a breathing absenceis concluded. However, this approach is sensitive to noiseand to the level of illumination, and hence, the used camerashould be of high quality to provide an accurate monitoring

of the babies subtle respiratory motion. It is worth pointingout that the present study extends a preliminary version pro-posed in [13] that aims to automatically detect OSA fromIR recordings. The detection of OSA depends on detectingthe breathing movements by using a robust optical flow esti-mation in order to alleviate the level of noise in IR images.Then, the summed motion magnitudes are considered as animportant feature to differentiate between normal breathingand OSA events. The obtained summed motion magnitudesduring OSA are noticed having atypical and isolated val-ues compared to those obtained during normal breathing.Hence, they are identified using a statistical test for outliersdetection. Although this method proved its efficiency andaccuracy in locating OSA events, it assumes that no bodymovements were present during the patient’s sleep. Besidesthat, the detection was only for OSA events. Based on thefeature proposed in [13], the present study proposes a newwaveform that allows to visually locate the suspected periodsof time that include different events of breathing difficultiesduring sleep such as OSA, CSA, hypopnoea besides bodymovements. These events are detected based on the analy-sis of real IR video recordings. It has the advantage of notdepending on the patient pose. Furthermore, we demonstratethat a breathing disorder is characterized by a specificmotionpattern and can be recognized through it. Thus, the proposedwaveform facilitates identifying suspicious periods by theclinicians based on a shape of pattern that reflects each type ofthe breathing difficulties. The remainder of the paper is orga-nized as follows: Used material and the proposed method aredescribed in Sect. 2, followed by some experimental resultsin Sect. 3 and discussion in Sect. 4. Finally, some conclusionsare drawn in Sect. 5.

2 Material andmethods

2.1 Data set

The data are provided by the sleep medicine and researchcenter at King Abdulaziz University Hospital, KSA. Theprovided data include five participants suffering from dif-ferent levels of apnea. The data corresponding to eachpatient consist of PSG signals and between 10 and 11 40-minute-infrared-video recordings. The IR video recordingsare captured using a fixed HD monitoring camera, AxisP1357 provided with an IR illuminator source. The cameraallows to capture 20 images per second at a spatial resolu-tion of 2048×1536 pixels and was placed in front of the bedto capture its entirety. All participants were informed andprovided their approval before participation. To evaluate thereliability of the proposed waveform, the visual interpreta-tion of this latter is compared to the diagnosis provided byan expert clinician after scoring the PSG signals.

123

Amotion-based waveform for the detection of breathing difficulties during sleep 869

Motion

Estimation Quantization

Apnea

Episodes

ROIMotion

{M (t,t+1)sub }

t

ImageSequence

ROI

selection

Fig. 1 Block diagram of the proposed approach

2.2 Proposed technique

The movements exerted by the patient during his sleep andtheir patterns are closely related to the disorder he suffersfrom. Hence, our rationale is to consider the motion mag-nitudes as an important feature to deduce a waveform. Theproposed waveform is intended to help clinicians to visu-ally recognize the different forms of breathing difficultiesencountered during sleep instead of analyzing many PSGsignals of many sleep hours. Before performing any analy-sis, the video recording is partitioned into a set of N imagesI (t), at each instant t = 1, . . . , D for the whole period D ofsleep. The proposed technique is composed of three modulesas shown by the block diagram in Fig. 1 and is detailed inwhat follows.

2.2.1 Selecting a region of interest

The aforementionedbreathingdifficulties events are reflectedby a noticeable change in the motion magnitude valueseither increased or decreased depending on the event andthe patient’s behavior during it. These changes in the motionmagnitudes are mainly related to the upper body parts, somespecific points of the face such as nose and mouth and chest.Thus, in order to reduce the number of computations, ouranalysis is restricted to a bounding rectangular region ofinterest (ROI) denoted by I (t)

sub of size Wr × Hr manuallyselected from an image I (t) of the video. It is worth notingthat the widthWr of the ROI and its height Hr are specific tothe patient and are selected in a way to include, in the ROI,the head, the chest, the mouth and the other side of the bed incase the patient moves. The other parts of the image do notcontribute in the detection, even though they can be used asa whole but will cause more computations.

2.2.2 Motion estimation

Any motion estimation technique could be applied for thegeneration of the waveform. In the present work, an opticalflow estimation technique robust to the presence of noise isused and is explained in what follows.

Robust optical flow estimation An optical flow (OF) estima-tion technique is a 2-D approximation of the displacements

occurred between two images. This technique is carried outon a set of sequential images representing the entire video inorder to obtain spatial information of the patient movements.The OF formulation depends on a couple of constraints:spatial smoothness and brightness constancy. The spatialsmoothness implies that neighboring pixels within a regionR of the image belong to the same object and thus havealmost the same motion. The brightness constancy impliesthat small regions persist during a period of time even iftheir positions change. A robust formulation considering theaforementioned two constraints is expressed in its spatiallydiscrete form by [14]:

E(u, v) =∑

(i, j)∈R(ρD(I (t)sub(i, j) − I (t+1)

sub (i + u(i, j), j + v(i, j)))

+γ [ρS(u(i, j) − u(i+1, j)) + ρS(u(i, j) − u(i, j+1)) (2)

+ρS(v(i, j) − v(i+1, j)) + ρS(v(i, j) − v(i, j+1))])

where u and v are, respectively, the horizontal and verti-cal components of the OF field estimated between I (t)

sub and

I (t+1)sub , γ controls the relative importance of the data and thespatial smoothness terms and ρD and ρS are penalty func-tions related to these two terms.

As the considered IR videos are noisy, using a robustpenalty function for both terms helps in reducing the impactof the noise in the OF estimation process [15]. Geman–McClure function ρGM that is used in this work is an exampleof a robust function. It is defined for all x ∈ R and for allσ ∈ R

∗+ by:

ρGM(x, σ ) = x2

x2 + σ.

where σ is a scale parameter. The motion estimation pro-cess is applied between each pair of successive images I (t)

sub

and I (t+1)sub giving rise to the estimated horizontal and verti-

cal displacements u(i, j) and v(i, j) at each spatial position(i, j) ∈ [1,Wr ]×[1, Hr ], stored in a matrix M (t,t+1)

sub of sizeWr × Hr × 2.

2.2.3 Motion quantization

Asproposed in our previouswork [13], the considered featureis the sum of motion magnitudes of all pixels located withinthe region of interest. The sum of motion magnitudes Sm(t)

at each time t is calculated as:

Sm(t) =∑

(i, j)∈I (t)sub

√

(u(i, j)2 + v(i, j)2). (3)

123

870 S. Lashkar, H. Ammar

Fig. 2 Illustration of a CSA event identified through the waveform

The set of the summed motion magnitudes for all t =1, . . . , N is expressed by S = {Sm(t)}t . Figure 2 illustratesthe generated waveform for an excerpt of a video duringwhich the moving pixels of periodic motion magnitudesreflect a normal breathing with a noticeable decrease to havenear zero magnitudes reflecting a CSA event.

The variation in motion magnitudes during OSA, normalbreathing, body movement and CSA events is illustrated inthe histogram of Fig. 3a. For the sake of clarity, the curverelated to the body movement is excluded in Fig. 3b to bettervisualize the distributions of the other events. According tothe definition of OSA, we expect that the apparent motionof a patient during this event has higher magnitude than hismovements caused by a normal breathing episode when nobody movement is present during both periods. Regardingthis, it is clearly depicted that the patient movements dur-ing normal breathing whether inhale or exhale have lowermotion magnitudes compared to those during an OSA event.For hypopnoea, the histogram illustrates the small motionmagnitudes of partial breathing.

3 Results

Real IR video recordings corresponding to 8h of sleep areused to evaluate the reliability of the proposedwaveform. Fora reference standard, apnea events are marked manually bythe sleep laboratory technicians based on PSG-scored signalsand used to evaluate the output of the proposed approach.As shown by the waveforms depicted in Figs. 2, 4, 5, 6and 7, each event could be differentiated as each of themhas a remarkable pattern. These results are generated fora severe patient that suffers from OSA, CSA, and hypop-noea according to the analysis done by the clinician. Thisdiagnosis well matches the patterns presented in the curves.Figure 2 illustrates the presence of a CSA event revealed

Motion magnitude0 2 4 6 8 10 12 14 16 18 20

Num

ber o

f pix

els

× 10 4

0

0.5

1

1.5

2

2.5

3OSACSAHypopnoeaBody movementNormal breathing

(a)

Motion magnitude0 0.5 1 1.5 2 2.5

Num

ber o

f pix

els

× 10 4

0

0.5

1

1.5

2

2.5

3OSACSAHypopnoeaNormal breathing

(b)

Fig. 3 Illustration of the variation of motion magnitudes during differ-ent sleep events

Fig. 4 Illustration of a body movement as identified by the proposedwaveform

123

Amotion-based waveform for the detection of breathing difficulties during sleep 871

Fig. 5 Illustration of an OSA event as identified by the proposed wave-form

Fig. 6 Illustration of normal breathing as identified by the proposedwaveform

by near zero values of Sm(t). In contrary to CSA, Fig. 4shows high values of the pattern due to a body movementas the patient was moving his arm above his head accord-ing to the video. Figure 5 illustrates the presence of anOSA event marked by high motion magnitudes. It is worthpointing out that as a body movement and an OSA eventboth lead to high values of Sm(t), the distinction betweenthem is possible through the visualization of the correspond-ing patterns. In fact, as an OSA event forces the patient tomake effort to resume breathing [6,16], this effort is reflectedby a continuous increase in the values of Sm(t) followedby a continuous decrease in them when the patient suc-ceeds to resume his normal breathing. A hypopnoea eventoccurs following a pre-event that initiates it. It is character-ized by a reduction in the breathing level of at least 30%and lasts for more than 10s. The waveform shows wellthis behavior as depicted in Fig. 7 where the pre-event isillustrated by relatively high values of Sm(t) followed by a

Fig. 7 Illustration of hypopnoea events as identified by the proposedwaveform

Fig. 8 Normal breathing detection in comparison with the ground truth(reference)

Fig. 9 Detection results related to a patient with a severe hypopnoea, amoderate CSA and no OSA

decrease of about 30%. The duration of this decrease exceeds10s.

The results are evaluated by comparing them with theground truth. In Fig. 8, the same episode of normal breath-ing is obtained matching the same starting time and durationas the episode in the ground truth. Figure 9 illustrates thedetection results of some events related to a patient suffer-

123

872 S. Lashkar, H. Ammar

Fig. 10 Body movement detection related to a patient suffering from asevere apnea

Fig. 11 Body movement detection related to a patient suffering from asevere hypopnoea and a moderate level of CSA

ing from a severe hypopnoea, a moderate CSA, whereas noOSA is present according to the PSG scoring results. Thevisual analysis of the proposed waveform shows a success-ful detection of a hypopnoea episode matching the startingtime and duration as the ground truth. After 2 s, a minor headmovement is also detected when the patient was trying tobreathe. Large body movements of 1-min duration are cor-rectly detected matching the ground truth as illustrated inFig. 10. Figure 11 illustrates two minor body movementsdetected correctly, one lasts for 17 s and the next one lasts for2 s. Figure 12 illustrates a hypopnoea episode that is detected19s earlier than the one in the ground truth but with the sameduration. These results correspond to a patient suffering froma mild level of hypopnoea. Furthermore, minor body move-ments are detected correctly such as the one in Fig. 13, andit lasts for only 2 s and matches the same event in the groundtruth.

4 Discussion

Motivated by the relationship between the motion magni-tudes and the breathing behavior explained in Sect. 1, awaveform is proposed that helps diagnosing different typesof breathing difficulties through the visual analysis of the

Fig. 12 Hypopnoea detection results related to a patient with a mildhypopnoea

Fig. 13 Body movement detection results related to a patient with amild hypopnoea

proposed waveform. The main advantage of the waveform isthat it provides a way to associate a specific pattern to each ofthe following breathing events: normal breathing, OSA, CSAand hypopnoea in addition to the possibility of recognizinga body movement. Furthermore, the temporal localizationof each event is possible. In contrary to the state-of-the-arttechniques that focus the study on a specific type of breathingdifficulty, the approach we propose allows the identificationof several types of it. This is compliant to the real situations asa patientmay undergomore than one type of breathing duringhis sleep. An illustration of the different events is depicted inFig. 14,where they are juxtaposed to better visualize their dif-ferences. The evaluation of the proposed approach is carriedout by comparing both the event recognition and its tempo-ral localization, to the PSG scoring results performed by theclinical experts. The results show the reliability of the pro-posed technique. Nevertheless, the proposed waveform canbe improved to better distinguish between a breathing eventand a body movement as this latter can vary from a subtle toa remarkable movement and so may lead to several rangesof the motion magnitudes. An automatic recognition of eachbreathing event is also to be emphasized in the future work.

123

Amotion-based waveform for the detection of breathing difficulties during sleep 873

Time in seconds0 40 80 120 160 200 240 280

Sm×105

0

2

4

6

8

10

12

14 Normal Breathing Body Movement Hypopnoea Central Sleep Apnea Obstructive Sleep Apnea

Fig. 14 Illustration of a waveform related to different sleep events: normal breathing, body movement, hypopnoea, central sleep apnea andobstructive sleep apnea

5 Conclusion and future work

A non-contact sleep breathing monitoring system is pro-posed. It consists of a waveform generated by the summedmotion magnitudes that allows to visually recognize thebreathing events through their respective motion patterns.Hence, locating the suspected periods of time that includeonly those events is important as it allows to restrict thescoring effort of the clinician to these periods of time. Eventhough the visual detection is helpful, an automatic detectionis necessary to further speed up the diagnosis. This featurewill be investigated in the future work and validated using adatabase of much more patients.

Acknowledgements We would like to acknowledge and thank KingAbdulaziz City for Science and Technology (KACST) represented inGrants Programs for Universities and Research Centers (GPURC) foradopting this scientific research under the Graduate Research Programwith reference number PS-38-2009 and supporting us with the requiredhardware.Wealso thank the sleepmedicine and research center (SMRC)at King Abdulaziz University Hospital (KAUH) for their cooperationand support in providing us with the required material and ground truth.

References

1. Spiegel, K., Knutson, K., Leproult, R., Tasali, E., Cauter, E.V.:Sleep loss: a novel risk factor for insulin resistance and type 2diabetes. J. Appl. Phys. 99(5), 2008–2019 (2005)

2. Gangwisch, J.E., Heymsfield, S.B., Boden-Albala, B., Buijs,R.M., Kreier, F., Pickering, T.G., Rundle, A.G., Zammit, G.K.,Malaspina, D.: Short sleep duration as a risk factor for hyperten-sion analyses of the first national health and nutrition examinationsurvey. Hypertension 47(5), 833–839 (2006)

3. Altevogt, B.M., Colten, H.R.: Sleep Disorders and Sleep Depriva-tion: AnUnmet Public Health Problem. National Academies Press,Washington (2006)

4. Sesanker, C.: Human Sleep Disorders: Apnea and Its DiagnosisUsing the ECG, Ph.D. dissertation,Worcester Polytechnic Institute(2010)

5. A.A. of Sleep Medicine et al.: The International Classification ofSleep Disorders: Diagnostic and Coding Manual. American Acad.of Sleep Medicine (2005)

6. Hsu, A.A., Lo, C.: Continuous positive airway pressure therapy insleep apnoea. Respirology 8(4), 447–454 (2003)

7. Tan, H.-L., Gozal, D., Ramirez, H.M., Bandla, H.P., Kheirandish-Gozal, L.: Overnight polysomnography versus respiratory polyg-raphy in the diagnosis of pediatric obstructive sleep apnea. Sleep37(2), 255–260 (2014)

8. Gederi, E., Clifford, G.D.: Fusion of image and signal processingfor the detection of obstructive sleep apnea. In: IEEE–EMBS Inter-national Conference on Biomedical and Health Informatics (BHI),pp. 890–893. IEEE (2012)

9. Wang, C.-W.: Video Monitoring and Analysis of Human Behav-ior for Diagnosis of Obstructive Sleep Apnoea. Ph.D. dissertation,University of Lincoln (2009)

10. Wang, C.-W., Hunter, A., Gravill, N., Matusiewicz, S.: Uncon-strained video monitoring of breathing behavior and application todiagnosis of sleep apnea. IEEE Trans. Biomed. Eng. 61(2), 396–404 (2014)

11. Sharma, S., Bhattacharyya, S.,Mukherjee, J., Purkait, P.K., Biswas,A., Deb, A.K.: Automated detection of newborn sleep apnea usingvideomonitoring system. In: 2015Eighth International ConferenceonAdvances in PatternRecognition (ICAPR), pp 1–6. IEEE (2015)

12. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Free-man, W.: Eulerian Video Magnification for Revealing SubtleChanges in the World (2012)

13. Ammar, H., Lashkar, S.: Obstructive sleep apnea diagnosis basedon a statistical analysis of the optical flow in video recordings. In:International Symposium on Signal, Image, Video and Communi-cations (ISIVC), pp. 18–23. IEEE (2016)

14. Black, M.J., Anandan, P.: The robust estimation of multiplemotions: Parametric and piecewise-smooth flow fields. Comput.Vis. Image Underst. 63(1), 75–104 (1996)

15. Black, M.J., Anandan, P.: A framework for the robust estimation ofoptical flow. In: Proceedings of the Fourth InternationalConferenceon Computer Vision, pp. 231–236. IEEE (1993)

16. Lehman, S., Antic, N.A., Thompson, C., Catcheside, P.G., Mercer,J.,McEvoy,R.D.:Central sleep apnea on commencement of contin-uous positive airway pressure in patients with a primary diagnosisof obstructive sleep apnea-hypopnea. J. Clin. Sleep. Med. 3(5),462–466 (2007)

123

874 S. Lashkar, H. Ammar

Samaher Lashkar is an Information Technology master student at KingAbdulaziz University, Jeddah, Kingdom of Saudi Arabia. She receiveda B.Sc. degree in Computer Science from Umm Al-Qura University,Makkah, Kingdom of Saudi Arabia, in 2013. Her research interestsinclude image and video processing.

Heyfa Ammar is an Assistant Professor in the Faculty of Comput-ing and Information Technology, King Abdulaziz University, Jeddah,

Kingdom of Saudi Arabia. Her research interests include video andimage processing. She received her Ph.D degree in Information Tech-nology and Communications in 2012 from the Higher School of Com-munications of Tunis (SupCom), her Master’s degree in Mathematicalengineering from the school of polytechnics of Tunisia in 2005 andher Engineer degree in computer science from the National School ofcomputer sciences (ENSI, Tunisia) in 2002. She also worked as a com-puter engineer in the R&D department of Alcatel (now Nokia).

123