Perceptual Watermarking for Stereoscopic 3DImage Based on Visual Discomfort

Sang-Keun Ji, Ji-Hyeon Kang, and Heung-Kyu Lee(B)

School of Computing, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea{skji,jhkang,hklee}@mmc.kaist.ac.kr

Abstract. As 3D content including images and videos has been commonand popular, the demand for copyright protection has been increased.To protect the copyright of 3D content, 3D image watermarking schemeshave been proposed. Given that visual discomfort can occur in 3D contentduring the watermark embedding process due to binocular mismatch,unlike in 2D content, 3D watermarking schemes should consider visualdiscomfort because it can decrease the performance of the human visionsystem (HVS). In this paper, a perceptual watermarking scheme forstereoscopic 3D images considering the issue of visual discomfort is intro-duced. The proposed scheme analyses the factors that cause visual dis-comfort during the watermark embedding process. In order to minimizevisual discomfort and prevent quality degradation, perceptual maskingusing an occluded map and a defocused map is applied. Experimentalresults show that the proposed scheme offers low visual discomfort whilepreserving the robustness against attacks.

Keywords: Stereoscopic 3D image · Digital watermarking · Percep-tual embedding · Binocular mismatch · Color mismatch · Sharpnessmismatch

1 Introduction

Given the advances in 3D technologies and displays, 3D content including imagesand videos has become common and popular. Unlike 2D content, 3D contentprovides depth perception to instill a feeling of reality by displaying two imageswith different perspectives [1]. There are two formats for 3D content distrib-ution: DIBR (Depth-Image-Based Rendering) and S3D (Stereoscopic 3D) [2].DIBR 3D images, consisting of a depth map and center image, generate left andright images using DIBR algorithms. Unlike DIBR, S3D images simply consistsof left and right images. While DIBR was preferred over S3D due to data storagelimitations and difficulties in 3D shooting techniques in the past, S3D content ismore widely used and commercialized due to advances in computing technolo-gies. S3D content has an advantage in that it can present high-definition 3Dcontent without the quality degradation issue arising.

Due to the demand for copyright protection of 3D content, much research onstereoscopic watermarking schemes has been conducted [3]. However, not onlyc© Springer Nature Singapore Pte Ltd. 2017K. Kim and N. Joukov (eds.), Information Science and Applications 2017,Lecture Notes in Electrical Engineering 424, DOI 10.1007/978-981-10-4154-9 38

324 S.-K. Ji et al.

are most outcomes DIBR-based watermarking schemes, but also relatively fewS3D-based watermarking schemes which consider the HVS and the discomfort.In one study [4], a watermarking scheme based on a visual sensitivity modelfor HD stereo images in the DCT domain was proposed. This scheme used avisual sensitivity model based on what is known as the just noticeable distor-tion (JND), which presents the maximum distortion thresholds in pixels usingHVS characteristics. However, the authors of that study did not consider binoc-ular characteristics because the JND model only considers a single image. Inanother work [5], a stereo image watermarking method based on the binocularjust noticeable model (BJND) which consider the binocular visibility of stereoimages was proposed. Because the BJND model can describe the sensitivity ofthe HVS to luminance changes in stereo images, unlike the JND model, thisscheme embeds a watermark while considering changes between the original andwatermarked images to be lower than the corresponding BJND values. However,the BJND model cannot be regarded as a completely objective measure becauseit was developed based on psychophysical experiments and modelling.

In this paper, a perceptual watermarking method with low visual discom-fort for stereoscopic 3D images is introduced. The proposed method analysesthe characteristics of S3D images and the factors that cause visual discomfortduring the watermark embedding process to reduce visual discomfort and pre-vent quality degradation. The rest of paper is organized as follows. In Sect. 2,the background knowledges about visual discomfort and characteristics of S3Dcontents is handled. In Sect. 3, the proposed watermarking scheme is described.In Sect. 4, the experimental setup and results are shown and Sect. 5 concludes.

2 Background

To design a S3D watermarking scheme, we consider two issues related to visualdiscomfort for perceptual watermarking: visual discomfort assessment and theDepth-of-Field (DoF).

2.1 Visual Discomfort Assessment

When a viewer views 3D content through a stereoscopic display, visual discomfortthat presents a perceived degree of annoyance can occur [6]. Distortion can ariseduring the watermarking process, as it is a type of noise addition. This distortioncan be considered as binocular mismatches among visual discomfort factors dueto increases in the photometry differences between the left and right images. Inbinocular mismatch cases, there are mainly brightness, gamma, contrast, color,and sharpness mismatches [7].

Voronov et al. proposed metrics for evaluating color mismatches and sharp-ness mismatches when analyzing visual discomfort of S3D contents [8]. The colormismatch metric can evaluate noticeable color differences between left and rightimages caused by inconsistencies in the camera settings and shooting environ-ment. An example of the color mismatch is shown as in Fig. 1. The larger the

Perceptual Watermarking for S3D Image Based on Visual Discomfort 325

(a) left image (b) right image

Fig. 1. Examples of color mismatch

(a) left image (b) right image

Fig. 2. Examples of sharpness mismatch

color mismatch metric is, the higher the visual discomfort becomes. The sharp-ness mismatch metric can evaluate differences at high frequencies caused by focusmismatches and inaccurate post-processing steps, as shown in Fig. 2. While theprocess is similar to that of the color mismatch metric, high-frequency informa-tion is used instead of the RGB color space. Likewise, the larger the sharpnessmismatch metric is, the higher the visual discomfort becomes.

Because 3D watermarking schemes should reduce binocular mismatchescaused by the watermark embedding process, the proposed method focuses ontwo types of mismatch, the color and sharpness mismatches, to mitigate visualdiscomfort.

2.2 The Depth-of-Field

The Depth-of-Field (DoF), which is called the focus range, is the distance of thefocused region. The higher the DoF is, the larger the area that can be seen clearlybecomes, as shown in Fig. 3. When the DoF in 3D contents increases, however,visual discomfort can occur due to the accommodation-vergence conflict [9]. In[10], the synthetic blur that decrease the DoF was activated to a defocusedregion for visual comfort in VR applications. Therefore, the proposed methoddetects defocused regions, which don’t contain visually important information,and embeds a watermark into them to improve the invisibility.

(a) DoF = 0.8 cm (b) DoF = 2.2 cm (c) DoF = 12.4cm

Fig. 3. Examples with different Depth-of-Fields

3 Proposed Method

In the proposed method, there are three main steps: perceptual masking con-struction, watermark embedding, and watermark extraction. The process of per-ceptual mask construction and watermark embedding is shown in Fig. 4.

326 S.-K. Ji et al.

Fig. 4. The process of perceptual mask construction and watermark embedding

3.1 Perceptual Masking Construction

To minimize the increase in visual discomfort occurred by binocular mismatchesduring the watermark embedding process, a perceptual mask consisting ofan occluded map and a defocused map is constructed. This process includesoccluded region detection and defocused region detection.

Occluded Region Detection. This process finds regions that can minimizethe increase in the color and sharpness mismatches. These mismatches tendto increase when distortion occurs in regions where the luminance difference isgreater rather than smaller between the left and right images. This tendency canbe explained by hidden pixels occurring in DIBR images [11]. In other words, aregion where the luminance difference is greater can be considered as an occludedregion, which is not common in left and right images but is visible to only oneview. Therefore, the watermark is embedded with a high weight in regions withgreater luminance differences, whereas it is embedded with low weight in regionswith smaller luminance differences.

The process of occluded region detection is as follows. First, stereo matchingis performed between the left and right images. For each view, the reconstructedimage I ′ of a certain view I is reconstructed from the other view using matchinginformation. Subsequently, the occluded map Mo of each view is computed bythe following equation:

M io(I) =

{1 , if Di

y(I) > ty0 , otherwise

, where Diy(I) =

n×n∑(Y (Ii) − Y (I ′

i))2

n2(1)

Here, I ′ is the reconstructed image of image I, Y is the luminance of the image,and n is the block size. Di

y is the luminance difference of the i-th block between

Perceptual Watermarking for S3D Image Based on Visual Discomfort 327

(a) left image (b) occluded map ofleft image

(c) defocused map ofleft image

(d) perceptual maskof left image

(e) right image (f) occluded map ofright image

(g) defocused map ofright image

(h) perceptual maskof right image

Fig. 5. Examples of perceptual masking

I and I ′, and ty is a predefined threshold. Examples of an occluded map areshown in Figs. 5(b) and (f).

Defocused Region Detection. Because regions aprt from the region-of-interest (ROI) are blurred for visual comfort, as noted in Sect. 2, the watermarkis embedded with a high weight for imperceptibility. To find defocused region,the differences Db between an image and a blurred image using DoF blur iscalculated by:

Dib =

∑ {Y (Ii) − b(Y (Ii))}2n2

, (2)

where b denotes the blur kernel and Dib is the luminance difference in the i-th

block between the two images. Regions with low differences are blurred regions;however, they can be regions at low frequencies, such as a flat region. Thus, itis necessary to consider the color distribution drgb to remove flat regions.

dirgb =√∑

(Iir − μir)2 +

√∑(Iig − μi

g)2 +√∑

(Iib − μib)2 (3)

Here, μir, μi

g, and μib are the average values of the ith block, respectively, and

dirgb is the color distribution of the ith block.Finally, the defocused map Md that presents the defocused region is detected

by the following equation:

Md = smin +Db · drgb × (smax − smin)

max(Db · drgb) − min(Db · drgb)(4)

Here, smin and smax represent the lower and upper bounds of the defocusedmap. Examples of an defocused map are shown in Figs. 5(c) and (g).

328 S.-K. Ji et al.

3.2 Watermark Embedding and Extraction

The proposed method embeds a watermark into the DCT domain using a spread-spectrum method. The watermark embedding process is performed as follows.First, the watermark pattern W , which has a normal distribution with a zeromean and unit variance, is generated by a pseudo-random number generatorusing a key. After a host image is transformed to the DCT domain, mid-frequencycoefficients V are selected to be watermarked. The watermark W is then embed-ded into V using the following equation:

v′i = vi + α|vi|wi, (0 ≤ i ≤ N) (5)

In this equation, V = {v1, v2, ..., vn} is the vector of the DCT coefficient and V ′

is the watermarked vector. W = {w1, w2, ..., wn} is the vector of the watermarkand α denotes the watermark strength. After embedding the watermark, thewatermarked image I ′ is obtained using the inverse DCT. Finally, the percep-tually watermarked image Iw is obtained by applying perceptual masking to I ′

using the following equation.

Iw = I × {1 − (αoMo + αdMd)} + I ′ × (αoMo + αdMd), where αo + αd = 1 (6)

Here, αo and αd are the weights of the occluded map and defocused map, respec-tively.

To extract the embedded watermark, the DCT coefficient V ∗ at a fixedposition using the watermark embedding process is extracted from a suspectedimage. A normal correlation is then calculated between V ∗ and W . If the cor-relation exceeds a predefined threshold, which is experimentally determined toensure a low false positive rate, it determined that a watermark is detected.

4 Experimental Results

For evaluation, we used S3D images, which have the resolution of 2872 × 1984,from Middlebury Stereo Datasets [13] that have been widely used for stereoevaluation. The parameter values used in the experiment are: n = 64, ty = 192,[smin, smax] = [0, 1]. αo and αd are 0.5, respectively. To verify the performance,the proposed method is compared with the BJND method [5].

4.1 Visual Quality

To verify visual quality, the structural similarity (SSIM) [14], color mismatch,and sharpness mismatch are evaluated by setting the PSNR to 48dB. As shownin Table 1, the proposed method show that the SSIM is higher than other meth-ods. Compared with the BJND method, the increases in color and sharpnessmismatches are reduced by 61% and 3%, respectively. To measure the effect ofperceptual masking, the proposed method without masking was compared. Asa result, we find that perceptual masking is effective in mitigating visual dis-comfort during the watermark embedding process, as the increases in color andsharpness mismatches are reduced by 66% and 65%, respectively.

Perceptual Watermarking for S3D Image Based on Visual Discomfort 329

Table 1. SSIM, color and sharpness mismatches with the PSNR 48 dB

Originalimage

BJND method w/o masking Proposedmethod

SSIM - 0.9861 0.9858 0.9904

Color Mismatch value 27.6083 28.0084 28.0688 27.7631

increase - (+0.4002) (+0.4606) (+0.1548)

Sharpness Mismatch value 0.6255 0.6486 0.6898 0.6479

increase - (+0.0232) (+0.0644) (+0.0224)

(a) JPEG compression (b) Gaussian noise addition

Fig. 6. Examples with different Depth-of-Fields

4.2 Robustness

To measure the robustness, two experiments were carried out on JPEG compres-sion and Gaussian noise addition, respectively. Experimental results show thatthe proposed method with perceptual masking preserves the robustness againstattacks similar to that without perceptual masking. Therefore, there was hardlyany decrease in the robustness due to masking.

5 Conclusion

In this paper, we proposed a perceptual watermarking for S3D image based onvisual discomfort assessment. To minimize visual discomfort caused by binocu-lar mismatch during the watermark embedding process, the proposed methodapplied perceptual masking, which consists of occluded map and defocused map,that exploits the characteristics of S3D content. To measure visual discomfort,we adopted the color and sharpness mismatch that evaluate binocular mis-match between left and right images. Experimental results show that the pro-posed method has low visual discomfort while preserving the robustness againstattacks. For the future work to improve the proposed scheme, various visualdiscomfort assessments should be considered for evaluating quality degradation.

330 S.-K. Ji et al.

Acknowledgments. This research was supported by a grant from the Advanced Tech-nology Center R&D Program funded by the Ministry of Trade, Industry & Energy ofKorea (10042252) and the National Research Foundation of Korea (NRF) grant fundedby the Korea government (MSIP) (No. 2016R1A2B2009595)

References

1. Reichelt, S., Haussler, R., Futterer, G., Leister, N.: Depth cues in human visualperception and their realization in 3D displays. In: SPIE Defense, Security, andSensing. International Society for Optics and Photonics, p. 76900B (2010)

2. Vetro, A., Tourapis, A.M., Muller, K., Chen, T.: 3D-TV content storage and trans-mission. IEEE Trans. Broadcast. 57(2), 384–394 (2011)

3. Chammem, A., Mitrea, M., Preteux, F.: Stereoscopic video watermarking: a com-parative study. Ann. Telecommun. Annales des Telecommunications 68(11–12),673–690 (2013)

4. Niu, Y., Souidene, W., Beghdadi, A.: A visual sensitivity model based stereo imagewatermarking scheme. In: European Workshop on Visual Information Processing(EUVIP), pp. 211–215 (2011)

5. Bitaghsir, S.A., Karimi, N., Azizi, S., Samavi, S.: Stereo image watermarkingmethod based on binocular just noticeable difference. In: International ISC Con-ference on Information Security and Cryptology, pp. 33–38 (2014)

6. Lambooij, M., Fortuin, M., Heynderickx, I., IJsselsteijn, W.: Visual discomfort andvisual fatigue of stereoscopic displays: a review. J. Imaging Sci. Technol. 53(3),30201–1 (2009)

7. Li, J., Barkowsky, M., Le Callet, P.: Visual Discomfort in 3DTV: definitions, causes,measurement, and modeling. In: Kondoz, A., Dagiuklas, T. (eds.) Novel 3D MediaTechnologies, pp. 185–209. Springer, New York (2015)

8. Voronov, A., Vatolin, D., Sumin, D., Napadovsky, V., Borisov, A.: Methodology forstereoscopic motion-picture quality assessment. In: IS&T/SPIE Electronic Imag-ing. International Society for Optics and Photonics, p. 864810 (2013)

9. Tam, W.J., Speranza, F., Yano, S., Shimono, K., Ono, H.: Stereoscopic 3D-TV:visual comfort. IEEE Trans. Broadcast. 57(2), 335–346 (2011)

10. Hillaire, S., Lcuyer, A., Cozot, R., Casiez, G.: Depth-of-field blur effects for first-person navigation in virtual environments. In: Proceedings of the 2007 ACM Sym-posium on Virtual Reality Software and Technology, pp. 203–206 (2007)

11. Lee, M.J., Lee, J.W., Lee, H.K.: Perceptual watermarking for 3D stereoscopic videousing depth information. In: International Conference on Intelligent InformationHiding and Multimedia Signal Processing (IIH-MSP), pp. 81–84 (2011)

12. Barni, M., Bartolini, F., Cappellini, V., Piva, A.: A DCT-domain system for robustimage watermarking. Sig. Process. 66(3), 357–372 (1998)

13. Scharstein, D., Hirschmuller, H., Kitajima, Y., Krathwohl, G., Nesic, N., Wang, X.,Westling, P.: High-resolution stereo datasets with subpixel-accurate ground truth.In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp.31–42. Springer, Cham (2014). doi:10.1007/978-3-319-11752-2 3

14. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment:from error visibility to structural similarity. IEEE Trans. Image Process. 13(4),600–612 (2004)