Global Bilateral Symmetry Detection Using Multiscale Mirror Histograms

IntroductionRelated WorkMethodology

Results and Discussion

Global Bilateral Symmetry Detection UsingMultiscale Mirror Histograms

M. ELAWADY1, C. BARAT1, C. DUCOTTET1 and P. COLANTONI2

1Universite de Lyon, CNRS, UMR 5516, Laboratoire Hubert Curien,Universite de Saint-Etienne, Jean-Monnet, F-42000 Saint-Etienne, France

2Universite Jean Monnet, CIEREC EA n0 3068, Saint-Etienne, France

ACIVS Conference, October 2016

UMR • CNRS • 5516 • SAINT-ETIENNE

M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 1 / 31



Table of Contents

1 IntroductionBackgroundApplicationsProblem Definition

2 Related WorkIntensity-based MethodsEdge-based Methods

3 MethodologyMotivationAlgorithm Details

4 Results and Discussion




BackgroundApplicationsProblem Definition

Table of Contents









Bilateral Symmetry

1Image from book: The Photographer’s Eye by Michael Freeman





Bilateral Symmetry in Computer Vision I

Medial Image Compression [1]

Depth Estimation [2]





Bilateral Symmetry in Computer Vision II

Object Segmentation [3]

Aesthetic Analysis [4]





Detection of Main Symmetry Axis

Axis Legend: Strong, Weak




Intensity-based MethodsEdge-based Methods

Table of Contents









Baseline and its Successors I

The general scheme (Loy and Eklundh 2006 [5]) consists of:

Example:

1Second figure from [5]





Baseline and its Successors II

Disadvantages:

Depending mainly on the properties of hand-crafted features (i.e. SIFT).

For example: (smooth objects with noisy background)little feature points =⇒ lost symmetry.

(Mo and Draper 2011 [6]) proposed refinements in the general scheme in:

1 Selecting all symmetry candidate pairs instead of finding closest matchesfor each point.

2 Using less complex hough voting scheme.





State of Art

Instead of SIFT, the general idea (Cicconet et al. 2014 [7]) is extracting aregular set of wavelet segments with local edge amplitude and orientation.

Disadvantages:

Lacking neighborhood’s information inside the feature representation.

Depending on the scale parameter of the edge detector.

For example: (high texture objects with noisy background)inferior symmetrical info =⇒ incorrect symmetry.




MotivationAlgorithm Details

Table of Contents









Proposed Idea

Investigating Cicconet’s edge features [7] within Loy’s scheme [5] byadding neighboring-pixel information.

Contributions:

1 Introducing a new local edge descriptor.

2 Using multiscale edge extraction exploiting the full resolution image.

3 Solving the orientation discontinuity problem in the voting space.

4 Introducing a symmetry dataset based on aesthetic analysis.





Symmetry Detection Algorithm

Main Steps:

(1) Mul�scale Edge Segment Extrac�on

(2) Triangula�on based on Local Symmetry Weights:

• Geometry Edge Orienta�ons (Cic)• Local Texture Histogram (Loy)

(3) Vo�ng Space for Peak Detec�on with Handling Orienta�on Discon�nuity.

θ

ρ0

π





Multiscale Edge Segment Extraction I

A feature point p and its local edge characteristics (Jp, τp) are extractedwithin each cell using a Morlet wavelet ψk,σ of constant scale σ andvarying orientation {τk , k = 1 . . . n}.





Multiscale Edge Segment Extraction II

Jk(p) denote the modulus of wavelet coefficients at point p, in whichlocal edge characteristics Jp and τp are obtained by seeking the maximumwavelet response and orientation over all orientations.





Multiscale Edge Segment Extraction III

Histogram count at a given orientation τk is:

hp(k) =∑

r∈N(p)

Jrδφk−φr (1)

where φk and φr are angles associated with τk and τr , and δx is theKronecker delta. hp is subsequently `1 normalized and circular shifted soas the first bin corresponds to τp.

0 36 72 108 1440

0.5

1

1.5

2

2.5

3

3.5

4

4.5#106

Magnitude Histogram

108 144 0 36 720

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hp)

0 36 72 108 1440

500

1000

1500

2000

2500

3000

Frequency Histogram





Multiscale Edge Segment Extraction IV

In most images, relevant information about the visual content mayappear at different scales. Feature points are computed with respect to aset of regular grids at different scales and a corresponding set of waveletscales {σl , l = 1..m}.





Triangulation: Local Texture Histogram

(Textural Information) Symmetry degree of the two regions around p andq can be measured by comparing their corresponding local orientationhistogram hp and hq. Texture-based symmetry measure is given by:

dI (hp, h∗q) =

n∑k=1

min(hp(k), h∗q(k)) (2)

108 144 0 36 720

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hp)

72 36 0 144 1080

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hq*)

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Intersection





Triangulation: Geometry Edge Orientation

(Edge Information) Pairwise symmetry coefficient f (p, q) is defined as [7]:

f (p, q) = |τqS(T⊥pq)τp| (3)





Triangulation: Symmetry Weights

Given a pair of feature points (p, q), the candidate axis T⊥pq perpendicularto (pq) is parametrized by the orientation of its normal θpq and itsdistance to the origin ρpq.

Mirror symmetry histogram HS(ρ, θ) is defined as the sum of thecontribution of all pairs of feature points such as:

H(cx , cy , θ) =∑p,qp 6=q

JpJqf (p, q)dI (hp, hq)δ(cx ,cy )− p+q2δθ−θpq (4)

HS(ρ, θ) =∑cx ,cy

H(cx , cy , θ)δρ−ρpq (5)





Voting Space and Peak DetectionA1 A2 A3 A4 A5

B

A1 A2 A3 A4 A5

B

BA1 A2 A3 A4 A5




Table of Contents








Algorithm Evaluation

From Real-World Images Competition CVPR 2013 [10], a symmetrydetection is correct if: (1) θ < 15◦ and (2) d < 0.2 ∗min(lenGT , lenR).

RGT d

θ




Quantitative Results��uD(t

M

CM

iM

cM

OM

uM

rM

�M

M

M

CMM

�� iMCM �� iMCC �� iMCc ��iMCr

��iMMr

��iMCC

��iMCO

��iMCr




Qualitative Results on PSU Datasets

(http://www.flickr.com/), around 200 images from PSU symmetrydetection challenges [9, 10] in ECCV2010, CVPR2011 and CVPR2013.

Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014




Qualitative Results on AVA Dataset

(http://www.dpchallenge.com/), around 250 images from AestheticVisual Analysis “AVA” [8] with our global-axis symmetry groundtruth.

Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014




Conclusion

Summary:1 A reliable global symmetry detection is developed among variants of visual cues.

2 A groundtruth of global symmetry axis is introduced and extracted from largescale Aesthetic Visual Analysis (AVA) dataset.

Future work:1 Real-world images is required to handle with large degrees of perspective view.

2 The proposed detection can be improved to avoid over-extended axes.

3 A stable balance measure can be introduced to describe the existence and degreeof global axes inside an image.

4 Possibility of integration within retrieval systems of visual arts.




References I

[1] V. Bairagi, “Symmetry-based biomedical image compression,” Journal of digitalimaging, pp. 1–9, 2015.

[2] L. Yang, J. Liu, and X. Tang, “Depth from water reflection,” Image Processing,IEEE Transactions on, vol. 24, no. 4, pp. 1235–1243, 2015.

[3] C. L. Teo, C. Fermuller, and Y. Aloimonos, “Detection and segmentation of 2dcurved reflection symmetric structures,” in Proceedings of the IEEE InternationalConference on Computer Vision, pp. 1644–1652, 2015.

[4] S. Zhao, Y. Gao, X. Jiang, H. Yao, T.-S. Chua, and X. Sun, “Exploringprinciples-of-art features for image emotion recognition,” in Proceedings of theACM International Conference on Multimedia, pp. 47–56, ACM, 2014.

[5] G. Loy and J.-O. Eklundh, “Detecting symmetry and symmetric constellations offeatures,” in Computer Vision–ECCV 2006, pp. 508–521, Springer, 2006.

[6] Q. Mo and B. Draper, “Detecting bilateral symmetry with feature mirroring,” inCVPR 2011 Workshop on Symmetry Detection from Real World Images, 2011.




References II

[7] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetryhistograms for capturing geometric properties in images,” in Computer Visionand Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 2981–2986,IEEE, 2014.

[8] N. Murray, L. Marchesotti, and F. Perronnin, “Ava: A large-scale database foraesthetic visual analysis,” in Computer Vision and Pattern Recognition (CVPR),2012 IEEE Conference on, pp. 2408–2415, IEEE, 2012.

[9] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First symmetrydetection competition: Summary and results,” tech. rep., Technical ReportCSE11-012, Department of Computer Science and Engineering, ThePennsylvania State University, 2011.

[10] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu,“Symmetry detection from realworld images competition 2013: Summary andresults,” in Computer Vision and Pattern Recognition Workshops (CVPRW),2013 IEEE Conference on, pp. 200–205, IEEE, 2013.




Questions?
