Subjective and Objective Quality ... - Pavan Chennagiri · SubjectiveandObjectiveQualityAssessmentofStitched ImagesforVirtualReality Pavan C M MTech(Research)Candidate Advisor: Dr

Subjective and Objective Quality Assessment of StitchedImages for Virtual Reality

Pavan C M

M Tech (Research) CandidateAdvisor: Dr. Rajiv Soundararajan

Department of Electrical Communication EngineeringIndian Institute of Science, Bangalore

Thesis DefenseOctober 26, 2018

Outline of the Talk

1 IntroductionProblem DefinitionChallengesPrior Work

2 Thesis OverviewDatabase and Subjective Quality AssessmentAutomatic Quality Assessment Algorithm

3 Experiments and Results4 Conclusion and Future Work

Pavan Stitched Image QA October 26, 2018 1 / 42

Outline of the Talk





Introduction

Virtual Reality (VR) - immersive experience through wide field of viewimages/videosVR applications - motion pictures, cinematic VR, immersivestorytelling etc.Head mounted displays (HMD) - freedom to choose desired viewsWide field of view images - stitching multiple images with overlappingviews


Introduction

Stitching algorithm - multiple stagesEach stage - influence on quality of stitched image

VR popularity - necessity for quality controlRelevance - benchmark, tune parameters and compare variousstitching algorithms.


Problem Statement

StitchingAlgorithm

Quality?

Constituent Images Distorted Stitched Image


Challenges

Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature points

Color distortion - images with different exposure levelsBlur and geometric distortion - improper blending of multiple images

Ghosting Blur Color Distortion Geometric

Stitching induced distortions - specific to panoramic images


Challenges

Ghosting


Challenges

Blur


Challenges

Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levels

Blur and geometric distortion - improper blending of multiple images




Challenges

Color Distortion


Challenges

Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levelsGeometric distortion - improper blending of multiple images




Challenges

Geometric


Challenges

Development of quality index - captures stitching induced distortionsGhosting and blur - inaccurate matching of feature pointsColor distortion - images with different exposure levelsGeometric distortion - improper blending of multiple images


Stitching induced distortions - specific to stitched images


Challenges

Absence of reference stitched images - not full reference qualityassessment

Constituent images - reference information

UnknownStitchingAlgorithm

Quality?


Problem Setup

Assumption - access to individual and stitched images, no knowledge ofstitching algorithm


Challenges

Absence of reference stitched images - not full reference qualityassessmentConstituent images - reference information

UnknownStitchingAlgorithm

Quality?


Problem Setup

Assumptions - access to individual and stitched images, no knowledge ofstitching algorithm


Problem Setup - Assumption

Absence of reference stitched images - not full reference qualityassessmentConstituent images - reference information

StitchingAlgorithm

Quality?


Problem Setup

Assumptions - access to individual and stitched images, no knowledge ofstitching algorithm


Prior Art - No Reference Quality Assessment (QA)

No Reference (NR) quality assessment - rich literature and widelystudiedNatural Scene Statistics (NSS) - DIIVINE[Moorthy2011], BRISQUE[Mittal2012], NIQE [Mittal2013]

StitchingAlgorithm

Blind QA


Problem Setup

Existing QA - do not address types of distortions observed in stitchedimages


Prior Art - QA in Stitched Images

[WeiXu2010] - evaluates color similarity and structural similarityRestrictive model - uses pointwise comparison, can be inaccurate

[Qureshi2012] - computes color and structural similarity in overlappingregions

Extension of [WeiXu2010] - uses high pass content in overlappingregion for structural similarity

Above algorithms - not evaluated on subjective database

[WeiXu2010] method [Qureshi2012] method


Contributions

Stitched image quality assessment databaseStitched images captured across diverse scenesSubjective evaluation - perception of distortions

Objective quality assessmentNatural scene statistics model

Bivariate statistics - increased correlations due to distortions

Correlates well with human perception


Outline of the Talk





Dataset

Stitched image quality assessment databaseImages from 26 scenes - buildings, gardens, indoor and public places264 stitched images - fusing multiple views with overlapping regionsStatic scenes - no object motion

Stitched image qualityChoice of algorithm for each stageParameter options associated with each block

Major impairments - ghosting, blur, geometric and color


Dataset





Dataset





Feature Detection, Matching and Outlier Removal

Image alignment - detecting keypoints in overlapping regionsDetection and matching- SIFTOutlier removal - Random Sample Consensus

Figure: Keypoint Detection Figure: Keypoint Matching

Figure: Outlier removal


Dataset





Homography and Image Warping

Warp - transformation on co-ordinates for aligning overlapping regionsHomography - generalized transform, Direct Linear Transform (DLT)Moving DLT [Zaragoza2013] (MDLT) - patch level homographyShape preserving warp (SPHP) [Chang2014] - constrained homography

Homography MDLT SPHP

Homography


Dataset





Image Blending

Blending - fusing multiple images to form single composite imageSmooth transition with no visible seams

Feathering - weighted averagingMultiband - Laplacian pyramid based blendingPoisson - gradient domain, optimizing the cost function

No Blending Multiband

Feathering with exposurecompensation

Poisson


Dataset





Subjective Study

Single stimulus continuous quality assessmentRating - viewing images on a VR head mounted device (HMD)Images rated by 35 subjects across 3 sessionsProcessing of scores - Mean Opinion Score (MOS) for each imageafter rejecting outliers

0 10 20 30 40 50 60 70 80 90 100

MOS

0

5

10

15

20

25

30

35

40

Num

ber

of Im

ages

Distribution of MOS

MOS = 21.546

MOS = 65.238


Subjective Study

MOS = 21.546

MOS = 65.238


Outline of the Talk





Stitched Image Quality Evaluator (SIQE) Framework

SteerablePyramids

DivisiveNormalization(Marginal)

SteerablePyramids


BivariateStatistics

BivariateStatistics

NaturalScene Statistics

Model

NaturalScene Statistics

Model

PatchWeighting

PatchWeighting

Difference Regression

StitchedImage

ConstituentImages

ModelParameters

ModelParameters

ModelParameters

ModelParameters

PredictedScore

Features from Stitched Image

Features from Constituent Images


Origin of Distortions

I1 I2 Original Ghosting

I Original Geometry

H

I(x) = (1− α(x))I1(x) + α(x)I2(Hx), where α(x) ∈ (0, 1)

I1(x) 6= I2(Hx) - combination of ghosting and blurPresence of additional edgesIncreased spatial correlation

Geometric distortion - presence of extraneous edges


Multi-Orientation Decomposition

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics

DistributionModelling


PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

ModelParameters

ModelParameters

PredictedScore




Multi-Orientation Decomposition

Structural artifacts from ghosting and geometric - orientationdependentSteerable pyramid decomposition - 6 orientations, 2 scales for eachN ×N patch

Ghosting s60◦

1 s150◦

1

Geometry s30◦

1 s90◦

1


Divisive Normalization

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

fs1−12

f c1−12

ModelParameters

ModelParameters

PredictedScore




Divisive Normalization

Divisive Normalization - y = y/p for subband coefficient y, with

p =√Y TC−1

U Y/N where CU is the covariance of neighborhoodaround y, N - number of neighbors

Contrast maskingReduce statistical dependencies - decorrelation

Previously shown to capture blur in [Li2009], [Moorthy2011]Besides blur, captures edges introduced due to distortions

Normalization factor p - measure of local variancep - higher values near edges


Divisive Normalization - Modeling

Ghosting and geometric distortions - presence of additional edgesDistribution of distorted patch - higher peak value at zero as y = y/p

Model - Generalized Gaussian Distribution (GGD)Features - GGD shape parameters

Original Ghosting Original Geometry

-8 -6 -4 -2 0 2 4 6 8

Subband Coefficients

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

Num

ber

of C

oeffi

cien

ts (

Nor

mal

ized

)

original patchghosted patch

P (d150◦

1 (x, y)

-15 -10 -5 0 5 10 15


0

0.01

0.02

0.03

0.04

0.05

0.06

Num

ber

of C

oeffi

cien

ts (

Nor

mal

ized

)

original patchgeometric distorted patch

P (d30◦

1 (x, y))


Bivariate Model

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

fs13−36

f c13−36

ModelParameters

ModelParameters

PredictedScore




Bivariate Model

Capturing increased spatial correlation in ghosting - bivariatedistributionBivariate statistics - adjacent subband coefficientsP (sθα(x, y), sθα(x+ 1, y)) (with no divisive normalization)Distribution of ghosted patch - higher peak value than undistorteddistribution

Original Patch Ghosted PatchPavan Stitched Image QA October 26, 2018 24 / 42

Bivariate Model - Conditional Distribution Interpretation

Conditional Statistics - P (sθα(x+ 1, y)/sθα(x, y) ∈ (−δ, δ))

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2


0

0.05

0.1

0.15

0.2

0.25

0.3

Num

ber

of C

oeffi

cien

ts (

Nor

mal

ized

)

Conditional Histogram (original patch)Conditional Histogram (ghosted patch)

Deviation between conditional distributions - higher than marginaldistributions


Bivariate Model - Conditional Distribution Interpretation

Conditional Statistics - P (sθα(x+ 1, y)/sθα(x, y) ∈ (−δ, δ))

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2


0

0.05

0.1

0.15

0.2

0.25

0.3

Num

ber

of C

oeffi

cien

ts (

Nor

mal

ized

)

Conditional Histogram (original patch)Conditional Histogram (ghosted patch)Marginal Histogram (original patch)Marginal Histogram (ghosted patch)

Deviation between conditional distributions - higher than marginaldistributions


Bivariate Model - GMM and BGGD

Previous approaches - Bivariate GGD - let sθα(x+ 1, y) = a,sθα(x, y) = b, z = [a, b]T

f(z) = K exp (−(zTM−1z)β)

Bivariate Gaussian (BVG) - BGGD with β = 1

Our method - Gaussian mixture model

f(a, b) =

M∑i=1

ωiN (0,Σi)

Components - zero mean, distribution modeled by ωi,ΣiParameter estimation - Expectation Maximization (EM) algorithmGMM - QA


Bivariate Model - Model Comparison

Model comparisons - GMM and BGGD

Pristine

-2 -1 0 1 2

Subband Coefficients (Left)

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

Sub

band

Coe

ffici

ents

(R

ight

)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

10-3

(KL - 0.1477)

-2 -1 0 1 2


-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

Sub

band

Coe

ffici

ents

(R

ight

)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

10-3

(KL - 0.2167)

Distorted

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5 0

1

2

10-4

(KL - 0.1547)

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5 0

1

2

10-4

(KL - 0.24)

Empirical GMM BGGD


Bivariate Model - Model Comparison

Model comparisons - GMM and BVG

Pristine

-6 -4 -2 0 2 4 6


-6

-4

-2

0

2

4

6

Su

bb

an

d C

oe

ffic

ien

ts (

Rig

ht)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

10-5

(KL - 0.1535)

-6 -4 -2 0 2 4 6


-6

-4

-2

0

2

4

6

Su

bb

an

d C

oe

ffic

ien

ts (

Rig

ht)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

10-5

(KL - 0.4635)

Distorted

-5 0 5


-5

-4

-3

-2

-1

0

1

2

3

4

5

Su

bb

an

d C

oe

ffic

ien

ts (

Rig

ht)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10-4

(KL - 0.1547)

-5 0 5


-5

-4

-3

-2

-1

0

1

2

3

4

5

Su

bb

an

d C

oe

ffic

ien

ts (

Rig

ht)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10-4

(KL - 0.3823)

Empirical GMM BVG


Bivariate Model - Features

For sθα(x+ 1, y) = a, sθα(x, y) = b

f(a, b) =

M∑i=1

ωiN (0,Σi)

Covariance C =∑M

i=1 ωiΣi, eigen values of C as featuresHorizontal - sθα(x+ 1, y) = a, sθα(x, y) = bVertical - sθα(x, y + 1) = a, sθα(x, y) = b


Patch Weighting

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

ModelParameters

ModelParameters

PredictedScore




Patch Weighting

All patches equal contribution?

e = 0.957 e = 0.459 e = 0.354 e = 0.106

Ghosting and blur artifacts - not perceived in smooth regionsGray level co-occurrence matrix (GLCM) [Haralick1973]Energy values of GLCM, e ∈ [0, 1], with e = 1 for constant imagePatch weight w = 1− eTextured patches - equal weights through non-linearity

g(w) = 1− exp

(−(wσ

)2)Pavan Stitched Image QA October 26, 2018 31 / 42

Model Summary

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

ModelParameters

ModelParameters

PredictedScore




Model Summary

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

fs1−12

f c1−12

ModelParameters

ModelParameters

PredictedScore




Model Summary

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

fs13−36

f c13−36

ModelParameters

ModelParameters

PredictedScore




Model Summary

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

ModelParameters

ModelParameters

PredictedScore




Prediction

SteerablePyramids


SteerablePyramids


BivariateStatistics

BivariateStatistics



PatchWeighting

PatchWeighting

DifferenceSupport

Vector Regression

StitchedImage

ConstituentImages

ModelParameters

f c1−36

ModelParameters

fs1−36

PredictedScore




Outline of the Talk





Correlation with Human Judgments

Database - 80% training and 20% testing with non overlap of scenesPerformance metric - Spearman rank order correlation coefficient(SROCC) and Pearson’s linear correlation coefficient (LCC)Median performance value - 1000 random train-test combinationsPerformance comparison - NR QA metrics BRISQUE [Mittal2012],NIQE [Mittal2012], DIIVINE [Moorthy2011]

SROCC LCCBRISQUE (trained on our database) 0.6224 0.5914

NIQE 0.1524 0.1051DIIVINE (trained on our database) 0.5706 0.5897

SIQE 0.8318 0.8380

Table: Median correlation across 1000iterations

BRISQUE NIQE DIIVINE SIQE

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Figure: Box plot of SROCCdistributions over 1000 trials


Significance of each conceptual feature

Each conceptual feature - tested in isolationFeatures only from stitched image - drop in performance, importanceof constituent images

NR setting - higher performance than NR-IQA methods

Feature SROCC LCCMarginal statistics model (f1−12) 0.7951 0.7934

Bivariate model (f13−36) 0.6825 0.6972Features from stitched image (fs1−36) 0.6524 0.6816

(when constituent image features are omitted)SIQE (fs1−36 and f c1−36) 0.8318 0.8380


Comparison with FR-QA Algorithms

FR metric - [WeiXu2010] and [Qureshi2012]Dependent on Pointwise correspondencesPerformance evaluation - 238 images, images obtained fromcommercial stitching algorithms ignored

Feature SROCC LCCXu (PSNR) 0.1795 0.2341Xu (SSIM) 0.3383 0.4077Qureshi 0.3238 0.3627SIQE 0.7848 0.8032


Analysis of Color Distorted Images

Scores for images with color distortion - close to images with little orno distortionColor distortion - less annoying when viewed on a HMD?HMD - 90◦ field of view, instances of non-appearance of colordistortion

(a) MOS = 61.7624 (b) MOS = 61.6771

(c) MOS = 59.7492 (d) MOS = 59.954


Outline of the Talk





Conclusion

Subjective quality assessmentStitched image quality databaseDistortions - ghosting, blur, geometric and colorSubjective evaluation on VR

Objective quality assessmentIndependent of underlying stitching algorithmCaptures stitching induced distortionsHigh correlation with human judgments, outperforms existing qualitymeasures

Path aheadModel - color distortion characterizationMethods to account for geometric shape changes and their relevance institched image QA


References

W. Xu and J. Mulligan, "Performance evaluation of color correctionapproaches for automatic multi-view image and video stitching," inIEEE Computer Vision and Pattern Recognition, June 2010, pp.263-270.

H. S. Qureshi, M. M. Khan, R. Hafiz, Y. Cho, and J. Cha,"Quantitative quality assessment of stitched panoramic images," IETImage Processing, vol. 6, no. 9, pp. 1348-1358, December 2012.

A. K. Moorthy and A. C. Bovik, "Blind image quality assessment:From scene statistics to perceptual quality," IEEE Trans. ImageProcess., vol. 20, no. 12, Dec. 2011.

A. Mittal, A. K. Moorthy, and A. C. Bovik, "No-reference imagequality assessment in the spatial domain," IEEE Trans. Image Process.,vol. 21, no. 12, pp. 4695-4708, Dec. 2012.


Acknowledgments

Advisor - RajivFunding agency - Department of Science and TechnologyVolunteers of subjective studyLabmates - Sameer, Biju and others


Thank You!


Documents

Subjective and Objective Quality ... - Pavan Chennagiri · SubjectiveandObjectiveQualityAssessmentofStitched ImagesforVirtualReality Pavan C M MTech(Research)Candidate Advisor: Dr