Sharpening from Shadow:

School of Computer ScienceSimon Fraser University

November 2009Sharpening from Shadows: Sensor Transforms for Removing Shadows using a Single Image

Mark S. DrewHamid Reza Vaezi Jozemark@cs.sfu.cahamid_reza@cs.sfu.caOutlineImage FormationInvariant Image FormationFinding invariant direction by calibrationFinding invariant direction by minimizing entropySharpening MatrixProposed Method Optimization problemResult

2

2Shadow Removal MethodTo generate shadowless images, there are two steps: Finding Illuminant Invariant image (grayscale) Creating colored shadowless images using edges in main image and invariant image. [Finlayson et al. (ECCV2002)]3

Main ImageShadowless ImageInvariant ImageThis Paper3Image Formation4

Surface reflectionCamera sensitivityLight spectralCamera Response:4Image Formation Simplification5Camera sensors represented as delta functions.

Illumination is restricted to the Planckian locus.

Wiens approximation for temperature range 2500K to 10000K.

We have:

Invariant Image Formation6Using the simplified model we form band-ratio chromaticities rk by dividing R and B by G and taking the logarithm:

As temperature T changes, 2d-vectors rk ,k=R,B, will follow a straight line in 2d chromaticity space. For all surfaces, the lines will be parallel, with slope (ek eG).

SurfaceDependentCameraDependentInvariant Image Formation7The invariant image, then, is formed by projecting 2-d colors into the direction orthogonal to the 2-vector (ek eG).So, the problem is reduced to finding the direction.

Why we are interested?Shadow is nothing just the surface in different illumination condition. (they should be in a line)

Finding Invariant Direction8Calibrating Camera to find the invariant direction. [Finlayson et al. (ECCV2002)] Need many images under different illumination.Good for camera company not images with unknown camera.

HP912 Digital Still Camera: Log-chromaticities of 24 patches; 7 patches, imaged under 9 illuminants.

Finding Invariant Direction9Without calibrating the camera, can use entropy of projection to find the invariant direction [Finlayson et al. (2004)]:

Correct direction smaller entropyWrong direction higher entropy

Sharpening Transform Matrix10Convert a given set of sensor sensitivity functions into a new set that will improve the performance of any color-constancy algorithm that is based on an independent adjustment of the sensor response channels.

Transform the camera sensors to made them more narrow band, which is one of the assumption that we made.

It also could apply to the image instead of sensors.

Proposed Method11Select shadow and non shadow pixels for the same surface material.Find the sharpening matrix which makes the chromaticities of selected pixels as linear as possible in log-log plane = an optimization problem.Transform the main image by sharpening matrix.Create illumination invariant image by entropy-minimization method [Finlayson et al. (2004)].

1243.7 0.15 .15.15 .70 .15.15 .15 .70

Shadow and Non Shadow Regions12The user selects the shadow and non shadow region of a surface.

For future work this could be automatic .

According to invariant formation in ideal condition, the chromaticities of these point in log-log plane should be in a line.

User Defined

Optimization Problem13To find best sharpening matrix M3x3 in order to make the chromaticity as linear as possible:

m11 m12 m13 m21 m22 m23m31 m32 m33sum is 1Linear combinationmore than 1-Colors dont change completelyObjective Function14F return the minimum entropy of log chromaticities projected to all directions. rank is meant to encourage a non-rank-reducing matrix M.

entropyLog chromaticities

Minimum entropyFor this MSharpening Matrix15

Sharpening MatrixShadow and non shadow region chromaticityLess linearMore linearResults16

DifferenceInvariantSharpenedOriginal16Good vs. poor sharpening matrix17

More linear.90 .30 -.14-.04 .79 .16.14 -.09 .98.75 -.20 .02.01 .86 .13.24 .34 .84minimumObj. Func. = .0942Obj. Func. = .0487

Result18

Result19

19Conclusion20We proposed a new schema for generating illumination invariant for removing shadow.

The contribution of this paper is using sharpener matrix to get better shadow removal.

The method use single images which is more practical compared to camera calibration methods which needs bunch of images in different illumination condition.

References21Sharpening Matrix: G.D. Finlayson, M.S. Drew, and B.V. Funt. Spectral sharpening: sensor transformations for improved color constancy. J. Opt. Soc. Am. A, 11(5):15531563, May 1994.Illumination invariant image: G.D. Finlayson, S.D. Hordley, and M.H. Brill. Illuminant invariance at a single pixel. In 8th Color Imaging Conference: Color, Science, Systems and Applications., pages 8590, 2000.Shadow removal method: G.D. Finlayson, S.D. Hordley, and M.S. Drew. Removing shadows from images. In ECCV 2002: European Conference on Computer Vision, pages 4:823836, 2002. Lecture Notes in Computer Science Vol. 2353.Entropy minimization method: G.D. Finlayson, M.S. Drew, and C. Lu. Intrinsic images by entropy minimization. In ECCV 2004: European Conference on Computer Vision, pages 582595, 2004. Lecture Notes in Computer Science Vol. 3023.

Questions?Thank you.22Thanks!To Natural Sciences and Engineering Research Council of Canada