The SHINE toolbox for controlling low-level image propertiesVerena Willenbockel1, Javid Sadr2, Daniel Fiset1, Greg O. Horne3, Frédéric Gosselin1, & James W. Tanaka3

1Centre de Recherche en Neuropsychologie et Cognition, Université de Montréal; 2Department of Psychology, University of Massachusetts Boston; 3Department of Psychology, University of Victoria

Results & DiscussionAbstract

Methods

Visual perception can be influenced by top-down processes related to the observer’s goals and expectations, as well as by bottom-up processes related to low-level stimulus attributes, such as luminance, contrast, and spatial frequency. When using different physical stimuli across psychological conditions, one faces the problem of disentangling the contributions of low- and high-level factors. Here we make available the SHINE (Spectrum, Histogram, and Intensity Normalization and Equalization) tool-box for Matlab*, which we have found useful for controlling a number of image properties separately or jointly (Willenbockel et al., in press; Williams, Willenbockel, & Gauthier, 2009). The toolbox features functions for specifying the (rotational average of the) Fourier amplitude spectra, normalizing and scaling mean luminance and contrast, as well as for exact histogram specification optimized for perceptual visual quality. SHINE can thus be employed for parametrically modifying a number of image properties or for equating them across stimuli to minimize potential low-level confounds in studies on higher-level processes. The toolbox can be downloaded here: www.mapageweb.umontreal.ca/gosselif/shine.

References

SHINE Toolbox

spectrumPlotsfPlot

SHINEgetRMSE

histMatchlumMatch sfMatchspecMatch

matchavgHisttarhist rescale

readImages

separate

hist2list

imstats

ssim_index(Wang, 2003)

ssim_sens(Avanaki, 2009)

Iterative approach: joint matching of histograms and spectra

0 5 10 15 200

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Iterations

RM

SE

of s

pec

tra

hist & specMatch Set 1hist & specMatch Set 2spec & histMatch Set 1spec & histMatch Set 2

0 5 10 15 200

5

10

15

x 10-4

Iterations

RM

SE

of h

isto

gram

s

0 5 10 15 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iterations

SS

IM in

dex

histMatch & specMatch

specMatch & histMatch

ITERATION 1

SET 2SET 1

*SHINE is written with functions from Matlab’s Image Processing Toolbox

avgHist: getRMSE: histMatch: hist2list:imstats: lumMatch: match: readImages: rescale: separate: sfMatch: sfPlot: SHINE: specMatch: spectrumPlot: ssim_index: ssim_sens: tarhist:

computes average histogramcomputes root mean square errorexact histogram matching across imagestransforms histogram into a sorted (dark-to-light) listcomputes image statistics across imagesscales mean luminance and contrastbasic histogram specificationloads image setluminance rescaling (to avoid clipping after the IFFT)basic figure-ground segregationequates the rotational average of the amplitude spectraplots the energy at each spatial frequencymain function for loading, equating, and savingamplitude spectrum matchingplots the amplitude spectrumcomputes the Structural Similarity index (SSIM; Wang, 2003)computes SSIM gradient (Avanaki, 2009)computes a target histogram

histMatchOverview of SHINE toolbox functions

In sum, SHINE is an easy-to-use Matlab toolbox for controlling low-level image properties across the foregrounds/backgrounds of an image set. The iterative equalization approach has success-fully been applied to reach a high degree of joint matching of his-tograms and Fourier amplitudes (e.g., Williams et al., 2009).

lumMatch

specMatch & sfMatch

output = (origIm-mean2(origIm))/std2(origIm)*newStd+newMean;

TARGET (average)

combining with original phase

combining with original phase

FFT IFFT & rescaling

pix

el c

ount

pix

el c

ount

value

ORIGINAL IMAGE

ORIGINAL HISTOGRAM

TARGET HISTOGRAM

POSSIBLE OUTPUT IMAGES

value

1 2 34 5 67 8 9

INDICES

3 1 2 6 9 4 5 7 8

SORTED LIST (DARK-TO-LIGHT)

randomized

3 1 2 6 9 4 5 78

a) Basic histogram matching

b) Optional: SSIM optimization (Avanaki, 2009)

INPUT

OUTPUT

1. Let X show the original input image. Set Xnew=X.2. Apply the match function to Xnew to obtain image Y with the given target histogram.3. Compute the SSIM gradient [∇ySSIM(X,Y)] and SSIM index [SSIM(X,Y)] using the ssim_sens function.4. If convergence is reached (e.g., output quality is good enough), then break.5. Set Xnew=Y+µN∇ySSIM(X,Y), where µ denotes step size and N the number of pixels, and go to 2.6. Output Y.

.∗

.∗

COEFFICIENT MATRICES

AMPLITUDE SPECTRA

OUTPUT

a) specMatch

b) sfMatch

sfMatch includes the steps of spec-Match and the additional computation of a coefficient for each spatial fre-quency, based on the target amplitude spectrum: coeffSF = sum(tarSpecSF)/sum(origSpecSF).

Avanaki, A. N. (2009). Exact global histogram specification optimized for structural similarity. Optical Review, 16, 613-621.

Wang, Z. (2003). Matlab implementation of the SSIM index. Available at: www.ece.uwaterloo.ca/~z70wang/research/ssim/

Willenbockel, V., Sadr, J., Fiset, D., Horne, G. O., Gosselin, F., & Tanaka, J. W. (in press). Controlling low-level image properties: The SHINE toolbox. Behavior Research Methods.

Williams, N. R., Willenbockel, V., & Gauthier, I. (2009). Sensitivity to spatial frequency and orientation content is not specific to face perception. Vision Research, 49, 2353-2362.

Acknowledgments: We thank Alireza Avanaki for providing the ssim_sens function.

specMatch & sfMatch

histMatch

lumMatch

histMatch & specMatch

specMatch & histMatch

ITERATION 20

Illustration of sfMatch and specMatch. a: Source images and their amplitude spectra. b: Using sfMatch, the rotational average of the spectra was equated while the energy distribution across orientations was preserved. c: Using spec-Match, the spectra were equated on spatial frequencies and orientations. The output in b) and c) is shown after the rescaling of the luminance values so that absolutely all grayscale values of the three images are in the range of 0 to 255.

1 10 100Spatial frequency (cycles/image)

Rel

ativ

e en

ergy

1 10 100

106

108

1010

1012

Spatial frequency (cycles/image)

Rel

ativ

e en

ergy

CarChairFace

106

108

1010

1012

106

108

1010

1012

1 10 100Spatial frequency (cycles/image)

Rel

ativ

e en

ergy

CarChairFace

CarChairFace

INPUT

OUTPUT sfMatch

OUTPUT specMatch

a)

c)

b)

INPUT

Num

ber o

f pix

els

Num

ber o

f pix

els

0 50 100 150 200 2500

500

1000

1500

Num

ber o

f pix

els

0

500

1000

1500

Num

ber o

f pix

els

0 50 100 150 200 250

0

500

1000

1500

0

500

1000

1500

0 50 100 150 200 250

0 50 100 150 200 250

Luminance Luminance

Illustration of basic histogram matching. Left: Two base face images with their luminance histograms. Right: The corresponding SHINE output images with their matched histograms. The target was obtained using the function tarhist.

meanA: 182meanB: 204stdA: 86stdB: 62

meanA: 193meanB: 193stdA: 73stdB: 73

Illustration of the luminance matching function. Left: Two base face images with their mean luminance and contrast (std). Right: Output images equated in mean luminance and contrast.

A AB B

INPUT OUTPUT

INPUT OUTPUT