Halftoning-Inspired Methods for Halftoning-Inspired Methods for Foveation in Variable Acuity Foveation in Variable Acuity Superpixel Imager CamerasSuperpixel Imager Cameras

Thayne R. CoffmanThayne R. Coffman1,21,2

Prof. Brian L. EvansProf. Brian L. Evans11 (presenting)(presenting)Prof. Alan C. BovikProf. Alan C. Bovik11

1 1 Center for Perceptual SystemsCenter for Perceptual Systems Department of Electrical and Computer Department of Electrical and Computer EngineeringEngineering The University of Texas at AustinThe University of Texas at Austin http://www.cps.utexas.eduhttp://www.cps.utexas.edu

222121stst Century Century Technologies, Inc.Technologies, Inc. Austin, TexasAustin, Texas

November 2, 2005, IEEE Asilomar Conference on Signals, Systems, and Computers

Motivation: Foveated ImageryMotivation: Foveated Imagery Foveated imagery has Foveated imagery has

variable spatial variable spatial resolutionresolution Human visual systemHuman visual system

Provides simultaneousProvides simultaneous Wide field of viewWide field of view High resolution on High resolution on

regions of interestregions of interest Low bandwidthLow bandwidth

19% bandwidth means 19% bandwidth means 19% of “superpixels”19% of “superpixels”

No compression in No compression in talktalk Full resolutionFull resolution

(100% bandwidth)(100% bandwidth)Variable resolutionVariable resolution(19% bandwidth)(19% bandwidth)

Motivation: VASI™ CamerasMotivation: VASI™ Cameras Variable Acuity Superpixel Imager (VASI) camerasVariable Acuity Superpixel Imager (VASI) cameras

Generate foveated images by sharing charges on focal plane arrayGenerate foveated images by sharing charges on focal plane array Achieve 1000-4000 frames/sec (e.g. to measure engine RPMs)Achieve 1000-4000 frames/sec (e.g. to measure engine RPMs) Pixel sharing reconfigured to achieve a particular frame ratePixel sharing reconfigured to achieve a particular frame rate

Use of 1x1, 2x2, and 4x4 pixel sharing Use of 1x1, 2x2, and 4x4 pixel sharing [McCarley [McCarley et alet al., 2002]., 2002]

VASI is a trademark of Nova Sensors, Inc.VASI is a trademark of Nova Sensors, Inc. Images from [McCarley Images from [McCarley et alet al., 2002]., 2002]

The CatchThe Catch Desired spatial acuity (resolution) is usually Desired spatial acuity (resolution) is usually

specified as specified as a continuous amplitude function on the range (0,1]a continuous amplitude function on the range (0,1]

Translate desired resolution function to VASI™ Translate desired resolution function to VASI™ binary share/no-share control signal at very high binary share/no-share control signal at very high frame ratesframe rates

Foveation like the human eye (left pixelation)Foveation like the human eye (left pixelation)Two fovea (right pixelation)Two fovea (right pixelation)

Halftoning for VASI Control Halftoning for VASI Control SignalsSignals

Select a small number of test imagesSelect a small number of test images Manually specify desired resolution (using Gaussians)Manually specify desired resolution (using Gaussians) Evaluate halftoning methods to control signal Evaluate halftoning methods to control signal

translationtranslation Figures of merit to predict object recognition Figures of merit to predict object recognition

performanceperformance Peak SNR (PSNR)Peak SNR (PSNR) Weighted SNR (WSNR)Weighted SNR (WSNR) Universal Quality Index (UQI)Universal Quality Index (UQI) Percentage of Bandwidth (PBW)Percentage of Bandwidth (PBW) Control

signal for

charge sharing

at a pixel X

X

Shared up Shared left

Halftoning Methods ExploredHalftoning Methods Explored Classical screeningClassical screening

9-level clustered dot9-level clustered dot 9-level dispersed dot9-level dispersed dot

Block error Block error diffusiondiffusion

Floyd-Steinberg Floyd-Steinberg error diffusionerror diffusion

Blue noise ditheringBlue noise dithering White noiseWhite noise Specialized (non-Specialized (non-

general) methodsgeneral) methods vasiHalftonevasiHalftone vasiHalftone2vasiHalftone2

Dispersed dot screeningDispersed dot screeningF-S error diffusionF-S error diffusion

White noiseWhite noise vasiHalftonevasiHalftone

Specialized MethodsSpecialized Methods Generate semi-regularly spaced squaresGenerate semi-regularly spaced squares Square size varies with inverse of desired bandwidthSquare size varies with inverse of desired bandwidth Side is 2Side is 2KK in vasiHalftone & unconstrained in in vasiHalftone & unconstrained in

vasiHalftone2vasiHalftone2Full-resolution Full-resolution imageimage

Binarized Binarized control control (sharing) (sharing) signalsignal

Foveated imageFoveated imageContinuous Continuous desired desired resolution resolution signalsignal

Nontrivial Translation of Nontrivial Translation of Control SignalControl Signal

Halftoning algorithms aim to achieve a specific ratio of white Halftoning algorithms aim to achieve a specific ratio of white or black pixels, e.g.or black pixels, e.g. For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”)For constant I(r,c)=0.1, 10% of pixels will be white (“don’t share”) For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”)For constant I(r,c)=0.8, 80% of pixels will be white (“don’t share”)

But bandwidth and resolution are functions of geometry alsoBut bandwidth and resolution are functions of geometry also

Example 1Example 1 Example 2Example 2

50% of pixels don’t share charge: 1% bandwidth50% of pixels don’t share charge: 1% bandwidth46% of pixels don’t share charge: 15% bandwidth46% of pixels don’t share charge: 15% bandwidth

Control signalControl signal Resulting imageResulting image Control signalControl signal Resulting imageResulting image

Nontrivial Translation of Control Nontrivial Translation of Control SignalSignal

Relationship between percent of “don’t Relationship between percent of “don’t share” pixels and bandwidth is different for share” pixels and bandwidth is different for every halftoning methodevery halftoning method Eliminate nonlinearity by Eliminate nonlinearity by

applying an inverse functionapplying an inverse function Implemented with lookupImplemented with lookup

tables storing x = ftables storing x = f-1-1(y)(y) Given target bandwidth andGiven target bandwidth and

halftoning method, findhalftoning method, findaverage value (average value (xx-axis) to use-axis) to usein continuous control signalin continuous control signal

Stairstep patterns inStairstep patterns inrelationship limitrelationship limitcontrol over bandwidthcontrol over bandwidth

Floyd-Steinberg gives piecewise linear map Floyd-Steinberg gives piecewise linear map and best bandwidth controland best bandwidth control

Nontrivial Translation of Nontrivial Translation of Control SignalControl Signal

Results are greatly improvedResults are greatly improved Better bandwidth controlBetter bandwidth control Better foveation resultsBetter foveation results

Floyd-Steinberg (F-S) results belowFloyd-Steinberg (F-S) results belowDesired bandwidth Desired bandwidth =11.9% from ideal =11.9% from ideal control signalcontrol signal

UncompensatUncompensated control ed control signalsignal

Achieved Achieved bandwidth = bandwidth = 2.6%2.6%

Compensated Compensated control signalcontrol signal

Achieved Achieved bandwidth = bandwidth = 12.5%12.5%

Results: F-S Error DiffusionResults: F-S Error Diffusion Good performance and good bandwidth controlGood performance and good bandwidth control

Good SNR in foveae means accurate object recognitionGood SNR in foveae means accurate object recognition Good SNR in periphery means good object detectionGood SNR in periphery means good object detection Good bandwidth control means precise VASI frame Good bandwidth control means precise VASI frame

rate controlrate controlOriginalOriginal Sharing SignalSharing Signal Resulting ImageResulting Image

PSNR = 17.5 dB (33.3 dB in ROI)PSNR = 17.5 dB (33.3 dB in ROI)WSNR = 16.4 dB (33.8 dB in ROI)WSNR = 16.4 dB (33.8 dB in ROI)

Desired BW = 11.6%Desired BW = 11.6%Actual BW = 12.1%Actual BW = 12.1%Inflation = 4%Inflation = 4%

Results: vasiHalftone and Results: vasiHalftone and vasiHalftone2vasiHalftone2

For a given desired resolution signal, methods For a given desired resolution signal, methods consistentlyconsistently Had better PSNR & WSNR than other methodsHad better PSNR & WSNR than other methods Overshot desired bandwidth by ~30-100%Overshot desired bandwidth by ~30-100%

Essentially “cheating” by using extra bandwidthEssentially “cheating” by using extra bandwidthOriginalOriginal Sharing SignalSharing Signal Resulting ImageResulting Image

PSNR = 13.3 PSNR = 13.3 dBdB

WSNR = 16.9 WSNR = 16.9 dBdB

Desired BW = Desired BW = 9.6%9.6%

Actual BW = Actual BW = 18.8%18.8%

Inflation = 97%Inflation = 97%

Results: Other Halftoning Results: Other Halftoning MethodsMethods

MethodMethod Performance Performance (SNR)(SNR)

Bandwidth Bandwidth controlcontrol

Block error Block error diffusiondiffusion

PoorPoor GoodGood

Classical Classical screeningscreening

DecentDecent PoorPoor

Stochastic Stochastic methodsmethods

PoorPoor ““Catastrophic Catastrophic gray-out”gray-out”

OriginalOriginal Blue noiseBlue noiseBlock error diffusionBlock error diffusion

OriginalOriginal Clustered dotClustered dot Dispersed dotDispersed dot White noiseWhite noise

ConclusionsConclusions Floyd & Steinberg error diffusion gives the best Floyd & Steinberg error diffusion gives the best

results while still being able to control bandwidth results while still being able to control bandwidth preciselyprecisely

vasiHalftone and vasiHalftone2 vasiHalftone and vasiHalftone2 Consistently the best PSNR, WSNRConsistently the best PSNR, WSNR Poor bandwidth control – overshot specifications by 30-Poor bandwidth control – overshot specifications by 30-

100%100% Bandwidth inflation means it’s not a fair comparison Bandwidth inflation means it’s not a fair comparison

(they’re cheating)(they’re cheating) Stochastic methods (white & blue noise) perform Stochastic methods (white & blue noise) perform

poorlypoorly Outperformed by deterministic approachesOutperformed by deterministic approaches Susceptible to “catastrophic gray-out”Susceptible to “catastrophic gray-out”

Classical screening performs marginally Classical screening performs marginally andand has has poor bandwidth controlpoor bandwidth control

Recent WorkRecent Work vasiHalftone3 and vasiHalftone4vasiHalftone3 and vasiHalftone4

Extensions to eliminate simplifying assumption that Extensions to eliminate simplifying assumption that VASI™ shareUp and shareLeft signals are equalVASI™ shareUp and shareLeft signals are equal

This eliminates single-pixel artifacts in non-foveal This eliminates single-pixel artifacts in non-foveal regionsregions

Eliminated lookup table (LUT) in F-S approach Eliminated lookup table (LUT) in F-S approach by determining closed-form inverse by determining closed-form inverse relationshiprelationship Significant speedupSignificant speedup

Greatly shrank LUT in vasiHalftone & Greatly shrank LUT in vasiHalftone & vasiHalftone2 approachesvasiHalftone2 approaches Leveraged “stairstep” form of inverse relationshipLeveraged “stairstep” form of inverse relationship 10x speedup in vasiHalftone, 4x speedup in 10x speedup in vasiHalftone, 4x speedup in

vasiHalftone2vasiHalftone2 2121stst Century Technologies and Nova Sensors Century Technologies and Nova Sensors

are actively collaborating on further workare actively collaborating on further work Sponsored by U.S. Air Force Research LaboratorySponsored by U.S. Air Force Research Laboratory

Background ReferencesBackground References B.E. Bayer, “An optimum method for two level rendition B.E. Bayer, “An optimum method for two level rendition

of continuous-tone pictures,” of continuous-tone pictures,” Proc. IEEE Int. Conf. on Proc. IEEE Int. Conf. on Communications, Conf. Rec.Communications, Conf. Rec., pp. (26-11)-(26-15), 1973., pp. (26-11)-(26-15), 1973.

R. Floyd and L. Steinberg, “An adaptive algorithm for R. Floyd and L. Steinberg, “An adaptive algorithm for spatial grayscale,” spatial grayscale,” Proc. SID’76Proc. SID’76, pp. 75-77, 1976. , pp. 75-77, 1976.

P. McCarley, M. Massie, and J.P. Curzan, “Large format P. McCarley, M. Massie, and J.P. Curzan, “Large format variable spatial acuity superpixel imaging: visible and variable spatial acuity superpixel imaging: visible and infrared systems applications,” infrared systems applications,” Proc. SPIE, Infrared Proc. SPIE, Infrared Technology and Applications XXXTechnology and Applications XXX, vol. 5406, pp. 361-, vol. 5406, pp. 361-369, Aug 2002. 369, Aug 2002.

V. Monga, N. Damera-Venkata, and B.L. Evans, V. Monga, N. Damera-Venkata, and B.L. Evans, Halftoning Toolbox for MatlabHalftoning Toolbox for Matlab.. Version 1.1 released Version 1.1 released November 7, 2002. Available online at November 7, 2002. Available online at http://http://www.ece.utexas.edu/~bevans/projects/halftoningwww.ece.utexas.edu/~bevans/projects/halftoning//. .

R.A. Ulichney, “Dithering with blue noise,” R.A. Ulichney, “Dithering with blue noise,” Proc. IEEEProc. IEEE, , vol. 76, pp. 56-79, Jan 1988. vol. 76, pp. 56-79, Jan 1988.

Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based Z. Wang, A.C. Bovik, and L. Lu, “Wavelet-based foveated image quality measurement for region of foveated image quality measurement for region of interest image coding,” interest image coding,” Proc. IEEE Int. Conf. Image Proc. IEEE Int. Conf. Image Proc.Proc., vol. 2, pp. 89-92, Oct 2001. , vol. 2, pp. 89-92, Oct 2001.