Fast and Stable Color Balancing for Images and Augmented Reality

Thomas Oskam 1,2 Alexander Hornung 1 Robert W. Sumner 1 Markus Gross 1,2

1 Disney Research Zurich 2 ETH Zurich

Abstract

This paper addresses the problem of globally balanc-ing colors between images. The input to our algorithm isa sparse set of desired color correspondences between asource and a target image. The global color space trans-formation problem is then solved by computing a smoothvector field in CIE Lab color space that maps the gamut ofthe source to that of the target. We employ normalized ra-dial basis functions for which we compute optimized shapeparameters based on the input images, allowing for morefaithful and flexible color matching compared to existingRBF-, regression- or histogram-based techniques. Further-more, we show how the basic per-image matching can beefficiently and robustly extended to the temporal domain us-ing RANSAC-based correspondence classification. Besidesinteractive color balancing for images, these properties ren-der our method extremely useful for automatic, consistentembedding of synthetic graphics in video, as required byapplications such as augmented reality.

1. IntroductionPablo Picasso highlighted the importance of color in

painting with the famous quote, “They’ll sell you thousandsof greens. Veronese green and emerald green and cadmiumgreen and any sort of green you like; but that particulargreen, never.” Today, the same struggle to capture the per-fect nuance of color lives on in digital photography andimage-based applications. The colors that are ultimatelyrecorded by both consumer-grade and high-end digital cam-eras depend on a plethora of factors and are influencedby complex hardware and software processing algorithms,making the precise color quality of captured images difficultto predict and control. As a result, one generally has to relyon post-processing algorithms to fix color-related issues.

This problem is generally known as color balancing orcolor grading, and plays a central role in all areas involvingcapturing, processing and displaying image data. In digi-tal photography and filmmaking, specifically trained artistswork hard to achieve consistent colors for images capturedwith different cameras, often under varying conditions and

Source Image Color BalancedTarget Image

Tracked Colors balancing Augmented Image

Rendered Objects

Figure 1. Two applications of our color balancing algorithm. Top:an underexposed image is balanced using only three user selectedcorrespondences to a target image. Bottom: our extension fortemporally stable color balancing enables seamless compositingin augmented reality applications by using known colors in thescene as constraints.

even for different scenes. With today’s tools this process re-quires considerable, cost-intensive manual efforts. Similarissues arise in augmented reality applications, where com-puter generated graphics must be embedded seamlessly andin real-time into a video stream. Here, achieving consis-tent colors is critical but highly challenging, since white-balance, shutter time, gamma correction, and other factorsmay continually change during acquisition and cannot beperfectly controlled or addressed through calibration alone.Therefore, an efficient correction and adaptation of colorsthat is agnostic to the underlying capture hardware is highlydesirable in such AR-related applications. However, asidefor a few preliminary efforts [13, 14] practically applicablesolutions to this problem are yet to be developed.

This paper contributes a real-time color balancing algo-rithm that is able to globally adapt the colors of a sourceimage to match the look of a target image, requiring onlya sparse set of color correspondences. Fundamentally, ourapproach is based on radial basis function (RBF) interpo-lation. However, we show that, for an optimal RBF-basedcolor matching, it is crucial to optimize for the shape of the

4321

employed basis functions. As a second main contribution,we then extend the per-image computation to be temporallyconsistent, enabling the application of the basic balancingmechanism in video-based applications such as augmentedreality. A key component to the success of this system isa reliable RANSAC-based classification algorithm that se-lects a temporally stable set of correspondences used for thebasic color balancing. We present a fast GPU-assisted im-plementation of our color balancing algorithm that achievesrun times of 1ms and below on real-world applications. Inthis paper and the accompanying video, we show results forinteractive color editing of images as well as for improvedvisual quality and consistency of video and computer gen-erated images in augmented reality applications.

2. Related WorkIn this section, we discuss related work on histogram

matching, user controlled color transfer, and color calibra-tion.

Statistics and Histogram Matching: This is the pro-cess of transferring the color distribution statistics from oneimage to another image, pioneered by Reinhard et al. [20].They proposed computing the mean and standard devia-tion of the color distribution in the color space. Then, byscaling and translating the parameters onto the ones of thetarget image, they achieve automatic color transfer. Theapproach has been applied to histograms and extended toachieve a mapping in different color spaces [25], minimumcost linear mapping [19], and per-region execution of thealgorithm [11]. Further extensions have been proposed thatoptimize the images after matching to avoid discontinuitiesin color gradients [18, 26].

While statistics and histogram matching methods for au-tomatic color transfer can produce very appealing results,they provide only limited control. Direct control over colorchanges is often required in image editing and difficult toachieve with these methods.

Color transfer with user interaction: Abadpour etal. [2] first proposed an image color transfer algorithm withuser interaction. In their work, the user is allowed to de-note corresponding regions between images. Colors arethen locally matched using fuzzy principal component anal-ysis. Other methods have been proposed that aid the user inlocal color matching [17, 22]. The user is allowed to se-lect a local region, and the actions are propagated through-out adjacent parts of the image. An et al. [6] propose ascribble-based framework for color transfer between im-ages. Fast and interactive methods for editing images andvideo have also been proposed. An et al. [5] propagate lo-cal edits by assisting a user with iteratively refining roughedits to areas of spatial and appearance similarity. Farbmanet al. [9] propose the use of diffusion maps to propagateimage editing information. In these approaches the impact

of user controlled edits is typically targeted towards localcolor changes. Our work, in contrast, targets global colorbalancing using only sparse and possibly incomplete input.

Li et al. [15] provide an approach for image and videoediting with instant results. They employ radial basis func-tion interpolation in a feature space to propagate sparse ed-its locally. In contrast to this work, we are formulating theproblem as a global color space transformation independentfrom the image space and we optimize the shape of basisfunctions to achieve color balancing behavior found in ex-ample images. We show that the basis function used by Li etal. is suboptimal for the purpose of global image balancing.

Color calibration and balancing: Color calibration isthe process of computing the color distribution function of acamera to allow recorded images to be consistently colored.Adams et al. [3] gives an overview of standard camera cal-ibration techniques using linear transformations. Usually,a 3x3 matrix is computed using regression. A related areais color consistency. Humans perceive colors to be consis-tent even under different lighting conditions. This propertyis not available for digital images. Colors have differentnumerical values under different illuminations. In order tocounter this issue, different algorithms have been proposed.An overview over different algorithms is provided by Ar-gawal et al. [4] and Cohen [7]. In contrast to our approach,many of these methods do not apply additional informationsuch as correspondences or known colors in a scene to bal-ance the images. Therefore, these methods are less interac-tive and less robust when applied to videos.

There is also significant work that enhances the qualityof imagery using a database of example images [8, 12, 21,23]. The work of Yang et al. [27] achieve color compensa-tion by replicating the non-linear relationship between thecamera color and luminance. Wang et al. [24] transfer colorand tone style of expert photographed images onto the im-ages from cheap cameras by learning the complex transfor-mation encoded in local descriptors. Recent work of Linet al. [16] revisits the radiometric calibration of camerasby investigating 10,000 images from over 30 cameras andintroduces a new imaging model for computing the colorresponse curves. Our parameter optimization of the radialbasis functions is also based on extracting image proper-ties form data sets. However, we design our functions tobe sparsely sampled so that they provide appealing resultswhen interpolated in order to considerably increase perfor-mance.

Color Balancing in Augmented Reality: Only recentlythere has been research in improving the compositing of im-ages in augmented reality. Klein et al. [13] examine andreproduce several image degradation effects of digital cam-eras to increase the realism of augmentation. However, theyare not considering color shifts due to camera parameterchanges. The works of Knecht et al. [14] propose to bal-

4322

Image Gamut + Correspondences

SmoothVector Field

Transformed Image Gamut

Figure 2. The idea of vector space color balancing. On the left,the source colors are shown in black and the corresponding targetcolors in white. The idea is to interpret all correspondences asvectors in color space and smoothly extend them to a vector fieldwith which the gamut of the image is transformed. This procedurecreates a continuous and flexible color transfer function.

ance rendered images using colors in the scene. However,they are applying a linear regression model and do not cor-rect for corrupted colors due to occlusions or reflections.

3. Vector Space Color BalancingGiven a sparse set of color correspondences, the goal is

to transform the color gamut of an image such, that (i) col-ors that have references get as close as possible to that pointin the color space, and (ii) colors, for which no referencesare available, are transformed in a plausible way.

We interpret the given color correspondences as vectorsin the color space. In order to produce a transformationof the gamut, we compute a vector field around the refer-ence vectors that achieves the aforementioned goals. Theconcept is demonstrated in Figure 2. The correspondencevectors themselves represent constraints. We use normal-ized radial basis function interpolation that propagates theseconstraints to the rest of the color space. Section 3.1 ex-plains how this is performed. The selection of basis func-tions, however, influences how the correspondence vectorsare propagated through the color space and ultimately howcolors without constraints are balanced. Therefore, we pro-pose optimizing the shape of a basis function based on ex-ample images. This is discussed in section 3.2.

3.1. Vector Field Computation

For a given pair of colors (ci, di) in the three-dimensional CIE Lab space, we define the ci as supportpoints, and the vectors vi = ||di − ci|| as data values. Foreach vector vi, a basis function φi is provided that describesthe weight with which the vector is distributed in space. Foreach new point e in the color space, an according transla-tion vector v(e) is computed as a normalized sum of theweighted φi

v(e) =1∑n

i=1 φi(e)

n∑i=1

φi(e)wi. (1)

The summation of the support functions allows an inde-pendent evaluation of each point in the vector field, which

norm. ShepardGaussianinv. quadric

avg. functiondeviation

1

00 30Color Space Distance

Figure 3. Plots of the basis functions we consider for our colortransfer. On the left, the three functions are shown in relative scale.Note their different fall-off behaviors. On the right, the curves areshown with optimized shape parameters in their absolute scale.The optimized curves significantly differ in their final spread andvariation.

suits a completely parallel implementation for each pixel inan image. The individual weights wi allow the treatmentof the φi as radial basis functions (RBF). RBF interpolationperforms a least squares minimization on the overlappingsupport between the individual φi by choosing weights wi

such, that the squared difference between the data vectorsvi and the weighted sum of the basis functions at ci is min-imized in the least squares sense

argminwi

(||vi − v(ci)||2). (2)

This results in a system of equations v = Φw per colordimension, where the matrix Φ is the same in all three sys-tems and contains, at each position (i, j), the normalizedsupport of all cj seen at the interpolation point ci. Theweight vectors wi can be composed by inverting Φ and as-sembling the individual components from the three systemsof equations.

Conflicting constraints of two source colors ci and cj cancause the matrix Φ to become near-singular. This causescomponents of the resulting wi to become negative. Neg-ative values create an inverted distribution of the constraintvectors vi in the color space an produce undesired arti-facts [15]. We avoid this problem by clustering supportpoints that lie within the limit of perceivable distance (1.0 inthe CIE Lab space). Additionally, we select basis functionsthat have a small spread to avoid large overlapping supportbetween constraints. This keeps the matrix Φ from degen-erating. In Section 3.2, we discuss how to optimize basisfunctions and show that our normalized Shepard functioncombines optimal and robust interpolation results with avery small spread, in contrast to the generally utilized Gaus-sian function.

3.2. Choice of Basis Functions

The choice of an appropriate basis function both influ-ences the global impact of color constraints and the stabilityof the interpolation [15]. Using RBF interpolation, we canmake sure that the effect of overlapping support betweenthe basis functions is minimized. However, it is still unclearwhich functions should be used for interpolation and how toselect their parameters. To find out how to best interpolate

4323

10

Erro

r

ε

Exposure Time Gain Color Intensity

5

norm. ShepardGaussianinverse quadric

Figure 4. Average reconstruction error of the considered basis functions while interpolating 50% of the color checker fields under changingcamera parameters. The error is shown as a function of the ε-values for the normalized Shepard (red), the Gaussian (green), and the inversequadric (blue) basis functions. The plots show that the ε value significantly influences the result.

colors given a sparse set of correspondences, we test the fol-lowing three functions with different fall-off behavior overdistance (see Figure 3).

normalized Shepard si(cj) = (1 + ||ci − cj ||)−ε

Gaussian gi(cj) = e−(ε||ci−cj ||2)

inverse quadric qi(cj) =1−α

1+(ε||ci−cj ||)2 + α

The normalized version of the Shepard function has thequickest fall-off. It gets surpassed after a while by the Gaus-sian function that decreases exponentially. The constantvalue we add to the inverse quadric function (α = 0.01 inour experiments) guarantees a minimum value in one of ourfunctions to distances at infinity.

Each of the functions is parameterized with a variable εthat influences the width of the function spread. This pa-rameter allows to find the optimal reconstruction capabilityof each function by optimizing for ε.

To compare the performance of each parameterizedfunction on reconstructing colors, we first create datasets by filming a color calibration checker (Digital Col-orChecker SG in our experiments). During video capture,internal camera parameters (e.g. exposure or contrast) arechanged individually so that the global color changes arepart of the data.

In each data set, one frame with default camera settings

Target Image Cool White Bal.

1 ColorRef.

2 ColorRef.

4 ColorRef.

6 ColorRef.

Figure 5. Example of the global impact of our color balancing us-ing only a few correspondences. The target image on the left is re-constructed using a photograph of the same scene with a differentwhite balance as source image. The top row shows the successiveaddition of references (red dots). The images in the bottom rowvisualize the difference in CIE Lab space between the balancedimage and the original.

is selected as the base frame. In this frame, 50% of the colorfields are selected at random, and used as data points. Thecorresponding color fields in all other frames in the dataset are used as references. The score of a function (witha specific ε-value) is now computed as the average recon-struction error of the other 50% (not referenced) colors inthe base frame across all frames of the data set. Figure 4shows plots from some of our tests. The error is shown forthe normalized Shepard (red), the Gaussian (green), and theinverse quadric (blue) functions for an increasing ε-value.The plots are shown with differently scaled ε for each func-tion, but with the same scaling across all tests.

For each of the three functions there is an optimal ε.This surprising result shows that there is a clear correla-tion between the spread of a basis function and its capabil-ity of reconstructing global color changes. Second, acrossall tests, the magnitude of the optimal ε is in a similarrange. The ε-values we determined though our tests areε = 3.7975 (σ = 0.5912) for the normalized Shepard,ε = 0.0846 (σ = 0.0289) for the Gaussian, and ε = 0.0690(σ = 0.0289) for the inverse quadric function, where σ-values are the standard deviations.

The shape of the optimized functions (and their devi-ation due to σ as dashed curves) are shown Figure 3 onthe right. Our normalized version of the Shepard func-tion clearly shows the smallest spread of all three func-tions, rendering it the most stable function for RBF inter-polation. Note, that the Gaussian function has significantlywider spread and also a much higher variance, making it asuboptimal choice for global color balancing. The impacton the number of references using our normalized Shepardfunction with optimal ε for global color balancing is demon-strated in Figure 5. With only a few correspondences, theerror in the CIE Lab space is reduced considerably.

4. Temporal Color Tracking

A typical scenario where the permanent update of thecolor balance improves the visual quality is augmented re-ality, where a synthetic rendering should be embedded ina seamless manner into a video stream. In order to tacklethis problem, we propose to use known color patches in thescene (See Figure 1). If these colors can be robustly tracked

4324

Frame

Tracked Colors

Reference Colors

Detect Outlier Reconstruct Colors

Last Valid Colors

Create References

Target Colors

Source Colors

Digital Image

RANSAC (A�ne Model)

Figure 6. Overview of our stable color tracking pipeline. First, color values are extracted from the video frame and corrupt ones are rejectedusing RANSAC and temporal coherence. Then, missing colors are added using information from the previous frames. Finally, the stablecolor list is combined with colors from a digital image to create the transfer constraints that can be used for balanced augmentation ofrendered objects.

over time, they can be used as color references for their sup-posed values. These references then automatically balancerendered images to the video footage, adapting the render-ings as the color of the video frames change due to cameraadjustments.

We base our color tracking on the observation that de-sired color changes in the video stream are global. Thesechanges are due to the camera automatically adjusting inter-nal parameters such as exposure or gain. The color patchesin the scene that receive these color changes will be usedas constraints for the balancing of their known unbalancedvalues. However, due to occlusions or specular reflections,some of the tracked color patches will be corrupt. Thesecolor distortions (e.g. occlusions, reflections) happen lo-cally. This fact can be utilized to remove them as outliers.

Figure 6 gives an overview over our robust color trackingpipeline. For each frame, the following steps are performed:

1 Extract Colors. The first step is to extract the colorsdi from the video frame. The position tracking provides thepositions in the image where the target colors are found. Inour implementation we use ARToolKit marker tracking [1],but any other position tracking solution can be used as well.To reduce per-pixel noise we perform a neighborhood aver-aging over a small window (usually 7 by 7).

2 Detect Outlier. To detect corrupted colors we can fit aglobal model between the extracted di and a set of referencecolors. These reference colors ri can be taken from the dig-ital image of the marker or extracted from the first framesof the video in a pre-processing step. In order to keep thenumber of false positives low a conservative model shouldbe fitted between the ci and ri. We chose the affine trans-formation model ci = A ri + t [3], which can be computedanalytically and avoids over fitting to corrupted colors.

Outlier in the scene colors di can now be detected by fit-ting the affine model using the random sample consensusalgorithm (RANSAC) [10]. This first step allows remov-ing most of the corrupted colors. However, false positivesmay slip though this outlier detection. In order to furtherincrease the robustness of the tracking, we separate the re-maining inliers into two groups. Trusted inliers are colorsthat have not been detected as outliers for more than κt

frames. These colors are used as valid target colors. Inliercolors that have been detected as outlier in one of the past κtframes may be false positives. These colors are also treatedas outlier. In our experiments we found that false positivesrarely appear for more than two frames. Therefore, in ourexperiments, we mostly used κt = 3.

3 Reconstruct Colors. At this point, corrupted colorshave been removed from the list of di. Additionally, therisk of false positives has been decreased by taking temporalconsistency into account. Now, in order to maximize theamount of colors available for color balancing, we can usethe last valid colors of each tracked point from the previousframes. These colors are stored when a tracked value isregarded as trusted inlier (being not an outlier for more thanκt frames). Colors values, that are suddenly corrupt due toocclusions or specular reflections can then be replaced byan updated version of their last valid value. The update ofthese last valid colors is performed by applying the globalaffine transformation performed by all trusted inlier colorssince the last frame. In our implementation, we perform thiscolor reconstruction only for outlier that have been inliersfor more than κc frames. This removes colors that wereonly detected as inliers for a short period and increases therobustness. We call these colors comeback points. In ourimplementation we have set κc = 10.

4 Create References. The final color references (ci, di)for the balancing of rendered objects can now be created.For all di, that are either trusted inlier or a comeback points,the corresponding color ci can be extracted from the digitalimage of the marker. The constraints (ci, di) now describethe color transfer function from rendered footage to the cur-rent video frame color space.

With our fast global color balancing for sparse color cor-respondences we are able to adapt the colors to fit to a targetimage. With our robust color tracking, we can extend thisinto the temporal domain. Using a known marker in thescene (e.g. the cover of a book) we can balance newly ren-dered footage to augment the video and increase the realismof augmented reality applications.

4325

5. ImplementationOur color balancing algorithm can be implemented by

simply extending any given fragment shader. The referencecolors ci and weights wi are transferred to the GPU as tex-ture data. The matrix inversion is calculated beforehand onthe CPU as it only needs to be performed once. The GPUfragment shader algorithm is described in Algorithm 1.

Algorithm 1 Fragment Shader Color Balancing1: e← pixelColor()2: eLab ← rbg2lab(e)3: ve ← (0, 0, 0)4: se ← 05: for i = 1→ n do6: [ci,wi]← dataTexture(n/i)7: r ← phi(ci, eLab)8: ve ← ve + rwi

9: se ← se + r10: end for11: eLab ← eLab + ve/se12: return lab2rgb(eLab)

0.05

1.30

5 10 20 40# References

Tim

e [m

s]

Figure 7. The perfor-mance of our imple-mentation. The greenfunction shows the run-time for the color bal-ancing on the GPU, theblue function the time tosolve the RBF equationsystem on the CPU.

First, the original color eof the pixel is determined (line1). The pixel color is the re-sult of the given shading al-gorithm. This color is thentransformed to CIE Lab space(line 2). Then, the variablesve and se are initialized (line 3& 4). They are used to com-pute intermediate results of thetransformation vector computa-tion. Now, a loop is performedn-times, which is the numberof color correspondences. Ineach loop, first, the color ci andweight vector wi are acquiredthrough texture lookup (line 6).Then, the function φ is applied using the current color eLaband the support color ci (line 7). This gives the functionvalue r, which is used to add the current weight wi to ve(line 8). Additionally, the sum of all the function values r isstored in se (line 9), so that after the loop the transformationvector ve can be normalized and used to translate the origi-nal pixel color in CIE Lab space (line 11). The final color isthen transformed back into RGB space and returned to theframe buffer (line 10).

We have implemented the color balancing algorithm onan Intel Core i7 3.2 GHz with Nvidia GeForce GTX 460.The run-time in ms is shown in Figure 7 for an increasingnumber of references. The solver for the equation system(blue graph) is the bottleneck, as it involves a matrix inver-

sion in the size of the number of references. The RBF inter-polation (blue graph) however performs with around 0.05ms almost independently of the number of references.

6. Results and Applications

In this section we present various color balancing resultsfor image editing and augmented reality applications, andcompare them to realated works. Please see also the accom-panying video where we show several examples for bothcolor balancing and augmented reality.

6.1. Interactive Color Transfer and Correction

Given a source image and a target image with a desiredlook, the user can interactively define a color mapping byclicking correspondences. The result is instantly visible,and the user can drag the correspondence points around tosee the effect. A number of examples are shown in Figure 8that demonstrate the flexibility of our approach. Due to theglobal nature of the vector field color transformation, it isvery easy to balance an image to exactly match an examplephotograph. Even completely changing the color composi-tion of an image can be achieved robustly using the samekind of input.

Figure 9 shows a series comparisons of our color balanc-ing method with other color balancing approaches. Sinceour method is designed and optimized for global color bal-ancing, it is able to transfer the colors from the target imageto the source image with only a sparse set of references.User edits in Photoshop changing indirect parameters (hue,saturation, etc.) can be unintuitive and time consuming, andit is difficult to achieve exact results. Achieving the correctbalance between the yellow and red colors in the bottomexample are difficult with only indirect manipulation. Boththe Photoshop Color Match function and the approach ofReinhard et al. [20] perform very similarly. Both methodsautomatically achieve a global balancing of the image butcannot recreate the exact shade of the Taj Mahal. Using thesparse input used for our color balancing, the approach of Liet al [15] struggles, as their method is specifically designedfor local edits. In areas with strong gradients (e.g. the domeof the Taj Mahal) their approach produces artifacts. Linearregression is, due to its limited flexibility, not able to satisfythe constraints.

6.2. Augmented Reality Color Balancing

While the color transfer tool only utilizes the flexiblevector space color transfer, applications in augmented re-ality become possible when combining it with our robustcolor tracking over time. A known marker in the scene, likethe cover of a book, can be tracked and the colors refer-enced with a digital copy of the image. Figure 10 showstwo examples of color balancing for augmented reality.

4326

Targ

et Im

ages

Sour

ce Im

ages

Our

Res

ults

Figure 8. Several examples of our sparse color balancing. The colored circles in images mark the color correspondences.

Target Image Our ApproachLi et al. 2010Lin. RegressionPhotoshopColor Match

PhotoshopUser Edits

Source Image Reinhard et al. 2001

Figure 9. Comparison of our approach with other methods for image balancing using sparse correspondences.

The top example shows a series of comparisons betweenour color correction and linear regression. It can be ob-served that our approach is more accurate for colors that areavailable in the video stream and more robust for colors thatare not tracked. The bottom row shows an example wherean animated image is placed in a book. Due to the colortracking of the real image on the same page we are able tomore realistically embed the animated figure into the video.Note that the adjustment of the colors in the flat shaded ex-amples would only work for one frame as the white balanceand global light change over time.

Generally, in augmented reality applications, it is crucialto perform proper image balancing for convincing augmen-tation. A viewer is especially sensitive to colors that alreadyappear in the video stream, and will notice when similar col-ored renderings are balanced differently. The results in Fig-ure 10 and in the accompanying video demonstrate, that ourcolor tracking and balancing is able to achieve the desiredaccuracy, while linear regression [14] struggles.

7. DiscussionIn this paper, we have presented an approach for interac-

tive image-based color balancing using only a sparse set ofcorrespondences, as well as an extension for temporally-consistent rendering for augmented reality applications.

Through our proposed global optimization of interpolationfunctions, we provide a tool to optimize function parametersto mimic color changes from example images and optimallyadjust colors accordingly.

The proposed color correction and tracking algorithmshave some limitations that direct us to areas of future work.The color correction is not able to reconstruct missing in-formation. Over- or underexposed images can flatten colorgradients to the point where only a few colors remain. Ourcolor transfer method will not smooth spatially, and there-fore is not able to increase the amount of colors present ina gradient. Coupling the color transfer algorithm to cameraexposure setting is an area of future work that could addressthis issue.

The color tracking algorithm relies on flat color areasin the image to work well. This requirement is orthogonalto many tracking algorithms that use features like cornersor edges. However, our algorithm will reject such colorsinstead of producing false results. In our current imple-mentation we employ equidistant sampling of points on themarker surface where the colors near features get removedby our tracking. Improving this sampling using informationsuch as edges or gradients in the marker image would be aninteresting topic to explore.

4327

Flat

Sha

ded

Our

Bal

anci

ng

Trac

ked

Poin

ts

Input Frames Tracked Points Lin. RegressionOur Balancing Our Balancing Lin. RegressionO

utlie

rU

ntru

sted

Trus

ted

Com

ebac

k

Figure 10. Two examples of our color tracking and balancing in augmented reality applications. Outliers (red x) arising due to specularreflection are detected and supported with comeback colors (yellow triangle). The top row shows a series of comparisons between ourcolor correction and linear regression. Our approach is both more accurate in recreating the exact image colors as well as produce moreplausible color balancing for unknown colors (i.e. green frog). The bottom row shows an example where an animated image is placed in abook and balanced by tracking the colors of a real image.

References[1] ARToolKit. http://www.hitl.washington.edu/

artoolkit/. [Online; accessed 29-June-2012].[2] A. Abadpour and S. Kasaei. A fast and efficient fuzzy color

transfer method. In Symp. on Signal Proc., pages 491 – 494,2004.

[3] J. Adams, K. Parulski, and K. Spaulding. Color processingin digital cameras. IEEE Micro, 18(6):20–30, Nov. 1998.

[4] V. Agarwal, B. R. Abidi, A. Koschan, and M. A. Abidi. Anoverview of color constancy algorithms. Journal of PatternRecognition Research, pages 42–54, 2006.

[5] X. An and F. Pellacini. Appprop: all-pairs appearance-spaceedit propagation. ACM Trans. Graph., 27(3), 2008.

[6] X. An and F. Pellacini. User-controllable color transfer.Comput. Graph. Forum, 29(2):263–271, 2010.

[7] N. Cohen. A color balancing algorithm for cameras. EE368Digital Image Processing, 2011.

[8] K. Dale, M. K. Johnson, K. Sunkavalli, W. Matusik, andH. Pfister. Image restoration using online photo collections.In ICCV, pages 2217–2224, 2009.

[9] Z. Farbman, R. Fattal, and D. Lischinski. Diffusion maps foredge-aware image editing. ACM Trans. Graph., 29(6):145,2010.

[10] M. A. Fischler and R. C. Bolles. Random sample consensus:A paradigm for model fitting. Commun. ACM, 24(6):381–395, 1981.

[11] S. Kagarlitsky, Y. Moses, and Y. Hel-Or. Piecewise-consistent color mappings of images acquired under variousconditions. In ICCV, pages 2311–2318, 2009.

[12] S. B. Kang, A. Kapoor, and D. Lischinski. Personalizationof image enhancement. In CVPR, pages 1799–1806, 2010.

[13] G. Klein and D. W. Murray. Simulating low-cost cameras foraugmented reality compositing. IEEE Trans. Vis. Comput.Graph., 16(3):369–380, 2010.

[14] M. Knecht, C. Traxler, W. Purgathofer, and M. Wimmer.Adaptive camera-based color mapping for mixed-reality ap-plications. In ISMAR, pages 165–168, 2011.

[15] Y. Li, T. Ju, and S.-M. Hu. Instant propagation of sparse editson images and videos. Comput. Graph. Forum, 29(7):2049–2054, 2010.

[16] H. T. Lin, S. J. Kim, S. Susstrunk, and M. S. Brown. Revisit-ing radiometric calibration for color computer vision. ICCV,2011.

[17] D. Lischinski, Z. Farbman, M. Uyttendaele, and R. Szeliski.Interactive local adjustment of tonal values. ACM Trans.Graph., 25(3):646–653, 2006.

[18] A. Neumann and L. Neumann. Color style transfer tech-niques using hue, lightness and saturation histogram match-ing. In Computational Aesthetics, pages 111–122, 2005.

[19] F. Pitie and A. C. Kokaram. The linear monge-kantorovitchlinear colour mapping for example-based colour transfer. Vi-sual Media Production, 2007.

[20] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley. Colortransfer between images. IEEE Comput. Graph. Appl.,21(5):34–41, 2001.

[21] H. Siddiqui and C. A. Bouman. Hierarchical color correctionfor camera cell phone images. IEEE Trans. on Image Proc.,17(11):2138–2155, 2008.

[22] Y.-W. Tai, J. Jia, and C.-K. Tang. Local color transfer viaprobabilistic segmentation by expectation-maximization. InCVPR (1), pages 747–754, 2005.

[23] B. Wang, Y. Yu, T.-T. Wong, C. Chen, and Y.-Q. Xu. Data-driven image color theme enhancement. ACM Trans. Graph.,29(6):146, 2010.

[24] B. Wang, Y. Yu, and Y.-Q. Xu. Example-based image colorand tone style enhancement. ACM Trans. Graph., 30(4):64,2011.

[25] X. Xiao and L. Ma. Color transfer in correlated color space.In VRCIA, pages 305–309, 2006.

[26] X. Xiao and L. Ma. Gradient-preserving color transfer. Com-put. Graph. Forum, 28(7):1879–1886, 2009.

[27] S. Yang, Y.-A. Kim, C. Kang, and B.-U. Lee. Color compen-sation using nonlinear luminance-rgb component curve of acamera. In ISVC (2), pages 617–626, 2011.

4328