Color Compatibility From Large Datasets Peter O’Donovan University of Toronto Aseem Agarwala Adobe...

Color Compatibility From Large Datasets

Peter O’DonovanUniversity of Toronto

Aseem AgarwalaAdobe Systems, Inc.

Aaron HertzmannUniversity of Toronto

Choosing colors is hard for many people

Choosing colors is hard for many people

?

How do designers choose colors?

Picasso

How do designers choose colors?

You the Designer

How do designers choose colors?

Krause [2002]

How do designers choose colors?

Goethe [1810]

Complementary Color Theory: colors opposite on the color wheel are compatible

Hue Templates: relative orientations producing compatible colors

Complementary Monochromatic Analogous Triad

Photo and Video Quality Evaluation:Focusing on the SubjectLuo and Tang 2008

Aesthetic Visual Quality Assessment of PaintingsLi and Chen 2009

Color Harmonization for VideosSawant and Mitra 2008

Color Harmonization Cohen-Or et al. 2006

Adobe Kuler

527,935 themes

Ratings: 1-5 stars

Adobe Kuler

527,935 themes

Ratings: 1-5 stars

Adobe Kuler

527,935 themes

Ratings: 1-5 stars

Adobe Kuler

527,935 themes

Ratings: 1-5 stars

COLOURLovers

1,672,657 themes

Views and “Likes”

COLOURLovers

1,672,657 themes

Views and “Likes”

Goals

1. AnalysisTest hypotheses and compatibility models

2. Learn ModelsPredict mean ratings for themes

3. New ApplicationsDevelop new tools for choosing colors

Goals

1. AnalysisTest hypotheses and compatibility models

2. Learn ModelsPredict mean ratings for themes

3. New ApplicationsDevelop new tools for choosing colors

104,426 themes Ratings: 1-5 stars

383,938 themes # Views and “Likes”

Kuler Dataset COLOURLovers Dataset

Mechanical Turk dataset

10,743 themes from Kuler40 ratings per theme1,301 total participants

Overall preference for warmer hues and cyans

Histogram of hue usage

Hue

% of all themes

Mean rating for themes containing a hue

Overall preference for warmer hues and cyansHue

Mean Rating

Histogram of hue adjacency (Kuler)

Histogram of hue adjacency (Kuler)

is more likely than

Histogram of hue adjacency (Kuler)

Significant structure

Histogram of hue adjacency (Kuler)

Significant structureWarm hues pair well with each other

Histogram of hue adjacency (Kuler)

Significant structureWarm hues pair well with each otherGreens and purples more compatible with themselves

Histogram of hue adjacency (Kuler)

Hue Template Analysis

Hue Templates: relative orientations producing compatible colors

Templates are rotationally invariant

Hue Templates: relative orientations producing compatible colors

Different templates equally compatible

Complementary Monochromatic Analogous Triad

Hue Templates: relative orientations producing compatible colors

Diagonal lines are hue templates (Kuler interface bias)

Hue adjacency in a theme (Kuler)

Complementary template

Hue adjacency in a theme (Kuler)

Diagonal lines are hue templates (Kuler interface bias)

Hue adjacency in a theme (Kuler)

Complementary:

Complementary: Data:

Hue adjacency in a theme (Kuler)

In template theory, diagonals should be uniform

Hue adjacency in a theme (Kuler)

In template theory, diagonals should be uniformLarge dark bands indicates no rotational invariance

Hue adjacency in a theme (Kuler)

Kuler CL

Hue adjacency in a theme

COLOURLovers’ has less interface biasTemplates are not present

Distance to template

Rating

Distance to template

Themes near a template score worse

Rating

Themes near a template score worse - “Newbie” factor - “Too simple” factor

Distance to template

Rating

MTurk has no interface bias: much flatter

Distance to template

Rating

Template Conclusions

1) Templates do not model color preferences2) Themes near a template do not score better

than those farther away3) Not all templates are equally popular

- Simple templates preferred (see paper)

Hue Entropy: entropy of hues along the hue wheel

Hue Entropy: entropy of hues along the hue wheel

Low Entropy

Few Distinct Colors

Hue Entropy: entropy of hues along the hue wheel

Low Entropy High Entropy

Few Distinct Colors Many Distinct Colors

Hue Entropy: entropy of hues along the hue wheel

Hue Entropy

Rating

Hue Entropy: entropy of hues along the hue wheel

Hue Entropy

Rating

Hue Entropy

Rating

Hue Entropy: entropy of hues along the hue wheel

Hue Entropy: entropy of hues along the hue wheel

Hue Entropy

Rating

Main Analysis Results

1. Overall preference for warmer hues and cyans

Main Analysis Results

1. Overall preference for warmer hues and cyans

2. Strong preferences for certain adjacent colors

Main Analysis Results

1. Overall preference for warmer hues and cyans

2. Strong preferences for certain adjacent colors

3. Hue templates a poor model for compatibility

Main Analysis Results

1. Overall preference for warmer hues and cyans

2. Strong preferences for certain adjacent colors

3. Hue templates a poor model for compatibility

4. People prefer simpler themes (but not too simple)

Main Analysis Results

1. Overall preference for warmer hues and cyans

2. Strong preferences for certain adjacent colors

3. Hue templates a poor model for compatibility

4. People prefer simpler themes (but not too simple)

See paper for other tests

Goals

1. AnalysisTest hypotheses and compatibility models

2. Learn ModelsPredict mean ratings for themes

3. New ApplicationsDevelop new tools for choosing colors

3.63

3.63

𝑓 (𝒙 )=𝑦

Mean rating over all users

3.63

𝑓 (𝒙 )=𝑦

𝑓 (𝒙 )=𝑦

Features (326 total)- Colors, sorted colors, differences, min/max,

max-in, mean/std dev, PCA features, hue probability, hue entropy

- RGB, HSV, CIELab, Kuler color wheel- “Kitchen Sink”

3.63

𝑓 (𝒙 )=𝑦

Models- Constant baseline: mean of training targets - SVM-R, KNN- Lasso

- Linear regression model with L1 norm on weights- Solutions have many zero weights: feature

selection

3.63

Dataset MAE

Constant Baseline

KNN SVM-R

Lasso Lasso over Baseline

Kuler 0.572 0.533 0.531 0.521 9%

COLORLovers

0.703 0.674 0.650 0.644 8%

MTurk 0.267 0.205 0.182 0.179 33%

Dataset MAE

Constant Baseline

KNN SVM-R

Lasso Lasso over Baseline

Kuler 0.572 0.533 0.531 0.521 9%

COLORLovers

0.703 0.674 0.650 0.644 8%

MTurk 0.267 0.205 0.182 0.179 33%

Dataset MAE

Constant Baseline

KNN SVM-R

Lasso Lasso over Baseline

Kuler 0.572 0.533 0.531 0.521 9%

COLORLovers

0.703 0.674 0.650 0.644 8%

MTurk 0.267 0.205 0.182 0.179 33%

Many more ratings per theme in MTurk

Dataset MAE

Constant Baseline

KNN SVM-R

Lasso Lasso over Baseline

Kuler 0.572 0.533 0.531 0.521 9%

COLORLovers

0.703 0.674 0.650 0.644 8%

MTurk 0.267 0.205 0.182 0.179 33%

MTurk has an average std dev of 0.33Kuler has an average std dev of 0.72

MTurk Test Set

Human Rating

Lasso Rating

High-rated

𝑦=3.90 , f (𝐱 )=3.41𝑦=3.79 , f (𝐱 )=3.50

High-rated

Low-rated

𝑦=3.90 , f (𝐱 )=3.41𝑦=3.79 , f (𝐱 )=3.50

𝑦=1.71 , f ( 𝐱 )=2.27𝑦=1.78 , f (𝐱 )=2.25

High-rated

Low-rated

High prediction error

𝑦=3.90 , f (𝐱 )=3.41𝑦=3.79 , f (𝐱 )=3.50

𝑦=1.71 , f ( 𝐱 )=2.27𝑦=1.78 , f (𝐱 )=2.25

𝑦=2.74 , f ( 𝐱 )=1.78𝑦=2.22 , f ( 𝐱 )=3.16

Model Analysis

Important Lasso Features

Positive: high lightness mean & max, mean hue probability

Important Lasso Features

Positive: high lightness mean & max, mean hue probability

Negative: high lightness variance, min hue probability

Goals

1. AnalysisTest hypotheses and compatibility models

2. Learn ModelsPredict mean ratings for theme

3. New ApplicationsDevelop new tools for choosing colors

1. Improve a Theme

Maximize regression score

Stay within a distance of original (L2 in CIELab)

Select order which maximizes score

Optimize colors with CMA [Hansen 1995]

Original Best Order Color and Order

f (𝐱 )=2.92 f (𝐱 )=3.04 f (𝐱 )=3.35

f (𝐱 )=3.00 f (𝐱 )=3.11 f (𝐱 )=3.37

f (𝐱 )=3.50 f (𝐱 )=3.50 f (𝐱 )=3.70

Original Best Order Color and Order

f (𝐱 )=2.92, 𝑦=3.04f (𝐱 )=3.04 , 𝑦=2.99f (𝐱 )=3.35 , 𝑦=3.40

f (𝐱 )=3.00 , 𝑦=2.96f (𝐱 )=3.11 , 𝑦=3.21

f (𝐱 )=3.50 , 𝑦=3.72f (𝐱 )=3.50 , 𝑦=3.69f (𝐱 )=3.70 , 𝑦=3.82

MTurk A/B test with original and optimized themes

Order and Color

2. Choose 5 colors that best ‘represent’ an image

One approach: k-means clustering

One approach: k-means clustering

This ignores color compatibility

Optimize 5 colors that1) Match the image well2) Maximize regression

score

Optimize 5 colors that1) Match the image well2) Maximize regression

scoreSee paper for details

With Compatibility Model

W/O Compatibility Model

MTurk A/B testwith and withoutcompatibility model

3. Given 4 colors for foreground, suggest background

Given 4 colors, choose 5th color to maximize score

Want contrast with existing colors

Find next best color, away from previous choices

, , …

Model Suggestions

Random Suggestions

MTurk tests selecting ‘Worst’ and ‘Best’

4 model & 4 random

Model Limitations & Future Work

Hard to interpret

Features

Weights

Model has very few abstract colors, only 1-D spatial layout

VS

Model does not understand how colors are used

VS

Conclusions

Color preferences are subjective, but analysis reveals many overall trends

Simple linear models can represent compatibility fairly well

Models can be useful for color selection tasks

Our datasets and learned models are available online