26
Analytics of Visual Data Seeing is Believing Lou Kratz [email protected]

Seeing is Believing · Seeing is Believing Lou Kratz [email protected]. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

  • Upload
    others

  • View
    11

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Analytics of Visual DataSeeing is Believing

Lou Kratz [email protected]

Page 2: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects
Page 3: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

People Communicate Visually

Images are an efficient and effective way to communicate.

Brain recognizes objects in 100ms

Increasingly popular on social networks

De facto choice for analytics

Page 4: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Keyword Analytics

count(“eagles”)

Data

Aggregate

Analyze

Visual Analytics

?

?

The image is the conversation.

Page 5: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

• Philadelphia Startup • Marketing suite for visual web • Visual analytics for social networks

[email protected] • Software Engineer • Machine Learning,

Computer Vision, Tech Transfer

Myself

Page 6: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Data

Aggregate

Visual Analytics

• Accurate • Efficient • Scalable

Image Matching?

Page 7: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Image MatchingAre these the same image?

Page 8: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Image MatchingAre these the same image?

Difference

• Compression • Resolution • Processing Time*

Page 9: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Efficient Matching• Image Fingerprinting

• Small numeric representation • Fast to compute • Represent as much

information as possible

Key Idea: Identical images have identical fingerprints

ImageBinary Encoding

Discrete Cosine Transform

Fingerprint

64 bit integer

Page 10: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Relaxing ConstraintBits can differ due to noise or compression

BIT_COUNT( XOR(fp1, fp2) ) <= 2

64 bits

64 bits

fp2

fp1

Threshold on hamming distance

Page 11: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Image Matching

Image

Fingerprint

Database

Fingerprint File

32432521355 Eagles-Logo.jpg

745907134 NFL-Philly.jpg

Search

Matching is now a search problem

Page 12: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Mysql Query Time

Page 13: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Option 2: NoSQL64 bits

l(b, r) =rX

k=0

✓b

k

1

For 64 bits and r=2, 2081 reads!

• b: number of bits in fingerprint (64) • r: distance threshold (2)

Fingerprint

Number of reads

Page 14: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

NoSql• 2081 reads for an image lookup

• substantially reduced with optimization

• Fast (Reads~10ms) • Scalable: Hosted solutions are distributed • Number of images only minutely affects search speed

Page 15: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Our System• Deployed in August of 2013 • Webservice using Amazon Elasticbeanstalk • Currently:

• 126 million URLS • 116 million Files • 86 million unique images • 600 images matched per second

Page 16: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Data

Aggregate

Analyze

Visual Analytics

Image Matching

?

Identical Images

Different Images

Page 17: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Image Fetures• Dominant Hue (color) • Face Detection • Texture • Many more!

face

hue:

Page 18: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Color

vs.

Images that are reddish-orange get roughly TWICE AS MANY repins than images that are blue.

REDDISH-ORANGE IMAGEBLUEISH IMAGE

Page 19: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Number of ColorsNUMBER OF COLORS

vs.

MULTIPLE DOMINANT COLORS SINGLE DOMINANT COLOR

Images with multiple dominant colors have 3.25 TIMES MORE repins than single dominant color images.

Page 20: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

LightnessLIGHTNESS

vs.

Images with medium lightness are repinned 20 TIMES MORE than very dark images.

MEDIUM LIGHTNESSVERY DARK

Page 21: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

SaturationSATURATION

vs.

Images that are 50% saturated have 10 TIMES MORE repins than very desaturated images.

50% SATURATION VERY DESATURATED

Page 22: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Aspect Ratio

vs.

2:3 ASPECT RATIOVERY TALL

Vertical images with an aspect ratio between 2:3 and 4:5 get 60% MORE repins than very tall images.

Page 23: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

BackgroundBACKGROUND

vs.

LESS THAN 10% BACKGROUND MORE THAN 90% BACKGROUND

Images with <10% background receive 2–4 TIMES MORE repins than images with >40% background.

Page 24: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

Texture

vs.

“SMOOTH” IMAGE“ROUGH” IMAGE

Images with a smooth texture are repinned 17 TIMES MORE than images with a rough texture.

Page 25: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

FacesFACES

vs.

DOESN’T HAVE A FACE HAS A FACE

Brand images without faces receive 23% MORE repins than images with faces.

Page 26: Seeing is Believing · Seeing is Believing Lou Kratz lou@curalate.com. People Communicate Visually Images are an e"cient and e#ective way to communicate. Brain recognizes objects

• Visual communications are important • Visual Analytics are awesome • Each step requires novel techniques - new paradigm for analytics

Conclusions

[email protected]