New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

DeepFont: Font Recognition and Similarity Based on Deep LearningHailin Jin | Principal Scientist | Adobe Research



Problem definition

Font recognition Font similarity Why does it matter? Use fonts effectively

Design inspiration

Text/image editing

3


Challenges

Extremely large scale recognition problem Over 100,000 known fonts

Beyond object recognition Recognize subtle design styles

Extremely difficult to collect real-world training data Have to use synthetic training data

Mismatch between training and test data

4


Our approach – DeepFont

Deep Learning – convolutional neural networks Effective at large-scale

Effective at fine-grained recognition

No need to do OCR End-to-end training and data-driven

5


DeepFont system overview

Localization Network

Recognition Network

6


GPUCPU

DeepFont training

Fonts Font maps

Offline rendering Online

rendering

Words

Images

Data augmentation

CNN training


Data augmentation

Noise Blur Perspective deformation Shading Variable character spacing Variable aspect ratio

9


Effects of data augmentation

10


Beyond data augmentation

Cannot enumerate all possible degradation Background, font decoration

May introduce degradation bias in training Leverage abundant unlabeled real-world images

11


Network decomposition for domain adaptation

Decompose the network into two parts Train the two parts separately

12

Unsupervised cross-domainvariable layers

Supervised domain-specific


Network decomposition for domain adaptation

Stacked convolutional auto-encoder Synthetic data and unlabeled real data

Unsupervised training Fix the subnetwork Train the rest in a supervised way Labeled synthetic data

13


Qualitative evaluation

4383 real-world images collected from font forums

Vary the number of unsupervisedly trained layers

14

Model Augmentation Decomposition Top-1 accuracy Top-5 accuracy

LFE (CVPR’04) Y N/A 42.6% 60.3%

DeepFont N N 42.5% 49.2%

DeepFont Y N 66.7% 79.2%

DeepFont Y Y 71.4% 81.8%

0 1 2 3 4 5

Train 91.5% 90.1% 88.8% 87.5% 84.8% 82.1%

Test 79.3% 79.7% 81.8% 81.0% 77.5% 74.0%


Text localization

Localization model


Text localization results


DeepFont for font similarity

Similarity Network

LushScript.ttf

Query Sample

Similar fonts


Font similarity

18


Demo in Photoshop

19

© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20

Mobile


Summary

DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization

Photoshop Mobile Dataset available at http://www.atlaswang.com/deepfont.html References Chen, Yang, Jin, Brandt, Shechtman, Agarwala, and Han. Large-scale Visual Font Recognition. CVPR 2014. Wang, Yang, Jin, Brandt, Shechtman, Agarwala, and Huang. DeepFont: Identify Your Font from An Image. ACM

Multimedia 2015.

21

http://www.atlaswang.com/deepfont.html

Documents

New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization