View
3
Download
0
Category
Preview:
Citation preview
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont: Font Recognition and Similarity Based on Deep LearningHailin Jin | Principal Scientist | Adobe Research
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Problem definition
Font recognition Font similarity Why does it matter? Use fonts effectively
Design inspiration
Text/image editing
3
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Challenges
Extremely large scale recognition problem Over 100,000 known fonts
Beyond object recognition Recognize subtle design styles
Extremely difficult to collect real-world training data Have to use synthetic training data
Mismatch between training and test data
4
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Our approach – DeepFont
Deep Learning – convolutional neural networks Effective at large-scale
Effective at fine-grained recognition
No need to do OCR End-to-end training and data-driven
5
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont system overview
Localization Network
Recognition Network
6
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
GPUCPU
DeepFont training
Fonts Font maps
Offline rendering Online
rendering
Words
Images
Data augmentation
CNN training
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Data augmentation
Noise Blur Perspective deformation Shading Variable character spacing Variable aspect ratio
9
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Effects of data augmentation
10
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Beyond data augmentation
Cannot enumerate all possible degradation Background, font decoration
May introduce degradation bias in training Leverage abundant unlabeled real-world images
11
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Network decomposition for domain adaptation
Decompose the network into two parts Train the two parts separately
12
Unsupervised cross-domainvariable layers
Supervised domain-specific
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Network decomposition for domain adaptation
Stacked convolutional auto-encoder Synthetic data and unlabeled real data
Unsupervised training Fix the subnetwork Train the rest in a supervised way Labeled synthetic data
13
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Qualitative evaluation
4383 real-world images collected from font forums
Vary the number of unsupervisedly trained layers
14
Model Augmentation Decomposition Top-1 accuracy Top-5 accuracy
LFE (CVPR’04) Y N/A 42.6% 60.3%
DeepFont N N 42.5% 49.2%
DeepFont Y N 66.7% 79.2%
DeepFont Y Y 71.4% 81.8%
0 1 2 3 4 5
Train 91.5% 90.1% 88.8% 87.5% 84.8% 82.1%
Test 79.3% 79.7% 81.8% 81.0% 77.5% 74.0%
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Text localization
Localization model
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Text localization results
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont for font similarity
Similarity Network
LushScript.ttf
Query Sample
Similar fonts
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Font similarity
18
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Demo in Photoshop
19
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20
Mobile
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Summary
DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization
Photoshop Mobile Dataset available at http://www.atlaswang.com/deepfont.html References Chen, Yang, Jin, Brandt, Shechtman, Agarwala, and Han. Large-scale Visual Font Recognition. CVPR 2014. Wang, Yang, Jin, Brandt, Shechtman, Agarwala, and Huang. DeepFont: Identify Your Font from An Image. ACM
Multimedia 2015.
21
Recommended