LightroomCV ECE-499 Group #4 Owen Hubner, Noah Guengerich, and Bryan Eriksson Introducon This project is designed to aid photographers—both amateur and professional—in sorng and cataloguing their photo libraries. It leverages machine learning and computer vision to automacally generate text capons for photos, saving photographers me and effort while sll enabling them to organize their collecons of photos with keywords. It is designed as an extension for Adobe Lightroom, a popular program for managing photo libraries. Design Objecves & Methodology The design of LightroomCV was divided into three primary objecves: Develop the machine learning & computer vision system to capon the photos ResNet-101 encoder paired to an LSTM decoder using the Show, Aend, and Tell memory mechanism [1] Design a GUI ed to Adobe Lightroom for user interacon Let user select which photos to capon and how those capons are applied Write back-end code to facilitate communicaon between these systems Use local network sockets to send image & capon data between Lightroom & s the CV system Results Accurate caponing: Using LightroomCV to organize photos by keyword: Conclusion & Recommendaons The final version of LightroomCV works as a plugin to Adobe Lightroom and allows users to select photos, generate capons, and apply them in a specified way. The CV system produces accurate capons for a majority of tested photos, however it was shown to generate nonsensical capons when the system encounters features for which it was not trained. Expanding the variety of the training set would greatly improve results. References [1] Kelvin Xu et al., “Show, Aend and Tell: Neural Image Capon Generaon with Visual Aenon”, 1502.03044v3 [cs.LG] 19 Apr 2016. [Online]. Available at: hps://arxiv.org/pdf/1502.03044.pdf. [Accessed June 17, 2021] Photo (leﬅ) in Adobe Lightroom automacally assigned the capon (right) “a closeup of a bear in the water” Series of photos in Adobe Lightroom with auto- generated capons containing “boat”

LightroomCV - ece.uvic.ca

Download PDF Report

Upload
others
View
2
Download
0

Embed Size (px)

Citation preview

LightroomCV ECE-499 Group #4

Owen Hubner, Noah Guengerich, and Bryan Eriksson

Introduction

This project is designed to aid photographers—both amateur and professional—in sorting and cataloguing their photo libraries. It leverages machine learning and computer vision to automatically generate text captions for photos, saving photographers time and effort while still enabling them to organize their collections of photos with keywords. It is designed as an extension for Adobe Lightroom, a popular program for managing photo libraries.

Design Objectives & Methodology

The design of LightroomCV was divided into three primary objectives:

Develop the machine learning & computer vision system to caption the photos

ResNet-101 encoder paired to an LSTM decoder using the Show, Attend, and Tell memory mechanism [1]

Design a GUI tied to Adobe Lightroom for user interaction

Let user select which photos to caption and how those captions are applied

Write back-end code to facilitate communication between these systems

Use local network sockets to send image & caption data between Lightroom & s the CV system

Results

Accurate captioning: Using LightroomCV to organize photos by keyword:

Conclusion & Recommendations

The final version of LightroomCV works as a plugin to Adobe Lightroom and allows users to select photos, generate captions, and apply them in a specified way. The CV system produces accurate captions for a majority of tested photos, however it was shown to generate nonsensical captions when the system encounters features for which it was not trained. Expanding the variety of the training set would greatly improve results.

References

[1] Kelvin Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, 1502.03044v3 [cs.LG] 19 Apr 2016. [Online]. Available at: https://arxiv.org/pdf/1502.03044.pdf. [Accessed June 17, 2021]

Photo (left) in Adobe Lightroom automatically

assigned the caption (right) “a closeup of a bear in

the water”

Series of photos in Adobe Lightroom with auto-

generated captions containing “boat”