instantcoke.files.wordpress.com€¦ · Web viewImage Recognition Algorithm for Instant Inventory of Coca-Cola Freestyle Cartridges . ECE4012 Senior Design Project. Coca-Cola Freestyle

Image Recognition Algorithm for Instant Inventory of Coca-Cola Freestyle Cartridges

ECE4012 Senior Design Project

Coca-Cola Freestyle Instant InventoryProject Advisor: Linda Milor

Project Sponsors: Scott Harrison & Thomas Stubbs

Peter Lyons Forrest MercierCurtis NorrisJingJing Pan

Quentin SamuelsBen Tran

SubmittedDecember 14, 2017

Table of Contents

Executive Summary 3

1. Introduction 4

1.1 Objective 41.2 Motivation 41.3 Background 5

2. Project Description and Goals 6

3. Technical Specification 7

4. Design Approach and Details 11

4.1 Design Approach 114.2 Codes and Standards 124.3 Constraints, Alternatives, and Tradeoffs 13

5. Schedule, Tasks, and Milestones 14

6. Final Project Demonstration 14

7. Marketing and Cost Analysis 16

7.1 Marketing Analysis 167.2 Cost Analysis 16

8. Conclusion 17

9. Leadership Roles 18

10. References 20

Appendix 21

Coca-Cola Freestyle Instant Inventory 2

Executive Summary

The basic idea of this project is as follows: An algorithm will be developed using computer

vision and deep learning techniques in order to train a model to recognize various brand cartridges for

the Coca-Cola Freestyle drink machines. This model will then be integrated into an application for

consumer use in which they will take a photo of a rack or box of their current inventory. This picture

will then become the input for the model and the model will recognize each cartridge and count how

many of each brand of cartridge is in the photo. This inventory will then be displayed on screen for the

user to see. Once the inventory is complete, the user will then have the option to order more cartridges

based on the inventory that is displayed. This project should be done because of the current struggles

that Freestyle machine owners have in keeping a good inventory of their current products, leading to

over-ordering and spending too much money for cartridges at a time that may hurt them as a business

in the long run. The cost of this project is estimated to be up to $1000. The expected outcome of this

project is a trained model with a robust underlying algorithm that will recognize and document

cartridges based on their individual colors and logos. This model will then be used for an aesthetically

pleasing, user facing application to potentially deploy to Freestyle machine owners everywhere.


Recognition Algorithm for Instant Inventory of Coca-Cola Freestyle Cartridges

1. Introduction

The Coke Instant Inventory team will be creating an image recognition algorithm for the Coca-

Cola freestyle cartridges that are located on storage racks in several restaurant locations. The team is

requesting up to $1000 of funding to develop the recognition algorithm for Coca-Cola Freestyle

Cartridges.

1.1 Objective

Our objective is to build an image recognition system in order to automatically create inventory

lists from photos of Coca-Cola syrup cartridges on the storage rack. The syrup will be filled into Coca-

Cola Freestyle Dispensers to serve soft drinks of a variety of tastes in the restaurants. Currently the

inventory is manually checked by Coca-Cola staffs on a daily basis, which entails a high manual cost.

Our automatic recognition system will make such process much more efficient.

1.2 Motivation

As mentioned above, our sponsor from Coca-Cola expects this automatic inventory system to

save their human labor cost of updating inventory list. There is no similar automatic system inside the

company, so they want us to initiate the project.

1.3 Background


Although there is no similar recognition system for syrup cartridges in use by Coca-Cola,

outside the company there are companies such as IBM [1] and NEC working on a general product

recognition system, which can help retailers to track the numbers and positions of different products on

their shelves for inventory management. Although some research is being conducted in academia [2],

this technology is more actively developed in the industry because it is profitable. Both companies use

traditional object recognition approaches instead of deep learning. While the deep learning approach

should be able to obtain higher accuracy in product recognition like it does in other tasks, it is hard to

stabilize the millions of parameters used in deep learning, and it requires heavy computation and hence

expensive hardware platforms like GPU. On the other hand, traditional approaches, using handcrafted

features such as SIFT and simple classifiers such as a support vector machine (SVM), have the

advantage of fast computation, but the accuracy is lower than that of deep learning as shown in large-

scale image recognition competitions.

Although the details of both companies’ methods are not disclosed, the reason for choosing the

traditional approaches could be that the robustness of deep learning was proved in 2012 [3]. The

commercial systems have been developed at a time when deep learning was not a reliable solution.

2. Project Description and Goals


The fundamental of goal of the Coca-Cola Freestyle Instant Inventory team is to design and test

a mobile application whereby a user can take a picture of a back room inventory rack, where Coca-

Cola Freestyle cartridges are stored, and translate the image into an inventory listing that can be

directly submitted to the central cartridge ordering location. The application utilizes a phone camera,

contains image recognition software, and code translating the image into an inventory listing in .xml

format. Via their phone cameras, users can take a picture of an entire rack of Coca-Cola Freestyle

cartridges and upload the picture to the Instant Inventory application. The software will recognize

individual Coca-Cola Freestyle cartridges based on unique characteristics such as color and product

font/logo. From the image, code formulated within the Instant Inventory application will tally the

number of each type of cartridge and translate raw data into an inventory listing in .xml format. The

features of the applications include:

● Seamless upload of Freestyle rack images

● Sophisticated image recognition scripting via MATLAB and Java

● Conversion of raw image data to an inventory listing in .xml format

● Support for multiple pictures for large inventories

3. Technical Specifications


Figure 1 shows the ingredient map that details the available labels of each freestyle product that will be

identified by the mobile app.

Figure 1. Layout of various Freestyle Products and Labels.

Figure 1. Coca-Cola Freestyle Ingredient Map

There are 2 types of labels that the app will identify: single and double cartridge. The dimensions for

each label are shown in Table 1.

Table 1. Specifications of Coca-Cola Freestyle Labels

Label Specification

Single Cartridge 4 x 1.5 cm

Double Cartridge 4 x 3.5 cm

The different Coca-Cola freestyle products that come as a double cartridge label as well as their

respective colors are listed in Table 2.


Table 2. Color of Double Cartridge Coca-Cola Freestyle Labels

Label Specification

Coca Cola Red and White

Coca Cola Zero Black, Red, and White

Diet Coke Grey, Red and White

Caffeine Free Diet Coke Tan, Red and White

Sprite and Sprite Zero Dark Green, White, and Grey

The different Coca-Cola freestyle products that come as a single cartridge label as well as their

respective colors are listed in Table 3.

Table 3. Color of Single Cartridge Coca-Cola Freestyle Labels

Label Specification


Ingredient Solution CC Lavender

HiC Flavors Purple, Yellow, Red, and Orange

Cherry (Coca-Cola Freestyle) Red

Peach (Coca-Cola Freestyle) Light Orange and Black

Grape (Coca-Cola Freestyle) Dark Purple

Vanilla (Coca-Cola Freestyle) Light Yellow

Orange (Coca-Cola Freestyle) Orange and White

The flavors and their proprietary data matrix numbers are shown below in Table 4.

Table 4. Cartridges and Datamatrix Values

Cartridge DataMatrix Number Cartridge DataMatrix Number

Coke Part 1 603 Hi-C 616

Coke Part 2 604 CC 602

Coke Zero Part 1 629 Cherry 33

Coke Zero Part 2 630 Vanilla 35

Diet Coke Part 1 607 Orange 43

Diet Coke Part 2 608 Grape 41


Sprite 615 Peach 42

The Standards and languages used to develop this application are shown in Table 5.

Table 5. Platform Specifications

Standard Application Language

XML MATLAB/Java

The desired success rate to picture ratio is shown below in Table 6. The picture ratio is represented by

pictures that are successfully recognized by the mobile app and transferred to the XML file.

Table 6. Desired Success Rate

Desired Success Rate Picture Ratio

90% 9 out of every 10 cartridges identified

4. Design Approach and Details

4.1 Design Approach


The design approach we are taking for the implementation of this project is as follows: a Java

application supporting multiple input pictures will be developed. This will take in photos of inventory

racks, and from there the pictures will be analyzed using a MATLAB script that binarizes the image

and identifies all the data matrix codes found in the image. Once the data matrices have been

identified, the MATLAB script will give them to the Java program which will then read each code one

by one and update the inventory as it reads every code. Once the analysis is complete, the Java

program will convert the resulting inventory map into an XML file of all of the products, their serial

numbers, and corresponding quantities counted. This will be saved into an output folder in the

program. From there, the user will have the option to send this list to the vendor so that the vendor can

ship the correct number of cartridges to the user. Figure 2 shows a UML sequence diagram of the

process.


Figure 2. Sequence Diagram for Mobile Application.

4.2 Codes and Standards

Due to the extremely software-centric nature of this project, no industry codes apply to this

project. One standard that applies to this project is the Extensible Markup Language, otherwise

referred to XML [5]. Figure 3 shows an example of the same entry above in XML.

Figure 3. XML Example of Cartridge Entry

4.3 Constraints, Alternatives, and Tradeoffs

One of the biggest discussions involved whether to train the model on the device as more

pictures are taken or to train it on a separate, more powerful device and then simply use the model on

the application. In the end, it was decided to not have the model trained on the device and to upload it

onto the application. This decision was mainly due to the fact that there are many cellphones on the


market that all have different amounts of memory and processing power. A model that is trained on a

device would potentially crash or slow down a user’s phone, causing undue stress on the user.

However, this discussion was made before pivoting our approach to our project, because we found that

the MATLAB approach to pre-processing the image and reading a universal standard yielded much

higher success rates than the image classifier models we were training.

5. Schedule, Tasks, and Milestones

A Gantt Chart is provided in the Appendix. Figure 5 shows the Pert Chart which contains the duration

of each task.


Figure 4. PERT Chart showing project timeline.

6. Final Project Demonstration

The application is designed to be uploaded to a web server (most likely via Amazon Web Services or a

similar provider) where a smartphone application can be developed to interface with it. The formal

demonstration will consist of one person doing the following:

1. The user will take a picture of a box of one or more freestyle cartridges with each cartridge

facing its data matrix code towards the user.

2. With the camera aimed at the inventory rack, the user will ensure that each item they want to be

inventoried can be clearly seen on the display of the device. If there is not enough space in the

storage room to stand far back enough to capture the entire cartridge rack, panorama mode may

be used to sweep capture a row at a time, or batch photo taking will work as well.

3. The pictures can then be input into the Java application, which can be run from the file

CokeInstantInventory.java. It is recommended to use this application in an IDE such as IntelliJ

for easy use.

The specifications will be demonstrated as follows:


■ XML Data Transfer: The inventory data will be formatted into an XML data file through the

application and can be transferred to other devices. Figure 6 demonstrates this process from

start to finish.

Figure 5. Coca Cola Instant Inventory Process.

7. Marketing and Cost Analysis

7.1 Marketing Analysis

Replacing traditional inventory methods of counting inventory, the Coca-Cola Instant Inventory

System will save the company both the time and manpower used in manually tracking backroom

inventory. The system will also reduce the amount of errors produced keeping track of the inventory,

therefore saving the company the potential cost of having incorrect amounts of product in its

dispensers each week.


7.2 Cost Analysis

Because the system will require little to no hardware, the expected cost to develop a prototype

of the Instant-Inventory System is very low. A graphics processing unit (GPU) will potentially be used

in the experiment to minimize the amount of time taken to perform experiments and trials. The

expected cost of one GPU is up to $1000. The main cost of developing the system would be due to the

labor, so with the average yearly salary of an engineer being $72,000, the project taking up to six

months for a team of six engineers, the labor cost would be approximately $216,000.

8. Conclusion

The Coca-Cola Freestyle Instant Inventory team met its objective of building an image

recognition system capable of creating an inventory listing from photos of Coca-Cola syrup cartridges

on a storage rack. Additionally, the image recognition system surpasses the desired success rate of

90% recognition.


Approximately halfway through the development of the image recognition system, the team

agreed on a change in algorithmic strategy, which proved instrumental in the eventual success of the

project. Initially, the image recognition software was trained to differentiate between Coca-Cola

Freestyle cartridges based on unique features such color, font, and logo. This method proved both

difficult to implement, and unsuccessful in recognizing cartridges. With the guidance of Coca-Cola

sponsors, the team pivoted on its approach, agreeing to train the image recognition algorithm to

identify cartridges based on unique data matrix codes, or two-dimensional barcodes consisting of black

and white "cells" or modules arranged in either a square or rectangular pattern. The new method

required fewer images to train, took less time to process, and recognized images with high accuracy.

In hindsight, had the team implemented the data-matrix strategy earlier, there would have been more

time to focus on other aspects of the project.

As of the writing of this proposal, we have conducted the appropriate research on the approach

implementation of our algorithm and the following application. A concrete implementation is the next

step. The Coca-Cola Freestyle groups plans to meet with its Coca-Cola sponsors to discuss a potential

small-scale rollout of the Instant Inventory application. Coca-Cola will then make a decision as to how

the application will be marketed and introduced to customers.

9. Leadership Roles


Application Architect: (Peter Lyons) The application architect will design, build, and test an application

whereby a user can seamlessly take a picture of a rack of Coca-Cola Freestyle cartridges and translate the

image into an inventory listing (.xml file) that can be directly submitted to an email address. Once the

application is built, the application architect is to monitor the application, understand all interactions

between components on a given level of abstraction, and define limitations and requirements for source

code.

Algorithm Scientist: (Forrest Mercier/JingJing Pan) The algorithm scientist will research and design

image recognition algorithms, or sequences of instructions that perform different types of tasks. The

algorithm scientist will devote time to determining methods in which less sophisticated, faster-running

algorithms can be substituted for more complicated algorithms.

Data Processing Analyst: (Ben Tran) The data processing analyst will interpret large quantities of

complex image data and identify particular trends or patterns within the data using statistical techniques.

The data processing analyst will provide ongoing reports as to new findings or suggested strategies.

Documentation: (Curtis Norris) The documentation chair will oversee the research, content planning, and

creation and maintenance of project documentation. The documentation chair will also ensure all

documentation satisfies technical specifications, and is completed and submitted in a timely manner.

External Relations: (Quentin Samuels) External relations will maintain consistent, professional, and

punctual communication with both project advisors, external project sponsors, and teaching assistants

(TAs). External relations will inform team members of the details of any project-related correspondence.


Project Coordinator: (Forrest Mercier) The project coordinator will gather and dispense needed

information and updates and coordinate the work throughout the project life cycle. The project

coordinator will organize team meetings, develop project strategies, monitor project progress, and oversee

the successful completion of the project.

Webmaster: (Quentin Samuels) The webmaster will create and maintain electronic tools/media used for

the project, including Google Drive, Google Docs, Google Slides, and T-Square. The webmaster will also

monitor the content and organization of any electronic media used.

10. References

[1] M. Marder, "Using image analytics to monitor retail store shelves," IBM Journal of Research and Development, vol. 59, no. 2/3, pp. 3:1 - 3:11, 2015.[2] M. Floerkemeier, "Recognizing Products: A Per-exemplar Multi-label Image Classification Approach," in European Conference on Computer Vision, 2014.[3] A. Krizhevsky, I. Sutskever and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in neural information processing systems, 2012.[4] W3, Extensible Markup Language, [Online] Available: https://www.w3.org/TR/REC-xml/ [Accessed April 14 2017]


Appendix

Gantt Chart


Documents

instantcoke.files.wordpress.com€¦ · Web viewImage Recognition Algorithm for Instant Inventory of Coca-Cola Freestyle Cartridges . ECE4012 Senior Design Project. Coca-Cola Freestyle