MOBILE COLOR BARCODE STREAMING FRAMEWORK

by

JEREMY YANIK

AdvisorHALA ELAARAG

A senior research proposal submitted in partial fulfillment of the requirements for the degree of Bachelor of Science

in the Department of Mathematics and Computer Sciencein the College of Arts and Science

at Stetson UniversityDeLand, Florida

Fall Term2012

TABLE OF CONTENTS

TABLE OF CONTENTS....................................................................................................ii

ABSTRACT........................................................................................................................1

1. INTRODUCTION.......................................................................................................2

2. Related Work...............................................................................................................4

3. Mobile Color Barcode Streaming Framework.............................................................6

3.1 MoCoBar Encoding Process..................................................................................7

Barcode Specification..............................................................................................8

Blur Prevention......................................................................................................10

Frame Display........................................................................................................12

3.2 MoCoBar Decoding Process................................................................................13

Blur and Motion Detection.....................................................................................14

Corner Detection....................................................................................................15

Code Extraction......................................................................................................16

4. Conclusion..................................................................................................................17

REFERENCES..................................................................................................................19

ii

ABSTRACT

In this paper we present the Mobile Color Barcode (MoCoBar) Streaming

Framework, a foundation for visible light communications for mobile devices. The

framework encodes information into a high-density 2D barcode specifically designed to

be decoded easily by mobile phones with cameras. Various image processing techniques

are used in both the encoding and decoding processes in order to achieve higher transfer

rates with fewer decoding errors. The framework is designed to decode barcodes in real

time to allow for streaming up to 15 frames per second. The framework was developed

and tested on the Android operating system, allowing for widespread adaptation.

1

1. INTRODUCTION

Today, consumers have readily available smart phones that are capable of reading

many different barcode formats with ease. With the advent of open formats like QR

Code[1] and Data Matrix[4], the use of barcodes is becoming an increasingly popular

medium for one-way, short range communications. This is achieved by storing bit

patterns in a specified format and decoding that format with an optical device, like a

camera. The amount of data stored in a single barcode restricts their storage capabilities

to short strings such as URLs or contact information [1]. To increase the storage density,

all sorts of different formats have been proposed, typically involving the addition of

colors and shapes. However, other implementations take this much further.

One such implementation is Color Barcode Streaming for Smartphone Systems

(COBRA) [2]. Communication via COBRA requires two devices, a sender and a

receiver, which typically consist of two off-the-shelf smartphones. The sending device

relays a message by displaying a stream of barcode images while the receiving device

decodes the images to reconstruct the message.

In this paper we propose a new short range communication scheme that is based

on mobile phones barcode streaming. Our technique is similar to COBRA, but applies

different barcode formats, image processing algorithms and measures for error prevention

and correction. Our goal is to form a point of comparison for other projects, improve data

transfer rates and reduce transmission errors. Specifically our framework will contribute

the following:

2

1. We develop a barcode specification to maximize data density while maintaining

readability. The framework also utilizes open standard barcode specifications such

as QR and DataMatrix. A point of comparison using a specification that has

already been widely adopted is important to consider while evaluating our results.

2. We examine color usage and error correction schemes and study how these

changes improve data transfer rates and and reduce the frequency of errors; two

important performance metrics.

3. We implement widely adopted corner detection algorithms in an attempt to

achieve more accurate or faster corner detection. Real-time corner detection is

achieved by utilizing the last known location for corners as a starting point for

each barcode frame. This methodology is referred to as “fast corner detection.”

4. We apply algorithms for motion detection and blur correction. Most smartphones

have accurate accelerometer sensors that can be used to predict movement. If we

know that the device is moving, we may apply such blur correction to improve the

readability of an image before we attempt to decode it. We use accelerometer data

to estimate the potential new location of corners for fast corner detection.

5. We evaluate the reliability of low-quality cameras and develop the functionality

of a two-way communication scheme

The rest of this paper is organized as follows: in Section 2 we cover related works

in regard to Visible Light Communication. Section 3 presents an overview of MoCoBar’s

framework followed by details of the encoding and decoding processes on mobile

devices. In Section 4, we provide a conclusion of the paper.

3

2. Related Work

Visible Light Communication (VLC) has become a topic of importance as it is

poised to compete with existing technologies for short-range communications such as

radio frequency. In fact, the Visible Light Communication Consortium (VLCC) has been

meeting annually since 2003. However, the focus of VLC in embedded devices is a

recent emergence. Smart phones are becoming more powerful; the costly computations

for image processing are now an easy feat for today’s average smartphone.

The most commonly known and globally adopted barcode is the Universal

Product Code (UPC). They are composed of vertical lines that represent a numeric code

up to 12 digits [3]. These are what we will refer to as 1-Dimensional barcodes; they store

data in a single dimension. Special barcode readers are purchased by retailers to read the

codes. The UPC can be seen in use every day at supermarkets and retailers. However,

their storage limitations prohibit their use for much else than simple numeric tracking.

Additionally, these numeric codes are meaningless without some sort of database

containing information for corresponding product codes.

More recently, 2D barcode standards have emerged. Some of the popular formats

are QR Code [1], Data Matrix [4] and Aztec Code [5]. The introduction of 2D barcodes

allows for raw binary data to be stored. In most cases, data is not restricted by type

(numeric, string, etc.) but by a maximum number of bits. The second dimension also

means that the barcodes have a greater data density. Fig. 1 compares a 12-digit UPC to

the same code represented as a QR Code.

4

Figure 1: UPC-A (left) and QR Code (right) Visual Comparison.

Many standards have sprung to life in order to achieve increased storage capacity

in 2D barcodes. Numerous high-density barcode formats have been proposed, but none

have been as widely adopted as formats such as QR Code. COBRA’s barcode

specification would be considered high-density [2]. Multiple colors are used and the

colors are arranged in a way to reduce blur. However, COBRA takes this a step further by

streaming many barcode images to transmit a dense message [2].

One of the problems with high density barcodes is that they become difficult to

decode, particularly in embedded devices. The most expensive operations in decoding an

image of a high-density barcode are all related to correcting distortions via image

processing [6]. Some of these operations include perspective correction, blur correction

and various color corrections. The goal is to take a camera image and convert it, via

image processing, into a usable form. There are a number of related projects that research

the best way to achieve this.

Aside from COBRA, there is little research on the topic of streamed barcode

images, particularly in the mobile environment. PixNet’s research is an early example of

research in the field of VLC. PixNet is a system for transmitting data over LCD screens

and camera links [6]. The project was not intended to be utilized on mobile platforms, but

5

it does provide incredibly useful insight into the world of VLC. It discuss useful

perspective correction techniques and algorithms that our framework makes use of.

The research by Chu, Yang and Chen for image stabilization on mobile devices is

incredibly important to our framework. In [14], the authors present algorithms for motion

correction of images coming from camera links on smart phones. The algorithms use data

from accelerometer sensors in order to estimate and restore an image that was blurred by

the movement of the device [14]. Similar research for blur detection and estimation is

presented in [15] and [16]. However, the algorithms themselves are not suitable for real

time processing.

A lot of research has been done for corner detection of 2D barcodes. Many works

propose corner detection algorithms that use algorithms such as hough transformations

[11] or various edge detection algorithms [8]. Analyzing these techniques, we have

determined that algorithms such as these are not suitable real time detection. Rather than

using expensive image processing algorithms, researchers were able to build fast corner

detection systems based on pattern detections [2]. However, the implementation of

pattern-based corner detection requires reliable, known visual structures to be present in

images.

3. Mobile Color Barcode Streaming Framework

In this section we present our MoCoBar framework. The entire process takes

place as follows. A header is attached to an arbitrary data message before it is encoded by

the sending device. The encoding process generates multiple frames containing barcode

6

images. The images of each barcode are displayed on the device’s screen in rapid

succession, an animation. A receiving device’s camera is pointed at the screen of the

sending device. Images of the barcodes are processed and decoded on the receiving

device. Once all frames have been received, the data message will be reconstructed.

The most challenging aspect for reading and writing barcodes is held within two

major processes: encoding and decoding. Encoding is the generation and display of a

barcode that depicts a piece of data [6]. Decoding is capturing the encoded image via a

camera link and converting that image back into data [6]. Each process has its own

difficulties and there are a number of ways to overcome them.

In the remainder of section 3, we discuss the details of MoCoBar’s encoding and

decoding processes. Within the encoding process there are three main components,

determining which barcode variant to use, blur prevention and frame display. The image

processing algorithms included in the decoding process is the most expensive

computational component. The low computer power of mobile devices must be taken into

consideration, especially within real-time applications. There are also three main

components for MoCoBar’s decoding process; blur and motion detection, corner

detection and code extraction.

3.1 MoCoBar Encoding Process

Encoding is a more trivial task than decoding. However, encoding data properly is

crucially important to aid the decoding process. Particularly if the decoding process must

be done in real time. As mentioned earlier, one major difficulty presented by encoding is

7

choosing which barcode specification to use. Each barcode has different features when it

comes to readability, data density and error correction. In Section 3.1.1, we compare

three widely adopted barcode standards we consider for this research. After choosing a

specification, the image must be displayed on the screen. There are some steps that can

be taken to treat the image before it is displayed. We present these steps in section 3.1.2.

Finally we discuss the frame speed component in section 3.1.3.

3.1.1 Barcode Specifications

There are a number of popularized barcode specifications that we consider for our

implementation. Codes such as QR [1], DataMatrix [4] and Aztec [5] are either in the

public domain or have un-enforced patents. Each barcode specification has different

levels of data density and error correction. In many cases, even slightly damaged

barcodes can be read. In Table 1, compare the three aforementioned popular barcode

specifications in terms of maximum storage, error correction and block count. In this

table, Block Count refers to the number of data blocks used to display a barcode given

our sample. The Block Count is calculated by counting every “block” or “module” within

each barcode using the lowest error correction possible provided by the specification.

This includes modules for corners, timing and other features.

8

Barcode Max Storage Error Correction Block Count

QR Code 2953 bytes 7 – 30% 8281

DataMatrix 1555 bytes 0 – 25% 8464

Aztec 1914 bytes 0% 3249

Table 1: Popular barcode specification information.

Our framework also supports our high-density MoCo Barcode (Figure 2). We

have designed the barcode with streaming in mind. As such, we have used some of the

key features found in QR and DataMatrix to create a robust code with high data density.

Three corners feature the Corner Module, a colored square that can easily be located and

used to identify orientation. The barcode also features Timing Modules that assist in the

decoding process to help form a sort of grid. A majority of the barcode is the Data Area,

an area filled with colored data modules that represent bits (encompassed by a translucent

line in Figure 2). Each color module in the Data Area represents 2 bits for a RGB color

scheme. In the specification, the colors white, red, blue and green represent the bits 00,

01, 10, and 11 respectively. The specification also supports an alternate color scheme for

3 bits per module using additional colors. However, we explain later why less colors may

result in a higher data transfer rate. The read order of data modules is horizontal, starting

at the leftmost module on each line.

The data density of the MoCo Barcode is limited by the resolution at which it is

being displayed. As such, it makes sense that if we use the smallest possible module size

(1 pixel) we can achieve a relatively high data density. Such a small module size would

not be possible in reality. The quality of smartphone cameras and the level of blur require

9

the size of modules to be greater [2][6]. Module size may vary depending on hardware

specifications such as screen size, screen resolution and camera quality. The design of the

MoCo Barcode is flexible enough to support any size module.

Figure 2: A sample MoCo Barcode and its features.

3.1.2 Blur Prevention

Although the topic is unimportant for processing single barcode, the blur effect is

very important to keep in mind when the intent is to rapidly decode many high density

barcodes. Blur can lead to inaccurate detection and reduced decoding rates [2]. Within

the encoding process, we are not concerned with blur caused by the movement of a

device. Instead, we are more interested in preventing blur caused between data blocks of

different color. This is of particular importance when using color barcodes and high

resolutions [2].

10

The most obvious way to deter the effects of blurring between different data

modules would be to adjust the size of the block. To have a minimal amount of blur, we

use larger block sizes. However, a larger block size would cause data transfers rates to

drop. We must utilize the smallest block size that does not cause too great of a blur effect.

This is a challenge as it is highly dependent on two variables: the distance between the

sender and receiver and the quality of the camera link. Without the use of two-way

communication, it is difficult to evaluate these variables in the real world. With this in

mind, we’ve developed the framework to work with a 5 Megapixel camera at a distance

of 1 to 2 feet.

Figure 3: Example of data represented in its original color order (left)

and with a stride of 3 (right).

As we have stated earlier, blur is caused between data blocks that are different

colors. By reducing the length of the border between colors overall, it is possible to

reduce the potential blur an image may contain once received [2]. We can achieve this by

setting using data strides. This will cause the order in which bits are stored to be altered,

possibly resulting in an image with a shorter border length between different colors.

11

Figure 3 demonstrates how the stride will affect the output. Unlike the technique in [2],

we apply our algorithm in 2 dimensions and do not rely on random strides. When N

represents the number of blocks in the data area, we can compare all possible strides from

2 to N/2, as well as the original. We can quickly compare each stride by counting the

number of neighbors each block has that are of a different color. We call this total the

Blur Potential (BP) of the frame. BP can be calculated using the algorithm below in Fig.

4. The framework uses the stride that contains the lowest BP.

BP = 0

for i = 0 to N - 1 do

if i has a right neighbor AND it is a different color then

BP += 1

end if

if i has a bottom neighbor AND it is a different color then

BP += 1

end if

end for

Figure 4: Blur Potential calculation.

3.1.3 Frame Display

The final step for encoding is the actual display of a barcode. As stated earlier in

Section 3, once barcode images are generated, they will be displayed in an animation on

the sending device’s screen. In a practical scenario, the speed of the animation should be

12

much lower than that of the processing speed of a single frame. There is great variability

in the computing capabilities of different mobile devices, so the exact display frame rate

and decoding rate will vary. Generally, it is agreed that the sending device display at

roughly half the framerate that is possible on the decoding device [2, 6]. In our tests, the

sending device updates its image at 15 frames-per-second (FPS) and the receiving device

attempts to read at 30 FPS.

3.2 MoCoBar Decoding Process

Decoding has greater computational difficulties than encoding. The goal here is

taking a camera-link image and process it to best match the original encoded image. It is

not feasible to form a pixel-to-pixel match [6]. As a result, the goal is to get close enough

to the original image so that we can use the barcode specification to retrieve the encoded

data. This can be achieved by applying a number of image pre-processing techniques.

Before attempting to decode an incoming frame, it is important to determine how blurred

the image is and if blur correction algorithms should be used. If the frame is suitable for

decoding, the actual barcode must be located within the image. This is normally achieved

through corner detection algorithms. Once the barcode has been isolated, the code may be

extracted.

13

3.2.1 Blur and Motion Detection

Blur detection is a rather expensive and computationally heavy process. The

authors in [14] have come up with a very accurate algorithm to detect the blur effect

caused by movement and loss of focus. However, the algorithm does not appear simple

enough to be run in real-time. Rather than run expensive algorithms, we can assume that

if corner detection fails that the image may be blurred.

Blur correction algorithms, such as the barcode restoration algorithm presented in

[14], have made their way to mobile environments. These algorithms use light traces

within images in order to estimate parameters. Even better, accelerometer sensors allow

us to detect motion. If motion can be detected, blur effects can be reduced. The authors in

[14] have also published an algorithm to counteract blur based on the movement of a

device. An additional blur correction and detection algorithm is presented in [16];

however, this algorithm was only tested for single-direction linear movement.

Autofocus is becoming increasingly available on mobile devices. This feature

allows the camera video feed to automatically focus to help prevent blurring. If autofocus

is available, it should be used to alleviate unneeded computational overhead. Moreover, if

autofocus features are not available, it may be feasible to request camera hardware to

perform a focus operation as the ZXing library demonstrates [13].

14

3.2.2 Corner Detection

Corner detection is a vital process for the receiving device. The corner detection

algorithm needs to be accurate and fast. The topic is not a new one, but attempting to

locate corners in real-time has some challenges. The complexity of corner detection

algorithms are exacerbated by the larger image resolutions on mobile devices today.

One of the most popular corner detection algorithms is the Hough

Transformation. As noted by [2] and made obvious in ZXing’s corner detection [13],

algorithms that detect all edge features are not suitable for real-time decoding. Both [2]

and [13] implement unique detection algorithms specific for the barcode specification in

question. This ensures that computing time is not wasted finding every edge in an image.

For example, ZXing [13] begins corner detection from the center of an image and moves

outward searching for specific features based on barcode symbology.

Another algorithm published for real-time detection was based on contour tracing.

Contour tracing can be used in combination with an algorithm by D. M. Tsai et al [8].

The goal of the authors in [8] was to achieve robust detection of corners based on shapes

(as opposed to edges).

Our framework takes corner detection a bit further. Because images are being

decoded at high rates, it can be assumed that the corners in the current frame are very

close to location of the corners in the previous frame. Corner detection begins at the

location of the previous corners instead of the center or corners of the entire image. This

greatly reduces the time needed to locate barcode corners. Of course, this means the first

frame to be decoded will require more computation. In order to ensure speedy decoding,

15

an accurate be inexpensive corner detection algorithm should be used for real-time

processing [2].

3.2.3 Code Extraction

The final step in the decoding process is code extraction or signal detection [6].

Once an image is processed and corners are found, a perspective transformation is

performed. The perspective transformation uses the corners detected previously in order

to resize the code area of a barcode into a rectangle [7][9]. Figure 5 demonstrates the

perspective transformation.

Figure 5: QR Code image from camera (left) and the same image after perspective

transformation (right).

After the image is transformed into a usable grid, the signals must be interpreted

and read. One popular way to read 2D barcodes is achieved via binarization. With a

monochromatic barcode image, binarization is used to reduce the image to black and

white extremes [7]. Liu showed that some binarization algorithms may be used for real-

16

time processing of barcodes [7]. Now, the signal of each data module may easily be

detected and the barcode is decoded.

However, binarization only works for monochromatic barcodes. High density

color barcodes rely heavily on timing modules. Some monochrome 2D barcodes, like

DataMatrix, include timing modules. Figure 5 (right) shows a DataMatrix barcode with

the data area dimmed so that timing blocks can be seen easily. These timing blocks are

essential for decoding while streaming high density barcode images [2].

4. Conclusion

In this paper we propose a framework for Mobile Color Barcode Streaming. Our

framework utilizes open barcode formats like QR, Aztec and DataMatrix. We design and

implement a high density barcode specification with attributes that make it easier to

decode in real time. This format will include easily identifiable corners and timing

mechanisms as well as a large data area. We will employ error correction techniques and

experiment with the alternate use of color selection and error correction techniques to

maximize data density while maintaining readability.

To achieve corner detection in real time, our framework will utilize pattern

detection as opposed to expensive algorithms like Hough transformations or edge

detection. With the use of colors in the barcode specification, searching for a pattern

containing a specified color will yield faster results. We will utilize a fast corner

detection approach to further enhance the corner detection process.

17

We will analyze the use of motion detection via accelerometer sensors to estimate

the new locations of corners and provide parameters to be used in reducing blur.

Primarily, we will use motion detection to run algorithms that reduce blur. In addition,

we will also utilize error correction within the barcode specification to fight blur.

We will evaluate our framework on low-quality cameras. Many smartphones

today have both front and rear facing cameras. However, front-facing cameras are

typically a much lower resolution than rear facing cameras. The use of front-facing

cameras allow us to achieve two-way communications using our framework. We will

develop a simple communication protocol to achieve this.

18

REFERENCES

[1] I. 18004:2006. Automatic identification and data capture techniques - QR code

2005 bar code symbology specification.

[2] I. 16022:2006. Automatic identification and data capture techniques - Data Matrix

barcode symbology specification.

[3] T. Hao, R. Zhou, G. Xing. COBRA: color barcode streaming for smartphone

systems. In Proceedings of the 10th international conference on Mobile systems,

applications and services, 2012.

[4] I. 15420:2009. Information technology - Automatic identification and data `

capture techniques – EAN/UPC bar code symbology specification.

[5] I. 24778:2008. Automatic identification and data capture techniques - Aztec Code

bar code symbology specification.

[6] S. Perli, N. Ahmed, and D. Katabi. Pixnet: interference-free wireless links using

lcd-camera pairs. In Proceedings of the sixteenth annual international conference

on Mobile computing and networking. ACM, 2010.

[7] C. Chu, D. Yang, and M. Chen. Image stabilization for 2D barcode in handheld

devices. In Proceedings of the 15th international conference on Multimedia.

Multimedia, 2007.

[8] H. Tong, M. Zhang, and C. Zhang. Blur Detection for Digital Images using

Wavelet Transformation. Multimedia and Expo, 2004.

19

[9] D. Nguyen, D. Bui, and T. Ha Le. Blur Estimation for barcode Recognition in

Out-of-Focus Images. Pattern Recognition and Machine Intelligence, 2011.

[10] H. Wang and M. Brady. Real-time corner detection algorithm for motion

estimation. Image and Vision Computing, 1995.

[11] X. Li, M. Zou, and J. Zhou. The multi-QR codes extration method in illegible

image based on contour tracing. Anti-Counterfeiting, Security and Identification,

2011.

[12] ZXing ("Zebra Crossing"). http://code.google.com/p/zxing/.

[13] Y. Liu, J. Yang, and M. Liu. Recognition of QR Code with mobile phones.

Control and Decision Conference, 2008. IEEE, 2008.

[14] A. Sun, Y. Sun, and C. Liu. The QR-code reorganizatio in illegible snapshots

taken by mobile phones. Computation Science and its Applications, 2007. IEEE,

2007.

20