Augmented Reality System

ECE 4901 -- Senior Design

Spring 2015

Final Report

Team 1510

ECE Project Members:

Daniel Adamowicz

Amy Clark

Victoria Tran

Faculty Advisor

John Chandy

Office: ITE 437

Phone: (860) 486-5047

E-mail: chandy@engr.uconn.edu

Graduate Student Advisor:

Rajeev Ghimire

E-mail: rag10005@engr.uconn.edu

Sponsor:

Qualtech Systems, Inc.

Professor Krishna Pattipati

UConn ECE Department

Summary:

The intent of our project is to create a product that will allow users to efficiently diagnose

and troubleshoot a system through the use of smart glasses. The smart glasses will act as a voice-

controlled user interface for the mobile application PackNGo. The smart glasses will access

Qualtech System’s (QSI) remote diagnostic server (RDS) to retrieve necessary diagnostic and

repair instructions and relay the information back to the user through the smart glasses.

Background:

To diagnose a given system, a technician needs to be familiar with that system.

However, any one technician, no matter how experienced, can not possibly know all the causal

relationships in a system or a family of systems. The problem becomes more acute over time as

newer systems are developed, older systems become less common, and technicians ultimately

have more things to remember. This knowledge gap could be potentially fixed with the use of

diagnostic support systems, whose recommendations can be displayed on devices such as

phones, tablets, or computers. This causes another problem: use of these devices requires the

worker to look away and move their hands from the system, which can negatively impact their

efficiency and safety.

With the use of smart glass technology and Augmented Reality (AR), our group will use

previously developed software to aid workers in diagnostic repair of a wide variety of systems.

The project will aim to improve the accuracy and reduce the chance of worker error due to

inexperience or the need to work with unfamiliar systems. The diagnostic information will be

stored in an XML file and uploaded to the glasses. The application must retrieve information

from the file and display it on the glasses. The main benefits of smart glasses over a mobile

phone are heads-up display of critical information and a completely hands-free experience by

utilizing voice commands.

Solution:

Software:

TEAMS-Remote Diagnostic Server (RDS) is a network-based application that gathers and

processes data taken from active equipment and systems. The resulting diagnostics and system health

information can be accessed via web browser. All results (e.g. procedures, diagnosis, testing) are recorded

to TEAMS-RDS for later analysis and future reference.

PackNGo is a software product that is part of QSI’s remote solutions. It extends the Guided

Troubleshooting functionality from the TEAMS system to smartphones and tablets as a mobile

application. The application utilizes JavaScript and communicates with TEAMS-RDS. When PackNGo is

opened, the user inputs the system being diagnosed and selects any relevant error codes and available

tools from a checklist provided by PackNGo. PackNGo then displays a series of instructional checks

through multiple pages. The user answers yes or no questions to obtain the most likely fault causing the

problem in the system.

One aspect of our design is to take the current UI as shown in Fig. 1 and adapt it to create an

efficient display for smart glass. We plan to utilize this current software and adjust it to be controlled with

voice commands to move through the troubleshooting questions and to bring the user to a repair solution.

The final product should have no remaining touch controls for navigation.

Android Studio is our main source of creating our Glass application. It allows us to create a fully

functional UI and process the information parsing using the Java programming language. Android Studio

is an ideal way to work with a Gradle based system which is what is used for google glass. Android studio

displays the project files in a way that is easy to work with. Each of the classes which are defined as

activities are organized and easy to navigate. The project view in android studio groups the files based on

function. For example it groups the build files, manifest files and resource files into there own folders so

it is easier for us to manage our project. Android studio was the main resource for programming google

glass.

In addition we also used Eclipse to create the parser and check that it was working correctly.

Eclipse can use console commands to output the information and check that it is correct. This is easier

and faster to do than on Android studio. Eclipse and Android studio have the same libraries that we can

use so there is no problem importing the code to Android studio later.

Current UI on mobile device

Fig. 1. Screenshots of current version of PackNGo. The top two screenshots are the lists of user

inputs to customize PackNGo’s instructions. The bottom two screenshots are examples of

PackNGo’s yes/no interface and sample instructions.

Use-Case Diagram

Fig. 2 Use-Case Diagram of network integration of final product

Sequence Diagram

Fig. 3: Sequence diagram, showing ideal flow of interaction of all of the systems in final design

Hardware:

Initially, our team researched technical specifications to narrow down our choice of smart

glasses to two brands: the Epson BT-200 and Google Glass. We considered attributes such as

display size, processing power, battery life, and wifi/bluetooth connectivity. It was also a

requirement that the device have a microphone in order for us to incorporate voice commands.

The Epson BT-200 is a simple smart glasses model we wish to use to integrate PackNGo

with as a proof of concept. The BT-200 is operated by a wired trackpad in the user’s hand which

operates similarly to a touchpad mouse on a laptop computer. The glasses use a 960x540 pixel

display that appears to float in the center of the user’s vision. The main screen is similar to a

desktop on a computer: there are icons for different applications and interaction is done using a

mouse pointer controlled by the touchpad. The relevant technical specs of the BT-200 are 1 GB

of memory, 8 GB of storage space, and Android 4.0.4 operating system. [1]

The battery life for

the Epson is advertised as approximately six hours at room temperature, although our team found

it to be closer to 3-4 hours in testing. Due to the BT-200’s similarity to a normal computer, the

UI changes necessary to integrate the existing PackNGo software is minimal. The BT-200 is

considered only a proof of concept as voice commands are not officially supported and would

require signal processing analysis to fulfill the hands-free requirement of the design.

We have selected Google Glass to be the brand of smart glasses for our final design

solution. Glass supports voice commands, but its general user interface is significantly different

than the BT-200 or a computer. The screen for the glasses is a prism positioned in the top right

corner of the right lens. This leaves the rest of the user’s field of vision unobstructed. Google

Glass has a minimalist style UI that has much less space for text and pictures than a computer

screen. There is also no support for direct mouse clicks or touch control. This means the current

PackNGo design would not work at all on Glass and must be completely redesigned. Glass

operates almost entirely on voice commands by default, and applications written in the Google

Developer Kit (GDK) can easily integrate voice commands. The relevant technical

specifications for Google Glass are 1 GB of memory, 12 GB of storage, and Android 4.4

operating system. [2]

Critics report the battery life of the Glass to be about three to five hours of

extensive use [3]

which we have found to be approximately correct.

Theory:

For the Epson model, the PackNGo functionality is essentially mirrored on the BT-200

interface as the BT-200 is usable similarly to a computer. PackNGo required no additional

modification to function on the BT-200.

For the Glass model, the application must have access to an XML document containing

information for some given system. This XML file is generated by Qualtech software and given

to us directly. The Android application was created from scratch to display the necessary

information in an organized manner. Part of this application parses the XML file and extracts

the necessary information, then displays this information on the Glass. The application is opened

with and fully controlled by voice. After development of this application, we will test and verify

the compatibility of our application with both models of smart glasses. The application running

on the Glass will then be tested in a field environment. The tree diagram can be interpreted as in

Fig. 4:

Fig. 4: Visualization of tree diagram structure

We will need to understand the diagnostic process of PackNGo to ensure that we can

utilize QSI data through the XML file. Contact with QSI chief architect and software engineer

was necessary to avoid complications.

We will need to export the current TEAMS model of air handling unit (AHU) to RDS to

the QSI server to convert diagnostic information to XML file format. This system will be the

test environment for the final product.

The final product should function properly in environments such as automotive facilities,

aerospace industry, hospitals, semiconductor fabrication plants, and more, all of which are

potential QSI clients.

Preliminary Observations:

After some basic testing of the Epson BT-200, we have noticed that the design is not

ideal for a hands free program. First, the touchpad input is connected by a wire. This feature is,

unfortunately, bulky. Ultimately, this means that the BT-200 is not intended to be a hands-free

device without significant manipulation of sensor data within an application. In order to utilize

voice commands on the Epson, third party speech recognition software would need to be

integrated into the QSI software. However, we can still use this device to display PackNGo as is

for a proof of concept for future augmented reality projects.

Another issue is in the way the BT-200 screen is displayed. The screen appears to float

in the center of the user’s field of vision. Although the opacity can be adjusted, it is still possibly

disruptive for someone trying to work on a system. For future augmented reality projects, this

feature could potentially be useful for projecting useful images and data directly onto a system.

For the purposes of this project, the positioning of the screen is only a disadvantage.

The biggest issue with Google Glass is its heat dissipation. We found that using the

Glass for extended periods of time (longer than 30 minutes) could cause the Glass to get

excessively hot. A warning screen appears when reaching critical temperatures. We have

experienced rare system crashes possibly caused by overheating. To remedy this, we

recommend not operating Glass for longer than intervals of thirty minutes, and turning it off in

between intervals to cool. The system could likely operate for longer, but in the interest of being

safe, we recommend limiting its on time to thirty minutes.

Glass voice commands can be overly sensitive for short, simple phrases. When the

microphone detects human voice, it starts attempting to match its microphone data to all possible

voice commands. This means that a voice command could register even if it is said improperly

or incompletely. For example, the command “Take a picture” sometimes works just by the user

saying “Take a pic” or similar incomplete commands. The purpose of this is so that people with

accents or speech impediments can still use the Glass. This causes issues with multiple voice

commands that sound similar and are ready at the same time. A related issue is that once a voice

command is processed (even if the user is not done saying it), the Glass reacts and, depending on

the action, will potentially “ready-up” another set of voice commands.

We encountered the aforementioned problems when using the contextual commands

inside the application. To open the contextual commands, the user must say “ok glass” (this

command can not be changed). After the “ok glass” command is processed, the program then

waits for the next set of commands, which includes the command “Yes”, to continue traversing

the PackNGo system data. We have noticed that:

1) “ok glass” seems to trigger before the phrase is finished,

2) prepares to listen for “Yes”,

3) hears “glass” (or at least, part of the word),

4) and then triggers “Yes” before the user has a chance to provide meaningful input. This

inconsistency in our application could be fixed by changing the voice commands from “Yes” and

“No” to something else; unfortunately, there does not seem to be any logical replacement for

these words that would make sense for any and all given diagnostic prompts. This is simply

because the diagnostic prompts are written in a way so that all PackNGo buttons are the same for

every step (yes and no).

Specifications:

Product Google Glass Epson BT-200

Price ($) $1500.00 $699.99

Display Prism projector, 640×360 pixels 960×540 pixels screen

Onboard Storage (GB) 16 flash total (12 GB of usable

memory)

8 GB

OS Android 4.42 Android 4.0.4

Power Lithium polymer battery (2.1 Wh) Lithium poly 2720mAh

CPU OMAP 4430 SoC dual-core TI OMAP 4460 1.2 GHz dual core

Memory 1GB RAM (682MB available to

developers)

1 GB

Sound Bone conduction transducer headset, surround sound

Camera Photos – 5 MP, videos – 720p VGA

Connectivity Wi-Fi 802.11b/g, Bluetooth, micro

USB

Wi-Fi (IEEE802.11/b/g/n), bluetooth,

microUSB 2.0

Table 1 - Relevant technical information for both brands of smart glasses

Implementation:

The XML parser was written in Java and built using Eclipse. The parser utilized the

DOM and SAX existing parser classes to gather data from the XML. The data we need to

retrieve from the XML is stored in nodes with numbered labels. The data we need to extract

from each node is the label number, instruction string, outcome actions (typically yes/no), and

the outcome pointers. The label number keeps track of which step the program is currently

reading. The instruction string and outcome actions tell the program what to print to the screen.

The outcome pointers tell the program which node to go to next based on what command

(yes/no) the user says. The parser simply scans the XML file for the relevant tag names and

returns the necessary data.

Glass applications can be designed using one of three templates: static cards, live cards,

or immersion. We chose to create an immersion application because it gives us maximum

control over the Glass interface. Immersion applications allow us to continually update

information on the screen without a heavy impact on Glass’s processing power.

Due to Glass’s minimalist UI, the amount of information displayed on each screen is

limited. The yes/no buttons have been changed to voice commands, so they do not take any real

estate on the screen. The pictures and text can be displayed well by resizing them slightly.

Fig. 5: Example of application screen with instruction string.

Fig. 6: Contextual pop-up menu containing typical yes/no choices.

There was one major issue with our original implementation. The XML file to be parsed

was included in the Java package and uploaded to Glass. When the parser runs, the program can

not locate the XML file and is completely unable to retrieve any data. As of this time, the reason

for this problem is unknown. Using Windows Explorer, very limited file paths can be accessed,

so it is difficult or impossible to locate files uploaded to Glass.

As a workaround, we converted the entire XML file to a string and wrote it explicitly

inside the parser code. Doing so requires the XML to be modified to eliminate all quotation

marks (due to Java string syntax), done using a find-and-replace tool in any text editing program.

This string workaround was able to be parsed properly by our application [4]

.

Budget:

The main focus of this project is developing an existing software for use on Glass. The

Epson BT-200 and Google Glass are both provided by the graduate student group and cost our

team nothing. The Integrated Development Environments (IDEs) and all Qualtech software is

free for us to use. Since the entire project is software based, no other parts need to be purchased.

Therefore, the budget for this project is zero.

Timeline:

The main focus of the first semester was to research the many types of different smart

glasses available and choose one most appropriate for the task of aiding remote diagnostics. We

also worked on Android development and gaining access to the current PackNGo software,

TEAMS-RDS. We experimented with BT-200 features and ran the original PackNGo software

on them.

In the second semester, we began full development of the Glass application. A mock-up

UI for the Glass application was designed. We analyzed the XML file and developed the Java

parser. We developed the full Android application with integrated voice commands, and ran the

parser program on the Glass to extract data.

Appendix:

Citations:

[1] http://www.epson.com/cgi-bin/Store/jsp/Product.do?sku=V11H560020

[2] https://support.google.com/glass/answer/3064128?hl=en

[3] http://www.techradar.com/us/reviews/gadgets/google-glass-1152283/review/

Software References:

https://www.teamqsi.com/products/teams-rds/

https://www.teamqsi.com/products/packngo/

http://developer.android.com/tools/studio/index.html

http://eclipse.org/

User Guide:

1) Create an XML file for a system using TEAMS software.

2) The XML file to be used must be modified (refer to Implementation, [4], page 14) and

written directly into the string variable in the Java package.

3) To upload the application, use Android Debug Bridge (ADB) Sideloading. For more

information refer to http://googleglassfans.com/archives/3054/sideload-google-glass-

apps-windows/

4) Once the application is uploaded, open from the main menu with “Ok glass” > “Check

system”

5) The first step of the used XML file will automatically appear. To continue, say “Ok

glass” followed by the related voice commands (e,g, “Yes” or “No”)

6) To go back to the previous step, say “Ok glass” followed by “Go back”

7) To exit the application, swipe down on the Glass touchpad.

Acknowledgements:

We wish to acknowledge and thank the previous senior design teams and graduate

students for providing real-time monitoring and creating a link between the sensor values of the

HVAC system in the Information Technology Engineering Building (ITEB) at UConn, to the

failure protection software, TEAMS, run by QSI. We also wish to thank Qualtech Systems, Inc.

and the University of Connecticut Electrical Engineering Department, for support with funding,

advising, and sponsoring this project. Special thanks to our graduate advisor Rajeev Ghimire

and project advisor John Chandy for guiding us through development.