221
GOOGLE GLASS PROJECT A Final Report Presented to The Advisory Committee and Project Sponsors Senior Design Course of Fall-2013 & Spring 2014 Florida Institute of Technology In Partial Fulfillment of the Requirements for the Course MAE 4190/4193/4194: Mechanical Engineering Design by Google Glass Group Cheng Lu, Kyle Coleman, Luke Glesener, Constantin Jessen, Brandon Mills, Shaopeng Han, Yanglong Lu April 14, 2014 Accepted by: Dr. Beshoy Morkos, Advisory Committee Chair Mr. Varun Menon

Google Glass Project

Embed Size (px)

Citation preview

Page 1: Google Glass Project

GOOGLE GLASS PROJECT

A Final Report

Presented to

The Advisory Committee and Project Sponsors

Senior Design Course of Fall-2013 & Spring 2014

Florida Institute of Technology

In Partial Fulfillment

of the Requirements for the Course

MAE 4190/4193/4194: Mechanical Engineering Design

by

Google Glass Group

Cheng Lu, Kyle Coleman, Luke Glesener, Constantin Jessen, Brandon Mills, Shaopeng

Han, Yanglong Lu

April 14, 2014

Accepted by:

Dr. Beshoy Morkos, Advisory Committee Chair

Mr. Varun Menon

Page 2: Google Glass Project

i

1. ABSTRACT

The google glass project was created to enhance the Google Glasses with an interface to allow it to

interpret audio input and output text to the google glass screen. The project motivation was to improve the

overall quality of life for those who are hearing impaired or completely deaf.

In the first semester, the project began by looking into which microphones could be used for the audio

input as well as general research into what made microphones work. Different specifications were taken

into account to include; sensitivity, polar pattern, range, and recording mechanic. Once the microphone

criteria were confirmed, primarily that the group would be utilizing two cardioid condenser microphones,

the microphone team produced a list of viable microphones available for purchase. This list was pared

down until specific microphones were selected and ordered. The POM-3535L-3-R was the best choice out

of four microphones that were sent through the final decision matrix. These findings were submitted to

David Beavers to conduct the ordering process. Once microphone research was completed, the team focused on the platform for the device as well as the dictation software. The team decided to focus their

efforts on the Raspberry Pi and an Android phone based app. The dictation software to be used would be

PocketSphinx for the Raspberry Pi and iSpeech for the Android phone. The group did additional research

into Sound Source Separation methods (sss) as well as filtering software. Testing software used in this

project for microphone analysis was a software called Praat. The group took the time to look into

additional products similar to the Google Glass in the expectation that they may not be able to acquire the

Google Glass for testing. The front runners for this expansion were; Vuzix M100 Smart Glasses, Meta 3D

Space Glasses, and the GlassUp.

In the second semester, the group was able to acquire a working Google Glass. With the purchase of the

Google Glass, much of the research from the previous semester had to be re-assessed. The google glass

had no support for external microphones, and it was discovered that the programing required to integrate

an external microphone would endanger the health of the Glass and was beyond the groups programming

ability. The Raspberry Pi was also removed from the project because it was too slow and did not interface

with the Google Glass. The group initially focused on programming for both the Android smart phone

and the Google Glass itself. Once a successful app, The Live Card App, was created, most of the group’s

efforts was focused on improving the code, and updating deficiencies. Eventually, the group was able to

create an app that continuously converted speech to text and printed the text to the Google Glass screen.

This application was named The Immersion App, later named Live Subtitles. Live Subtitles is effective

within 3 to 5 feet when used in environments with varying dB. In order to improve range and accuracy,

the group began researching filtering technology, however the expected programming requirements

would push the project beyond its slated timeline. The group has formulated multiple suggestions for

future project advancement. Overall, the group was successful in achieving its prescribed goals; the

following document denotes the specifics of all research and results for the project.

Page 3: Google Glass Project

ii

2. ACKNOWLEDGMENTS

The google glass group would like to personally thank their senior advisor Dr. Beshoy Morkos for his

support and guidance on this project. The team would also like to thank David Beavers for his support in

ordering of the microphones and Raspberry Pi and for lending his technical expertise in the field of

electronics. Finally the group would like to thank Dr. Crawford, Varun Menon and Shaun Davenport for

their support and advice on the project.

Page 4: Google Glass Project

iii

3. TABLE OF CONTENTS

1. Abstract ......................................................................................................................... i

2. Acknowledgments........................................................................................................ ii

3. Table of Contents ........................................................................................................ iii

4. List of Tables .............................................................................................................. vi

5. List of Figures ............................................................................................................ vii

6. Introduction .................................................................................................................. 1

6.1. Problem Statement ................................................................................................ 1

6.2. Motivation ............................................................................................................. 1

6.3. Goal ....................................................................................................................... 1

6.4. Terminology, Constraints and Requirements ....................................................... 1

6.5. Global, Social, and Contemporary Impact ............................................................ 3

6.6. Group Logistics..................................................................................................... 4

7. Background .................................................................................................................. 5

7.1. General Research .................................................................................................. 5

7.2. Google Glass Alternatives .................................................................................. 16

7.3. Patent Search....................................................................................................... 19

7.4. Current State of the Art ....................................................................................... 20

7.5. Reverse Engineering ........................................................................................... 21

8. Recommended Solution ............................................................................................. 22

8.1. Solution Principles .............................................................................................. 22

8.2. Analysis .............................................................................................................. 23

8.3. FMEA ................................................................................................................. 25

8.4. Decisions ............................................................................................................. 27

8.5. QFD .................................................................................................................... 32

8.6. Bill of Material.................................................................................................... 34

9. Programming.............................................................................................................. 36

9.1. Programming for the Google Glass .................................................................... 36

9.2. Programming Process ......................................................................................... 38

Page 5: Google Glass Project

iv

9.3. Generated Applications....................................................................................... 39

9.4. Application Operation: ....................................................................................... 39

9.5. Android Smartphone Programming .................................................................... 42

9.6. Programming the RPi.......................................................................................... 43

10. Testing...................................................................................................................... 44

10.1. Introduction ....................................................................................................... 44

10.2. Praat Software ................................................................................................... 45

10.3. Read Time Research ......................................................................................... 45

10.4. Audio-to-Displayed Text .................................................................................. 46

10.5. L/R Comparison ................................................................................................ 47

10.6. Dictation Software Comparison........................................................................ 50

10.7. Audio Test for Google Glass ............................................................................ 51

10.8. Launch Time of “Speech to Text” .................................................................... 51

10.9. Working Range of “Speech to Text” ................................................................ 52

10.10. Incremental Test for “Speech to Text” ........................................................... 52

10.11. Ambient Noise Threshold Test for “Speech to Test” ..................................... 54

10.12. Ambient Noise Threshold Test for “Live Subs” ............................................. 56

10.13. Accuracy of “Live Subs” ................................................................................ 57

10.14. Polling ............................................................................................................. 57

11. Conclusions and Future Work ................................................................................. 59

11.1. Final State of the Project ................................................................................... 59

11.2. Major Accomplishments ................................................................................... 59

11.3. System Validation ............................................................................................. 60

11.4. Recommendations for Future Work ................................................................. 63

Appendices ........................................................................................................................ 65

Appendix A Meeting Minutes ................................................................................... 66

Appendix B Emails .................................................................................................. 102

Appendix C Executive Summaries .......................................................................... 129

Appendix D Raspberry Pi Programming Log.......................................................... 153

Appendix E Dictation Research Initial Research Log ............................................. 156

Appendix F Microphone Search Log....................................................................... 157

Appendix G Experiments......................................................................................... 159

Page 6: Google Glass Project

v

Appendix H Smartphone Audio-to-Text ................................................................. 166

Appendix I Filter Tester App ................................................................................... 173

Appendix J AGC MATLAB CODE ........................................................................ 189

Appendix K Google Glass Application ................................................................... 192

12. References .............................................................................................................. 205

Page 7: Google Glass Project

vi

4. LIST OF TABLES

Table 1: Smart Watch Comparison ............................................................................................... 18

Table 2: Microphone Requirements.............................................................................................. 24

Table 3: FMEA ............................................................................................................................. 26

Table 4: Platform Decision Matrix ............................................................................................... 28

Table 5: Microphone Decision Matrix .......................................................................................... 31

Table 6: Budget Allocation ........................................................................................................... 34

Table 7: Expenses ......................................................................................................................... 34

Table 8: Audio-to-Display Text Results ....................................................................................... 46

Table 9: L/R Comparison at 3ft .................................................................................................... 48

Table 10: L/R Comparison Results Chart at 2 ft........................................................................... 49

Table 11: Dictation Software ........................................................................................................ 50

Table 12: Microphone Range........................................................................................................ 51

Table 13: Launch Time of “Speech to Text” ................................................................................ 51

Table 14: Accuracy of “Speech to Text” in Various Distance and Words ................................... 53

Table 15: Accuracy of “Speech to Text” in Various Environment & Distance ........................... 54

Table 16: Working Range of “Speech to Text” in Different Environment .................................. 54

Table 17: Accuracy of “Live Subs” in Various Environment & Distance ................................... 56

Table 18: Working Range of “Live Subs” in Different Environment .......................................... 56

Table 19: Accuracy of "Live Subs" in Different Environment ..................................................... 57

Table 20: Result of Polling ........................................................................................................... 57

Table 21: System Validation Checklist......................................................................................... 61

Page 8: Google Glass Project

vii

5. LIST OF FIGURES

Figure 1: Fiber optic microphone ................................................................................................... 5

Figure 2: Dynamic Microphone ...................................................................................................... 5

Figure 3: Ribbon Microphone ......................................................................................................... 6

Figure 4: Condenser electrets microphone ..................................................................................... 6

Figure 5 - Polar Pattern ................................................................................................................... 7

Figure 6: Raspberry Pi [11]............................................................................................................. 8

Figure 7: Butterworth Filter [20] .................................................................................................. 15

Figure 8: Bessel Filter [20] ........................................................................................................... 15

Figure 9: Chebyshev Filter [20] .................................................................................................... 16

Figure 10: Vuzix M100 Smart Glasses [24] ................................................................................. 17

Figure 11: Meta 3D Space Glasses [25] ....................................................................................... 17

Figure 12: GlassUp [26]................................................................................................................ 18

Figure 13: SmartWatch, Pepple, Galaxy Gear (left to right) ........................................................ 18

Figure 14: Functional Flow Block Diagram ................................................................................. 22

Figure 15: Final Concept .............................................................................................................. 23

Figure 16: Approximate frequency ranges [37] ............................................................................ 24

Figure 17: QFD ............................................................................................................................. 33

Figure 18: Implicit intent [47]....................................................................................................... 36

Figure 19: Live Card Operation [50] ............................................................................................ 37

Figure 20: Low Frequency (Left) and High Frequency (Right) Live Cards [X8] ........................ 38

Figure 21: Immersion [51] ............................................................................................................ 38

Figure 22: Exporting Code to the Glass [52] ................................................................................ 39

Figure 23: Launching Application ................................................................................................ 39

Figure 24: App Listening .............................................................................................................. 40

Figure 25: Review of Conversation .............................................................................................. 40

Figure 26: Text History ................................................................................................................. 40

Figure 27: Return to Speech Recognition ..................................................................................... 41

Figure 28: Tap to Enter Speech Recognition Mode...................................................................... 41

Figure 29: Deleting Files .............................................................................................................. 41

Figure 30 Subtitle Research .......................................................................................................... 45

Page 9: Google Glass Project

viii

Figure 31: Audio-to-Display Results ............................................................................................ 46

Figure 32: L/R Comparison Results Chart at 3 ft. ........................................................................ 47

Figure 33: L/R Comparison Results Chart at 2 ft. ........................................................................ 48

Figure 34: Speech to Text Accuracy ............................................................................................. 52

Figure 35: Accuracy vs. Distance & Words ................................................................................. 53

Figure 36: Final State and Design ................................................................................................. 59

Figure 37: Experiment Setup ...................................................................................................... 162

Page 10: Google Glass Project

1

6. INTRODUCTION

6.1. Problem Statement

The objective of this project is to modify the Google Glass with a system which will allow people who

are hearing-impaired to participate in spoken conversations. This will be done by transforming that audio

into text through noise filters and voice to text software, and then displaying it on the Google Glass.

6.2. Motivation

The primary motivation for this project was to improve the overall standard of living for the hearing

impaired or completely deaf. Relying on others to interpret what someone is saying for them can be

difficult. So can limiting oneself to only being able to talk to a small community of people who

understand sign language. By modifying the google glass to be used as a voice to text device, we are

allowing the hearing impaired and deaf the freedom to interact with more people in more places without

being hindered by their disabilities. Ideally, science will eventually advance enough to allow the hearing

impaired or deaf to completely recover their faculties. However, until that time comes, it makes sense to

use the technology of today to make that wait just a little easier.

6.3. Goal

The major goal for the team was to determine the applicability potential for the Google Glass and any

secondary software/hardware needed to support the functionality. By the end of the first semester the

group was required to have a working prototype of a voice to text to a remote display and by the end of

the second semester the team needed to have the voice to text working and the text being displayed on the

Google Glass.

6.4. Terminology, Constraints and Requirements

6.4.1. Terminology

System - Refers to all the combined components of the product from the audio capturing

device to the text being displayed.

Dictation Software - The software that converts audio signals to textual form.

User - The person directly operating the system.

Audio Capture Device - The device used to capture the audio, such as a microphone.

Filtering - The process of removing undesired noise from a sound signal.

For the following constraints and requirements, “shall” designates requirements that must be fulfilled in

order for the experiment to be deemed a success. Items that contain “should” are items that are desired,

but are not needed in order for the project to be deemed a success.

6.4.2. Constraints

The system shall use the Google Glass to display text.

The system shall be housed in a portable package.*

Page 11: Google Glass Project

2

The complete system cost shall remain within the budget assigned by the sponsor.

6.4.3. Requirements

The team was given four initial stakeholder requirements: The system shall do real time speech to text, be

mobile, safe to use, and user friendly. From this the team generated functional and nonfunctional

requirements to meet the key stakeholder requirements. The functional requirements from the list were

later placed in a QFD to determine their relative importance.

6.4.3.1. Real Time Speech to Text (Functional)

1. The audio capturing device shall be able to capture all audio within the human sound

threshold at the point of capture. (Up to 100 decibels sound pressure level).

a. Based on human hearing range from microphone research.

2. The audio capturing device shall have a minimum frequency response range of 300 Hz - 3.4

kHz.

a. Based on human speech range from microphone research.

3. The filtering function of the system should be able to filter out all ambient noises.

a. Ambient noise is defined as any sound other than the desired sound source.

4. The system shall be able to display the text under 3 seconds from the audio being detected.*

5. The system shall capture audio from a nominal distance of 3 ft.**

6.4.3.2. Mobile (Functional)

6. The system shall be under 8” x 3” x 1”.

a. Based on Phablet (Samsung Galaxy Note 3) and Portable gaming devices (PS Vita

and Nintendo 3DS)

7. The system should have a battery life greater than 4 hours.

a. Initial estimate.

8. The system should be weather resistant.

9. The system shall be durable enough to withstand daily use.

a. “Daily Use” shall be defined in the future.

10. The system should weight under 8 oz.

a. Based on average weight of mobile phone.

6.4.3.3. Safety

11. No component of the system shall exceed a temperature of 140F.

a. Based on ASTM C10552 for metallic surfaces.

12. No components of the system shall have sharp edges.

13. All electrical components shall be electrically grounded.

14. The system shall not reuse the user’s personal information without the user’s permission.

6.4.3.4. User Friendly (Feel)

15. The system shall be user friendly.

a. User friendly shall be defined as having an average user friendliness rating of 7 or

greater on a 0-10 scale (10 being the greatest) by a polled group.

Page 12: Google Glass Project

3

16. The system should be comfortable to wear.

a. Comfort shall be defined as having an average user comfort rating of 7 or greater on

a 0-10 scale (10 being the greatest) by a polled group.

17. The user should have the ability to select the formatting of the text displayed. (i.e. font, font

size)

18. The system should be aesthetically pleasing.

a. Aesthetically pleasing shall be defined as having an average aesthetics rating of 7 or

greater on a 0-10 scale (10 being best) by a polled group.

19. The dictation software shall be controlled easily.

20. The user should be able to indicate which sound source to be displayed.

a. “Indicate” to be defined in the future.

21. The system should be able to indicate the sound level of the source.

a. This is to give the user some context. Example textual output: “I was driving down

the road and “BAM” there was something in the road.” where “BAM” was shouted.

22. The system should notify the user when it detects audio and begins converting the audio after

the system leaves a dormant state.

a. This feature would notify the user that it is going to be displaying text if the system

had not detected audio for a length of time.

b. The purpose is to get the users attention so they do not miss information coming on

the display.

23. The system should be able to display to non-Google Glass sources.

a. This is to widen the systems customer base.

24. The system should have different mode settings.

a. Example: Have a lecture setting to increase the range and sensitivity of the audio

capturing settings.

25. The system should be able to tell which source the sound is from and store it, so that

information can be relayed to the user.

26. The system shall have a cold start-up time of under 3 mins.

a. This is an initial estimate.

27. The dictation software should have multi language capability.

28. The system should have a data storage feature to allow the user to recall past data

a. This will also aid in debugging the software.

29. The system should indicate to the speaker when it is not able to keep up.

* Based on the read-time research

** Based on the audio-to-text experiment

6.5. Global, Social, and Contemporary Impact Overall the impact this project will have on the social, global, or contemporary scale is expected to be

very low. The project will have a primarily focus on the social scale. By focusing on the hearing

impaired, our group hopes to allow this sector of the population to more easily interact with the rest of

society. Historically, those who are deaf or hearing impaired have been limited in their ability to interact

with people who do not understand sign language or do not have the time to write down everything they

need to impart to the hearing impaired/deaf. This barrier in communication results in: confusion, social

issues, trouble obtaining or holding certain modes of employment, and various other issues. With the

implementation of our modified Google Glass, or devices similar to what we have proposed, we are

offering the hearing impaired more freedom to interact with their environment and ultimately improving

their standard of living.

Page 13: Google Glass Project

4

The following are some instances where the Google Glass would not only help those wearing them, but

also people around them: lectures, presentations, award ceremonies, security checkpoints, or interacting

with police personnel, emergencies, etc. Any instance where information must be transmitted to someone

quickly when there may not be a mediator who knows sign language available, will be positively affected

by this device.

Some issues that may be presented by our project would primarily be financial, as well as the creation of a

social stigma. As far as financial issues are concerned, it is not entirely clear how expensive the finalized

Google Glass will be nor who will be able to access it, however, the app itself will not carry any apparent

additional cost. Those who are unable to acquire this device or similar devices, may be left behind if

performances, lectures, and award ceremonies no longer feel the need to provide interpreters because

everyone is expected to own one of these devices. Though this is a far reaching assumption, which may

not come to fruition anytime soon and hopefully become a “non-issue” with advanced medical

technology. Eventually, the results of this project might be available through health care or insurance; if

this is the case, some insurance or health care providers may have to change their policies and rates. With regards to social stigma issues, similar to how people may be viewed differently when they use a cane or

animal assistants, or even hearing aids, these people wearing obvious augmented reality (AR) glasses may

be automatically singled out as different or in some cases “weak” or “easy targets” for violent crimes.

Anything that draws attention to someone’s handicap is going to somehow affect how they interact with

the public. It is possible though, since the App provided is attached to a social media device, others who

are not impaired will wear the Glass in abundance and dilute this stigma cue. Although any device that

impairs active or passive situational awareness is going to make someone an easier target. In addition to

this, people who are older or less accepting of technology may have issues with using or wearing these

devices.

6.6. Group Logistics The group had one formal meeting a week to discuss the direction the project was going, tasks that

needed to be done, who would do them, and to address any questions or concerns. Meeting minutes were

taken for these meetings and are included in Appendix A. There was an Advisory meeting on a near

weekly bases throughout the year, in which the team would present their current works and findings to the

advisory committee and get feedback from said committee. Minutes for the advisory meetings were also

recorded and can be found in Appendix A. Part of keeping the advisory committee informed was writing

an executive summary of everything accomplished during the prevailing week. Any issues and upcoming

projects were also recorded in the executive summary. These summaries are located in Appendix C. The

group communicated via the group email when contacting people and companies external to the group, a

copy of these emails can found in Appendix B.

Page 14: Google Glass Project

5

7. BACKGROUND

7.1. General Research

7.1.1. Microphones

A microphone is a sensor or transmitter which converts sound waves into an electric signal. In the world

of microphones, there is a wide variety of different types that are designed for different purposes. The

type of microphone has a major impact on its performance depending on which situation it is used in.

Certain microphones are very unreliable when put under conditions the microphone was not designed for,

this lead to the focus on researching the different types of microphones and what they are designed to do.

7.1.1.1. Types

7.1.1.1.1. Fiber Optic Microphone

A fiber optic microphone replaces the metal wire, which in traditional

microphones is used to pick up sound waves, with an extremely thin glass

strand. A laser travels inside the glass strand. This allows the microphone to

use an optical system to convert acoustic signals into a modulated light

intensity signal [1]. Those signals are very accurate and while not being

electrical they do not cause any magnetic fields [2]. The microphone itself

does not require any metal components and therefore does not interfere with

many other sensitive electronic systems.

The fiber optic microphone is extremely accurate, as well as very light and

small in size, potentially making it very effective in this project. The

problem is that the fiber optic microphones are very expensive with starting

prices around $2000 and are not within the affordable range for this project.

7.1.1.1.2. Dynamic Microphone

A dynamic microphone uses a magnet and a coil of wire to induce a current

through the wire for the electrical output of the microphone. The magnet or

coil is connected to a diaphragm which vibrates as a result of incoming sound

waves. A dynamic microphone induced current follows the electromagnetic

principles of Faraday’s law of induction [3].

The dynamic microphone’s sensitivity is generally very low (around -90 dB)

[4] as well it’s capability to record human speech. The only issue with dynamic microphones is that dynamic microphones do not come in a small

scale and are generally large.

7.1.1.1.3. Ribbon Microphone

The ribbon microphone uses a metal ribbon suspended in a magnetic field to

absorb the sound vibrations. Inside the magnetic field, the ribbon microphone is connected to the output

of the microphone through an electrical connection. The alteration caused by the vibration of the ribbon

as the result of the sound waves, is transmitted electrically over to the microphone’s output [5].

Figure 1: Fiber optic

microphone

Figure 2: Dynamic

Microphone

Page 15: Google Glass Project

6

Though the sound quality of a ribbon microphone is very clear and does not

need any external power supply. The ribbon microphone is sensitive to

movement. The metal ribbon suspended in the magnetic field moves when the

microphone itself starts moving, affecting the output [6]. This ineffectiveness in

a mobile environment makes the ribbon microphone unsuitable for this project.

7.1.1.1.4. Condenser Microphone

A condenser microphone or capacitor microphone consist of two sides of a

capacitor. One side is a stretched diaphragm so that it can achieve a high

resonating frequency. The other side of the condenser is fixed. The condenser

microphone needs a constant current flowing through both plates of the

capacitor to function. Incoming sound waves cause the diaphragm to vibrate

which causes a change in current inside the capacitor. This change in current

then gets transmitted as the output from the microphone [7].

As for Electret Condenser Microphones, a sub-category for condenser

microphones, they use a permanently charged diaphragm, such as a magnet, and

therefore do not need a power input. This allows the electret microphone to be

built in small compressed sizes. They are commonly used in cellphones and

other electronic devices and in high demand. This reduces their price to a very

small amount, starting at $0.50 per microphone.

7.1.1.2. Sound Pressure Level

Sounds exert a pressure on the surroundings which is typically measured in decibels (dB). The dB scale is

based on the range of human hearing. A pressure at zero dB is at the extreme minimum that a human can

hear, whereas the human threshold for pain is around 130 dB.

7.1.1.3. Sensitivity

Sensitivity of microphones is a rather vague topic. It is typically measured with a 1 kHz sine wave at a 94

dB sound pressure level or 1 Pa of pressure. The accepted definition of sensitivity for a microphone,

typically given in negative dB, tells how much voltage will be output from a microphone at a given sound

pressure level. In combination with this, all microphones need a preamp gain which changes based on the

sensitivity of the microphone. The challenge is balancing the gain with the sensitivity of the microphone

[8].

7.1.1.4. Polar Patterns

A polar pattern is the sensitivity of a microphone relating to the direction the sound is coming from,

typically represented as a graph. For example in the graphs below, each band, going from the center

towards the outside, represents increments of 5 dB of sensitivity [9]. What polar pattern a microphone is

using, influences the microphones capability to capture sound from different directions. There are four

main types of polar patterns are presented in Error! Reference source not found. (from left to right):

cardioid, omnidirectional, bidirectional, and shotgun.

Though the choice of polar pattern is not essential, it has to be regarded to optimize the performance of

the microphone. This means, if the condenser microphone has an omnidirectional polar pattern, the housing of the microphone has to be built accordingly, so that there exist damping towards the not

focused directions, for example the 180 degree mark.

Figure 3: Ribbon

Microphone

Figure 4:

Condenser electrets

microphone

Page 16: Google Glass Project

7

7.1.1.5.

7.1.2. Platforms

7.1.2.1. The Raspberry Pi

The Raspberry Pi (RPi) is a small single-board computer developed by the Raspberry Pi Foundation in

the United Kingdom [10]. The RPi is approximately the size of a credit card and most often runs a

modified version of Linux called Raspbian, a cross between Raspberry and Debian. Although Raspbian

is the most common operating system (OS) other lesser known OSs can also be used. Most computers

have a permanent hard drive, which contains the operating system and data storage. RPi, on the other

hand, uses a removable SD card. The main features of the RPi are (Figure 6):

ARM11 700 MHz processor

512 MB of RAM

HDMI and RCA video out

Ethernet port

2 USB 2.0 slots

SD card slot

Figure 5 - Polar Pattern

Page 17: Google Glass Project

8

Figure 6: Raspberry Pi [11]

7.1.2.2. Android Devices

The two main smartphone categories currently on the market are Android devices, made by many

different developers and iPhones, made by the Apple Corporation. The actual android language is based

on Java and was created by Google. The Android language is simple and free to develop applications for.

It also has a massive online resource library filled with tutorials and example code. For these reasons, it

was chosen as a strong alternative to programming with the Raspberry Pi.

7.1.3. Speech Enhancement

Speech enhancement is the field that deals with the signal processing associated with speech recognition

and dictatio [12]. Its two main goals are to increase the intelligibility and to increase quality. Intelligibility

is how understandable the words are to a listener. Quality is how good the sound is. These are typically

mutually exclusive meaning when one is increased the other is often decreased. This can be minimized to some degree by using multiple microphones. The field of speech enhancement consists of three main

fields; noise reduction, signal separation and dereverberation.

7.1.3.1. Noise Reduction

Noise reduction is the process of removing ambient noise from a signal. This can be broken into two

main fields; parametric and nonparametric algorithms.

Page 18: Google Glass Project

9

7.1.3.1.1. Parametric

Parametric methods tend to be classified by the fact that they model the speech signal as an autoregressive

(AR) process embedded in Gaussian noise [12]. Parametric techniques use the following steps; they

estimate the coefficients of the AR and noise variances and then apply the Kalman filter to estimate the

clean speech.

7.1.3.1.2. Nonparametric

The leading types of nonparametric denoising techniques are spectral subtraction and signal subspace

decomposition.

Spectral Subtraction

Spectral subtraction is the most commonly used method in real-world applications. It does have the

drawback of introducing artifacts known as musical noise [12].

There are different appoaches of spectral subtraction method for Enhancing the Speech Signal in Noisy

Environments. The introduction of a5oaches is as follows:

1. Basic spectral subtractions algorithm

Assumptions- Noise is additive and its spectrum doesn’t change with time

Principle- Estimate and update the noise spectrum when speech signal is not clear and highly

influenced by noise.

Advantage- Simple and easy to implement.

2. Shortcomings of S.S.

Principle- This Algorithm Lead to negative values, resulting from differences between the

estimated noise frame and actual noise frame.

Advantage- Accuracy.

Disadvantage-Too sensitive. Hard to implement, if subtracted too little, the much noise will

remain.

3. Spectral subtraction with over subtraction

Principle-This method are subtracting an overestimate of the noise power spectrum and prevent

it going below to the minimum spectrum.

Advantage-this mainly focused on lowering the musical noise effect.

4. Non-linear spectral subtraction (NSS)

Principle- NSS is a method by modifying the over subtraction factor dependent on subtraction

process non-linear.

Page 19: Google Glass Project

10

Advantage- Since the subtraction factor is various, it can be used in different types of noise.

Disadvantage- This method effect low frequency region more than high frequency.

5. Multi band spectral subtraction (MBSS)

Principle- In MBSS approach, the speech spectrum is divided into different bands and each band

has different subtraction coefficient.

Advantage- Accuracy and easy to operate.

6. MMSE Spectral Subtraction Algorithm

Principle- A method for selecting the best subtractive parameters then apply it.

Advantage- Fast

Disadvantage- May subtract the useful sound.

7. Selective spectral subtraction algorithm

Principles- Due to the spectral differences between vowels and consonant. This method proposed

to treat the voiced and unvoiced segment differently than separates the sound into two band

spectral subtraction.

Advantage- Treating voiced and unvoiced segments differently can make a substantial

improvements in performance

Disadvantage- The accurate and reliable voiced and unvoiced decisions particular at low SNR

(Signal-to-noise ratio) conditions cannot be guaranteed.

8. Spectral subtraction based on perceptual properties

Principles- This method is doing expectation based on the perception of human beings.

Consider the perceptual properties of the auditory system.

Disadvantage: More ideal.

These methods would be a good reference for our filter software’s selection. The main goal of the

approaches are add a subtraction factor which related to Laplace transform and Fourier transform.

Page 20: Google Glass Project

11

7.1.3.1.3. Signal Subspace Decomposition

Signal subspace decomposition is the process of decomposing the vector space of the noisy signal into

two orthogonal subspaces; the signal plus the noise and the noise [12]. After which, the clean speech can

be modeled. By applying the Wiener filter the musical noise can be mitigated [12].

7.1.3.1.4. Dereverberation

Dereverberation is the process of removing reverberations from a signal source. Reverberation is the

noise generated by signal source bouncing off of objects on the way to the listening device and reaching

the listening device at different times, resulting in echoes and spectral distortions [12].

7.1.3.1.5. Blind Source Separation (BSS)

Blind source separation is the process of separating the signals in a recording without prior knowledge of

the signals attributes.

7.1.3.1.6. Acoustic Beamforming

Acoustic beamforming is a method for locating the source of noise using a microphone array in a far field

[13]. A far field is defined as having a distance from the signal source greater than that of the microphone

array. Is measured in spatial resolution and dynamic range. Spatial resolution is the minimum distance

two signal sources can be placed and still be distinguishable from one another. Dynamic Range is

difference in sound level between the sound source and its environment.

Advantages

Size of the microphone array can be smaller than the signal source

Fast

Disadvantages

Only typically good for frequencies over 1kHz

Cannot be used to find the intensity of sources, therefore cannot do ranking

Tends to use a lot of microphones

Recommendation

Although it only works above a certain frequency it could still come into use. It also does not require a

large microphone array, which is an additional benefit. It may be part of a future upgrade of the system,

which would allow the wearer to be shown who is speaking.

7.1.3.1.7. Near-Field Acoustic Holography (NAH)

Similar to acoustic beam forming, but is used in the near field. Near field is defined as being closer than

one or two wavelengths of the highest frequency [13]. It also calculates the sound pressure of the area.

The microphone array needs to be the same size as the object generating the sound. The microphone

arrangement determines the maximum frequency of interest and the granularity of the signal.

Page 21: Google Glass Project

12

Advantages

Calculates sound pressure.

Is good over a variety of frequency ranges.

Disadvantages

Must be very close to the sound source.

Requires that the array be the dimensions as the sound source.

Recommendation

Due to the microphones need to be extremely close to the sound source and for the microphone array

shape to match the source, it would be impractical to use it for the google glass project.

7.1.3.1.8. Spherical Beamforming

Spherical beamforming is a technique used for locating sound sources within a cavity, such as a car

interior [14]. It requires a specialized microphone array, typically a 3D acoustic camera.

Recommendation

Due to needing a specialized microphone and that it can only be used in cavities, it has been ruled out for

this project.

7.1.4. Filtering

7.1.4.1. Definition of a Filter

A Filter is any medium through which a sound signal is passed, regardless of whether the signal is in a

digital, electrical or physical form. [15] A physical filter uses the physical shape to alter the signal; an

example of this is the human mouth forming words. An electronic filter uses electrical hardware, such as

resistors and circuits. A digital filter uses an algorithm in code to alter a digital signal. Here the focus is

put on digital filters due to size limitations of the project.

7.1.4.2. Major Categories of Digital Filters

7.1.4.2.1. Time Domain vs. Frequency Domain

These Filters operate in the domain that their name implies, time domain filters operate in the time

domain and frequency domain filters operate in the frequency domain. The time domain filters utilize the

difference equation [16].

𝑦(𝑚) = ∑ 𝑎𝑘𝑦(𝑚 − 𝑘)𝑁𝑘=1 + ∑ 𝑏𝑘𝑥(𝑚 − 𝑘)𝑀

𝑘=0 (1)

Page 22: Google Glass Project

13

In the above equation ‘y’ is the output, ‘x’ is the input, and ‘m’ is the position or index of the signal being

fed into the equation. The coefficients ‘ak’ and ‘bk’ are dependent on the type of filter and are based on

analog filters [17]. The filter order is the larger of the N or M from the difference equation [16].

Since the audio signal is in the time domain when captured, it requires performing a Fourier transform

(FT) and inverse Fourier transform to convert the signal from time to frequency domain for filtering and

then back to the time domain for the speech to text. The most common method found to do this is a Fast

Fourier Transform (FFT). FFT is a method of performing a FT that reduces the number of calculation

from 2N^2 to 2N Log N [18].

7.1.4.2.2. Infinite vs. Finite Impulse Response

When operating in the time domain and using the difference equation, another choice arises: Infinite

Impulse Response (IIR) or Finite Impulse Response (FIR). The main difference, mathematically, is that

an IIR takes into account the previously calculated values as it processes the data. FIR, on the other hand,

does not. Conceptually, this means that once there is an impulse being passed through an IIR filter, the

filter generates an infinite response with an exponential decay that never truly goes to zero [16]. From a

programming perspective, an IIR requires fewer coefficients to achieve the same user defined frequency

response as the FIR filter. The lower the number of coefficients means lower storage requirements, faster

processing and a higher throughput, but an IIR is more likely to become unstable [16].

7.1.4.2.3. Time-Invariant vs. Time-varying

Time-Invariant filter coefficients do not change with time [16]. Where the Time varying does the opposite

and requires more complex calculation for determining the coefficients over time.

7.1.4.2.4. Linear vs. Non-Linear

Linear filters response is the weighted sum of the filter's individual responses. Non-Linear is as its name

implies, and is not a linear combination of responses and like time-varying filters require more complex

mathematics to derive.

7.1.4.3. Common Types of Digital Filters

For all filters the bandwidth or frequency being targeted cannot be greater than half of the sample rate,

this limiting frequency is called the Nysquist frequency [15]. A frequency of between 5 Hz to 3.7 Hz has

been proven to be ample for speech recognition [19], but some researchers/companies go up to 8 kHz.

This means that the minimum recording sampling frequency must be 8 kHz for the 3.7 kHz and 16 kHz

for the 8 kHz, with recording at a sampling frequency of 8 kHz being most common.

Page 23: Google Glass Project

14

7.1.4.3.1. Band Filters

Band filters either allow or prevent frequencies (depending on how they are designed) within a given

bandwidth from passing through the filter [15]. The most common types of these are: Low-Pass, High-

Pass, and Band-Stop. The Low-Pass Filter (LPF) eliminates frequencies greater than the prescribed

frequency. Based on the needed frequencies for speech recognition the prescribed frequency would either

be 4 kHz to 8 kHz. High-Pass Filters (HPF) eliminate frequency less than a prescribed frequency, approx.

5 Hz. Band-Pass filters (BPF) are a combination of the LPF and HPF allowing frequencies within given

ranges or bands to pass. Although an ideal filter eliminates all the undesired frequencies, real filters can

only attenuate the frequencies. A notch filter is a BPF that has a narrow stop band, or band that it is

rejecting.

7.1.4.3.2. Methods of Implementing Band Filters

Band filters come in several flavors, the most common of which are the Butterworth, Bessel and

Chebyshev [20]. These all accomplish the goal of filtering out a band/s, but use different algorithms to

accomplish it, each with their own pro’s and con’s. The Butterworth filter (Figure 7) does not introduce

any major ripples (areas in the band-pass that are attenuated due to the algorithm), but is slow to attenuate

the signal in the band-stop region, in digital signal processing it is said to have a slow roll-off. The Bessel

filters (Figure 8) also do not introduce any major ripples, but begin the roll-off well before the cut-off

frequency. This would cause the desired frequencies in that region to be attenuated, which may result in

missing some parts of the speech. The Chebyshev filter (Figure 9) has a sharp roll-off, but introduces

fairly significant ripples in the band-pass region. Similar to the Bessel filter, this could cause some of the

speech to be distorted or too weak for the speech recognition to work. The Type II Chebyshev eliminates

this, but does not efficiently attenuate the signal after the roll-off region.

Page 24: Google Glass Project

15

Figure 7: Butterworth Filter [20]

Figure 8: Bessel Filter [20]

Page 25: Google Glass Project

16

Figure 9: Chebyshev Filter [20]

7.1.4.3.3. Other Types of Filters

Bell filters, also called peaking filters, allow all bands to pass, but boast or attenuate frequencies around a

user chosen center frequency [21]. The result of applying the filter when seen on a dB vs. Frequency

graph is reminiscent of the classic “Bell Curve” seen in statistics. A Low-Shelf filter passes all frequency,

similar to the bell filter, but also boasts or attenuates the frequencies below the shelf frequency by a user

specified amount [22]. A High-Shelf filter does the same thing, but above the shelf.

7.1.5. Automatic Gain Control

Another method of manipulating a signal is by simply adjusting its gain. Adjusting the gain can increase

the signal strength of a weak signal or decrease the signal of a strong signal. A very simple way of doing

this digitally is by multiplying the signal by a scalar. A positive scalar will increase the strength and a

negative will decrease it. Making this process automated makes an automatic gain filter (AGC) [23]. An

AGC will automatically adjust the level of the signal to a user defined level. To do this the signal is

sampled finding the power of the signal, and then a scalar is calculated to change the signals strength to

the desired level. This comes in very handy in speech recognition since different speakers may speak at

different sound levels or from different distances. Incorporation of an AGC could potentially increase the

effective range of a speech recognition application.

7.2. Google Glass Alternatives

7.2.1. Vuzix M100 Smart Glasses

Designed to model a bluetooth headset, this display is meant to integrate with a smartphone and show

visual updates on Facebook, email, text messages, and much more. It cost $1,000, but is available right

now. This device is very similar to the Google Glass except in the shape of a headset instead of glasses

[24].

Page 26: Google Glass Project

17

Figure 10: Vuzix M100 Smart Glasses [24]

7.2.2. Meta 3D Space Glasses

These glasses are designed to make a 3D virtual overlay on the user’s environment. It uses a 3D graphics software called Unity 3D to program all of the applications it uses which right now includes chess,

Minecraft, and a 3D graphic design program [25]. It costs $667 which makes it much cheaper than the

Google Glass, and it ships in January; however, it has several drawbacks. First off, being goggles, it is not

very discrete in that it covers the whole front of the face. Also, for the primary application of this project

all of the 3D graphics are unnecessary. It only needs to be able to display text.

Figure 11: Meta 3D Space Glasses [25]

7.2.3. GlassUp

Much simpler than the Meta Space Glasses, GlassUp displays only text and basic diagrams on the screen

through the uses of a small projector. It is also much cheaper and only costs $400; however, it is supposed

to ship in February 2014, but this project might not be able to get one until July 2014 due to the

constraints of the GlassUp indiegogo campaign, which is far too late [26].

Page 27: Google Glass Project

18

Figure 12: GlassUp [26]

7.2.4. SmartWatch/ Pebble/ Galaxy Gear

The Smartwatch works as the accessory of the smartphone. The main functions of Smartwatch are

notifications (calls, messages, and reminders), music control, fitting application (distance, speed, and

calorie) and safety application (track the missing phone). In the market, we considered three Smart

Watches as alternatives: Pebble, Sony SmartWatch and Samsung Galaxy Gear.

Figure 13: SmartWatch, Pepple, Galaxy Gear (left to right)

Table 1: Smart Watch Comparison

Product

Name

Price

(USD)

Platform

Compatibility

Input

Types

Dimensions

(mm)

Display (inch) Weight Battery

Life

Pebble [27] 150 iOS/Android buttons 52×36×11.5 1.26×144×168 38g 5~7

days

Smart

Watch [28]

200 Android Multi-

touch

36×36×8 1.6×220×176

OLED display

41.5g 1~7

days

Galaxy

Gear [29] 300 Samsung

Galaxy Note

III

Multi-

touch

55.9×38.1×10.2 1.63×320×320

Super

AMOLED

display

73.5g 1 day

Page 28: Google Glass Project

19

As alternative system for Google Glass, Smart Watches are not a preferable choice: their displays are so

small that few words can be shown; they don’t have operating systems to run the software we need; users

don’t like staring at the watch while conversing.

7.3. Patent Search

7.3.1. Patent 1

Title: Captioning glasses [30]

Patent Number: US6005536 A

Publication date: Dec 21, 1999

Key Features:

1. Large Over the frame housing for projection device. Utilizing A projector bounced off a 45

degree mirror onto a partially reflective beam splitter.

2. Can only be mounted over one eye

3. Compatible with prescription lenses if integrated into the mount initially (made to order)

4. Display of subtitles for media or from dictation software (unspecified)

5. No mention of microphones.

7.3.2. Patent 2

Title: Translating eyeglasses [31]

Patent Number: US20020158816 A1

Publication date: Oct 31, 2002

Key Features:

1. Uses a “plurality of microphones” as little as two, specifying that they are all

omnidirectional.

2. Microphones are integrated into the glasses frame itself or can be mounted on the outside.

3. Utilizes a means of directionally filtering received sound using a “sound localization

algorithm” (sources noted)

4. Filtering software: SPHINX, or ASR.

5. Translator: SYSTRAN

6. Signal generator: Using either a prism system or screen

7.3.3. Patent 3

Title: Assistive device for converting an audio signal into a visual representation [32]

Patent Number: WO2013050749 A1

Page 29: Google Glass Project

20

Publication date: Apr 11, 2013

Key Features:

1. Specifies at least one receiver, more specifically two microphones; one omnidirectional

and one single direction, used for source localization/ambient noise filtering.

2. Signal processing unit: Mobile phone (unspecified)

3. Converter: Speech recognition (unspecified)

4. Projection: Embedded Grating Structure, with an optical coating adjusted to specific band

width

5. Small projector in the arm or mounted next to the temple

6. Can modify existing prescription glasses

7.3.4. Effects on the project

These patents describe devices similar to the one this project describes. The key points that make this

project different from the patents stated above are that; this project describes modifications made to

existing devices to allow for the ability to turn speech to text. The device itself would be working under

its own patents and any “new” or “revolutionary” technique that is come up within the project would have

to be patented. As it stands, the processes that this project is using falls under common application of the

art and will not have to specifically patented. The result, if sufficiently different from the above patents,

specifically the facial recognition interaction, will have to be patented. The project has utilized these

documents as references for some of the design schemes mainly US20020158816 A1 Translating Glasses,

and similarities to the microphone setup from WO2013050749 A1. In the future, the project will also be

looking into translating capabilities much like Patent 2. However the final design will not be specifically

attributed to any one individual patent that was previously filed.

7.4. Current State of the Art Currently there is only one known project that is attempting to complete the same project. That is the

SpraakZien (SpeechView in English) being developed by Leiden University Center for Linguistics [33].

Similar to the Google Glass project it is still in its initial phases and does not currently offer much

information besides saying that they are using an augmented reality glasses to display text that a computer

running dictation software is translating. In their initial news release on their website they state that they

are currently wireless and are looking to use a mobile phone for the computing in their next phase. They

were scheduled to release another paper December 2013 [33] but as of yet no paper has been found.

Another project that is similar to the Google Glass project is “Project Glass: Real Time

Translation...Inspired” which was done by Will Powell in the United Kingdom [34]. He created a near

real-time translation system using augmented reality glasses, mobile phones and Raspberry Pies. His

system is extremely accurate but takes around 10 seconds to process. It also requires both of the

individuals in the conversation to have microphones. The system also has limited portability due to the

requirements of the augmented reality glasses that he used (Vuzix 1200 Star glasses) which use a wired

connection to the computing platform. As it currently stands, there does not seem to be any indication that

he will be attempting to increase the portable of the system, or use it aid the hearing impaired. He was

contacted in November through his website email and did not respond.

The last system that could be seen as similar to this project is Sony’s Entertainment Access Glasses [35].

These glasses are currently being sold to movie theaters equipped with Sony’s projectors and show

subtitles for the movie the wearer is watching. It is completely wireless and uses a special holographic

method for displaying the text. The main disadvantage of the glasses is that it does not use real time

dictation to generate the subtitles, but instead the subtitles are pre-programmed for the movies.

Page 30: Google Glass Project

21

7.5. Reverse Engineering Currently there is only one system, Will Powell’s system, which has released enough information to be

useful for reverse engineering. From what he released on his website it gave the group the idea to look

into using the Raspberry Pi as a computing platform and gave some ideas as to what dictation software to

use. It is recommended that when Leiden University releases more information on their project that it

should be evaluated to see if any additional information can be gleaned off of it.

Page 31: Google Glass Project

22

8. RECOMMENDED SOLUTION

8.1. Solution Principles In order to get a better understanding of the project and what needs to be done a system level functional

flow block diagram was implemented (Figure 14).

From this the team held a brainstorming sessions on how each block would be addressed. Most of the

blocks, like capture audio, had limited options when it came to which device to use. Other blocks, like

filter audio, had many options that can work in series with one another. The final concept that was

selected is presented in Figure 15.

Figure 14: Functional Flow Block Diagram

Page 32: Google Glass Project

23

This lead to the group breaking the group into subgroups to address the different blocks.

8.2. Analysis

8.2.1. Requirements

Before choosing a microphone for this application, a set of numerical requirements needed to be

developed. The main criteria needed are; maximum sound pressure level, frequency range, sensitivity,

size, and weight.

For the maximum sound pressure level, since it is not as critical of a specification for this project, a

general range was estimated based on some comparisons to real world applications. A normal

conversation at three feet is approximately 60 dB of sound intensity, whereas a jackhammer or propeller

aircraft is around 120 dB [36]. An example of something in between would be a heavy machine shop or a

subway train at about 100 dB [36]. Using these numbers for comparison, the maximum sound pressure

level for this application was chosen to be a5oximately 100 dB to avoid audio clipping and still capture all

the necessary sound while disregarding the extremely loud sounds that can be assumed as not part of the

conversation.

The frequency response range was chosen based on the minimum and maximum frequency range for

human speech. Humans can hear sounds in the range of 20Hz to 20 kHz [36]; however, human speech is a

little more limited in range. The most fundamental range for human speech intelligibility is from 300Hz

to 3.4 kHz [36]. Although this range is ideal for capturing human speech, the ranges for singing and their

harmonics are from 100Hz to 16 kHz [37]. By using this larger frequency band, the system is not limited

Figure 15: Final Concept

Page 33: Google Glass Project

24

to just human speech which greatly increases its future expandability. The firm requirement is a minimum

frequency response of 300Hz to 3.4 kHz, but anything larger than that would be preferable.

The only real requirement of the microphone based on sensitivity is that the microphone is sensitive

enough to at least pick up the sound from the required distance. To get some reasonable numbers for the

sensitivity required by the microphone, the sound level of a conversation at 15 ft. needed to be found.

Using the relationship that the sound level of a source decreases by 6 dB for a doubling of distance [36]

and that the sound level of normal conversation at 3 ft. is 60 dB [36], the sound pressure level at 15 ft.

should be about 45 dB.

60 𝑑𝐵 − 6 𝑑𝐵 ∗ 15 𝑓𝑡 / 3 𝑓𝑡 ∗ ½ = 45 𝑑𝐵

This number gives ballpark estimate of the sensitivity value required by the microphone as -45 dB. Since

the sensitivity values are fairly arbitrary, a range around -40 dB was chosen to be conservative. The

weight and dimensional requirements were estimates based on the dimensions of the Google Glass itself.

It was decided, that the microphone should not weigh more than the weight of the glasses. The weight of

the Google Glass is 50g and therefore the weight of the microphone should not exceed 50g. As for the

dimensions of the microphone, the group looked at the dimensions of the Google Glass and approximated

a nominal size (approximately 3in x ¾ in x ¾ in) to be attached to the outside of the glasses without

interfering with the total design.

Table 2: Microphone Requirements

Figure 16: Approximate frequency ranges [37]

Page 34: Google Glass Project

25

Requirement Value

Minimum Frequency Response Range 300 Hz - 3.4 kHz

Max Sound Pressure Level ~100 dB

Max Dimensions 3in x ¾ in x ¾ in

Max Weight 50g

Sensitivity -38 to -42 dB

8.3. FMEA A key aspect of refining ideas is trying to predict future problems that may arise that will need to be

addressed. To accomplish this, the team deployed a failure modes effects analysis (FMEA) to determine

future problem areas. A list of components that are currently known and ways they can fail were

generated. Complete failure tags were given to items that are key to the system working which if that item

were to fail the complete system would also fail. Component failures are given to items that are optional

or extra to the system that do not affect the over system performance. The chance of occurrence (10 being

extremely likely to occur), the severity (10 being extremely severe) and chance of detection (10 being

unable to be detected) were evaluated for each of the failure modes. Since there is no real threat of bodily

harm in this project high severity means that the item is tied to the critical items in the system and could

not be quickly fixed by the user. If when the product of the occurrence, severity and detection was 200 or

greater, then there would need to be an action to remedy the problem. Although these problems were not

addressed, if the project were to further developed for the consumer they would be addressed.

Page 35: Google Glass Project

26

Table 3: FMEA

FMEA

Google Glass Project Group: Google Glass Group (G^3)

O: Chance for Occurrence; S: Seriousness, D: Chance of Detection

Function/Item Potential Failure Potential Reason

for Failure Effects of

Potential Failure O S D Score Action

Microphones Short Out

Exposure to Water Complete Failure 5 9 5 225

Water Proof Microphones

Power Surge Complete Failure 1 9 10 90

Break Being Dropped Complete Failure 1 5 10 50

Sound Filter Data Corruption

Virus Component Failure 1 5 10 50

Program Bug Faulty Programming

Component Failure 6 5 5 150

Dictation Software

Data Corruption Virus Complete Failure 1 9 10 90

Program Bug Faulty Programming Complete Failure 6 6 5 180

Display Short out

Exposure to Water Complete Failure 5 9 5 225

Build waterproofing enclosure

Power Surge Complete Failure 1 9 10 90

Break object being dropped Complete Failure 2 9 10 180

Data Storage Data Corruption

Exposure to magnets

Component Failure 1 5 10 50

Unmounted during writing

Component Failure 4 5 5 100

Break Being Dropped

Component Failure 1 5 10 50

Computing Platform

Break Overheating Complete Failure 4 8 7 224 Incorporate Heat Sink

Being Dropped Complete Failure 1 8 10 80

Short Out

Power Surge Complete Failure 1 9 10 90

Exposure to Water Complete Failure 5 9 5 225

Water Proof/resistant enclosure

Page 36: Google Glass Project

27

8.4. Decisions

8.4.1. Platform Decision

When it came to selecting a computing platform to run the dictation and filtering software on, a

preliminary search of available and reasonable platforms was generated: laptop, tablet, cell phone,

Raspberry Pi. These platforms were then evaluated using a decision matrix (on a 1-5 scale with 5 being

the best) based on criteria that were derived by our system requirements and what the system needed to

do. The criteria are as follows:

8.4.1.1. Portability

Portability is defined here as the overall dimensions, weight and battery life of the platform. The

Raspberry Pi and cell phone received 5’s in this category due to their small size and being light weight. Both also had all day battery life, the Raspberry Pi requires the purchase of a battery pack

though. The tablet got a 3, because depending on the type they are typically have the same size display as

a small laptop, but are fairly light and have good battery life. The laptop received a 2, because compared

to the other devices they are heavy and have poor battery life.

8.4.1.2. Cost

Cost is defined here as the complete cost of the platform. Due to the cost of purchasing a pair of

augmented reality glasses or similar display device, the budget for the computing platform is tighter. The

Raspberry Pi received a 5 because it has a low cost (under $100). Cell phones also received a 5 due to the

fact that most potential users already owning a smartphone and due to having a cost ranging from free to

a5oximately $600 depending on whether the phone was bought through a carrier. Tablets received a 3 due

to its purchase price ($200-$1000 depending on brand and specifications). Laptops received a 1 do to

their high cost ($300+) for a decent laptop.

8.4.1.3. Programmable

Programmable is defined as the platforms ability to either run the required dictation and filtering software

natively or to be able to program applications for the platform capable of doing it. Laptops got a 5 in this

category due to their native ability to support programming and software and can use a wide array of

operating systems. The RPi got a 4 because of its native ability to support programming and software, but

had fewer options in operating systems. Tablets and Cell phones both received 4’s for similar reasons to

the RPi.

8.4.1.4. Display

Display here refers to the ability of the platform to display to an external screen. The Laptop and RPi both

received 5’s because they natively support wired video out options and can use software to wireless

output video. Cell phones and tablets received 4’s because they do support wired video out options, but

typically require special adapters. They also support wireless video output, but have few options of what

they can display to.

Page 37: Google Glass Project

28

8.4.1.5. Microphone

Microphone refers to the ability of the platform to have a microphone connected to it. Laptops received a

5 since it has the ability to accept both USB and 3.5 mm microphones without the need for converters.

The Raspberry Pi received a 4 because it can accept USB microphones without converters and can accept

3.5 mm through the use of an external sound card. Cell phones and tablets received 3’s because they can

accept microphones, but require special microphones or require knowledge of programming to use non-

specialty microphones.

8.4.1.6. Evaluation

Table 4: Platform Decision Matrix

Platform Decision Matrix

Platform

Tablet Cell Phone Laptop Raspberry

Pi

Crit

eria

Portability 3 5 2 5

Cost 3 5 1 5

Programmable 4 4 5 4

Display 4 4 5 5

Microphone 3 3 5 4

Total 17 21 18 23

From the decision matrix the top two devices (RPi and Cell Phone) were selected for further tests. After

developing a working speech to text app for the RPi, it proved to have insufficient processing power for

the project and was abandoned; this is discussed in greater depth in Problems with Using RPi.

8.4.2. Display Platform Decision

8.4.2.1. Criteria

When it came to selecting a display device the programming team got together to brainstorm criteria. The

main ones were:

Available Now

o The AR glasses are fairly new to the consumer market there are few of them that are

either currently on the market or will soon be on the market. Since this is a time

sensitive project the team decided that the AR glasses selected would need to be “in-

hand” by the beginning of the spring semester. This criteria greatly reduced the

amount of options

Developer Support/Base

o To aid in the development of the program the display, it had to have a strong

developer support and developer base to ensure that there would be enough

documentation available to learn how to program for the selected device.

Page 38: Google Glass Project

29

8.4.2.2. Devices Reviewed

For this section the following format will be used:

Name of Device, Manufacturer

Cost (if known)

Main Attribute

Why it was ruled out

Manufacturer’s Website

At the time that the survey of devices was done, the following devices were found.

Jet, Recon

o $599

o Built for sports enthusiast (theoretically very durable)

o No concrete release date, the manufacturer only said “spring 2014”

o http://www.reconinstruments.com

Epiphany Eyewear, Epiphany Eyewear

o $399

o Only used for recording video

o No textual display

o http://www.epiphanyeyewear.com

GlassUp, GlassUp

o $299

o Simple design

o No official release date

o http://www.glassup.net

Telepathy One, Telepathy

o Unknown

o Smallest form factor of all options

o No official release date

o http://tele-pathy.us

Meta 0.1, Meta

o $667

o Holographic overlay

o No official release data

o https://www.spaceglasses.com

ION Glasses, ION Glasses

o $100

o Indicates when messages are received

o No textual display

o http://www.ionglasses.com

ORA-S, Optinvent

Page 39: Google Glass Project

30

o $949

o Bright see-through display

o No concrete release date (Sometime in March)

o http://optinvent.com

Oculus Rift, Oculus VR

o $300

o Complete VR system for computer gaming

o Pure virtual reality, no ability to see outside world

o http://www.oculusvr.com

SmartGoggles, Sensics

o Unknown

o Complete VR system for entertainment

o Currently only available to governments and manufacturers

o http://smartgoggles.net

M100, Vuzix

o $1000

o Compact size

o Does not have the developer documentation/base that the Google Glass has

o http://www.vuzix.com/consumer/products_m100.html

8.4.2.3. Ideal Selection

If the time constraint was not a factor the Ideal choice was the Meta for this project, because it would

have increased the user friendliness by having the words appear on or near the individual speaking them

limiting eye fatigue of the user.

8.4.2.4. Device Selected

The team decided to still go with the Google Glass because it is one of only two options that are currently

available with an adequate display. Also, due to its popularity and its manufacturer, it has strong

developer support. If the team does not get the Google Glass, then the team will purchase a Vuzix M100,

since it is currently the only other device capable of displaying.

8.4.2.5. Computing Platform Refinement

The use of the Google Glass or Vuzix M100 for the display limited the computing device to any bluetooth

enabled phones [38]. Currently there are four main operating systems for smart phones: Android, iOS,

Windows, and Blackberry OS. The team limited the selection to the most common phone OSs: Android

and iOS [39] [40]. Both OSs have plenty of documentation and support, but Google offers a Glass SDK

add-on for their Android SDK [41]. Similar support for the iOS was not found. This lead the team to

select the Android platform.

Page 40: Google Glass Project

31

8.4.3. Microphone Type Decision

The decision on which microphone type the group is going to implement in the project was a major

milestone for the group’s progress. Though some of the decisions of sorting out microphones were easier

than others. For example, while the fiber optic microphones would fit all the criteria for the project,

however it was too expensive. The ribbon microphone was not suitable for mobility and would not make

a good fit if the microphone would have been mounted on the glasses.

The hardest decision was to decide between condenser microphones and the dynamic microphones. Both

of those microphones made it to the final consideration. The speech recognition of a dynamic microphone

is better than the speech recognition of a condenser microphone and for purposes such as focusing on one

person and eliminating surroundings sounds, the dynamic microphones is also better than the condenser

microphone. As it turns out, while the microphone research group looked for suitable microphones, the

majority of microphones found were electret condenser microphones. This lead to the decision that not

only the electret microphones are much cheaper than other microphones, and produced to be included in

mobile electronic devices, such as cell phones, the electret microphones would be the optimal choice of

microphone for this project.

8.4.4. Microphone Final Decision

8.4.4.1. Evaluation

After developing microphone requirements and researching microphones, a list of microphones was

compiled and then compared with these requirements. This list was then narrowed down based on the

requirements to a top four. The results of a decision matrix on the top four are shown below.

Table 5: Microphone Decision Matrix

Criteria Weight SP-MMM-1 MT830R CME-1538-100LB POM-3535L-3-R

Cost 7 4 1 9 10

Sensitivity 5 10 8 7 9

Max pressure 1 7 10 8 9

Size 2 7 8 10 9

Frequency 3 10 10 10 10

Total (unweighted) 38 37 44 47

Total (weighted) 129 103 156 172

According to this matrix the POM-3535L-3-R is the best choice out of these four microphones for this

application. These top four microphones, the original list of microphones, and the requirements developed

in the previous section were all brought to the Electronic Support Lab where David Beavers found similar

microphones and ordered them.

Page 41: Google Glass Project

32

8.4.5. Filter Decision

The decision of what category of filter to use was based on processing time constraints, the group’s

programming ability and the difficulty of implementation. When it came to the decision of frequency or

time domain, it came down to the ease of implementation and the programming skill of the group. For the

frequency domain, doing FFT would be required and android does not currently natively support FFT.

This meant the group would either need to program it themselves or find a source [42] which would be a

hurdle in the programming aspect; it would also increase the code length and processing time due to the

extra steps. Time domain filtering, on the other hand, only required the implementation of a difference

equation and did not require any FFTs. Based off of the research, an IIR filter was found to be quicker

and less expensive computationally than the FIR and was therefore chosen. Based off research, Linear,

time-invariant filters did not introduce new spectral components (artifacts from the filtering process) [15]

and were therefore chosen. For the particular types of filters to implement, the Butterworth Low-Pass

Filter and Low Shelf Filter were selected. The Butterworth LPF was chosen because it did not introduce

any major ripples and had a cleaner (defined as closer to the cut-off frequency) roll-off than the Bessel filter. A low-pass filter was selected since the lower end of the speech range is very narrow and the bulk

of the unwanted frequency would be between the upper speech frequency and the Nysquist limit. The low

shelf filter was selected based off the need to increase the gain of the audio in the speech frequency range

to increase the overall range of the speech recognition.

8.5. QFD In order to determine the functional requirement, their importance and any interdependencies in those

requirements a quality function deployment was used. The four key stakeholder requirements that were

analyzed were: Real time speech to text, mobile, safety and user friendly. The most important of these

was safety, given a 10, since this will be a consumer product and should not harm the user. The next most

important was the real time speech to text since and the product being mobile, both given 9’s, these are

the essence of the project and must be done in order for the project to be deemed a success. The least

important of the requirements was the user friendliness, given a 4. Although being user friendly is

important for any consumer product it did not have the same import as the other requirements. Functional

requirements were sought as to how, from an engineering standpoint, the stakeholder requirements could

be solved and their dependency and correlations were evaluated both with regard to the stakeholder

requirements and to each other. The template that was used did the mathematics to determine which

requirements were more important than others based on their dependency on the stakeholder requirements

and the weight of the stakeholder requirements. The correlations that are in the QFD shows that the

weight and size are negatively impacted by the weather resistance and battery life. This is due to the fact

that in order to increase battery life, the battery must typically also increase in size and weight. The same

is believed for the weather resistance attribute.

The QFD was also used to benchmark the current prototype against the currently known

competitors. Fields that are blank do not have the required information, due to not being release by the

competitor, to make reasonable estimates.

Page 42: Google Glass Project

33

Figure 17: QFD

Page 43: Google Glass Project

34

It can be noted from the QFD that the importance percentage of the requirements are fairly evenly split.

The only exceptions to this are the settings requirements to address being user friendly. Due to the latency

time of Mr. Powell’s system it received low marks in “Real-Time Speech to Text” whereas the team’s app

received high marks for being near instantaneous when connected to a strong data connection. For the

usability Mr. Powell’s and SpeechView both received low marks 2 and 3 respectively. For SpeechView

this is due to its dependency on a laptop for its computing and for Mr. Powell’s system it for the amount

of devices and wires required to work. The Glass received high marks on this due to being completely

housed on the Glass and can use a smartphone for a data connection should Wi-Fi not be present. For

safety the competition did not give enough information to make an informed estimate of their ratings,

however the teams project received high marks again since they only added code which does not interfere

with the Glasses hardware and therefore should be no more of a safety concern than any other Glass app.

Finally for user friendly, both the team’s product and Mr. Powell’s received a median mark since both

present the text in a similar and user friendly way, but do not incorporate more in-depth features. From

this benchmarking it can be seen that the current Glass app is ahead of the current known competition.

8.6. Bill of Material The team divided the budget into the following categories (Table 6) based on the FFBD and Concept.

Table 6: Budget Allocation

Budget Allocation

Category Allotted

Google Glass/Display $1,500

Microphones $500

Software $250

Testing Equipment $100

Printing $100

Computing Platform $250

Cushion Fund @ 10% $300

Total $3,000

From the allotted budget the team purchased a RPi and related items to try as a computing platform, a

microphone to try with an Android phone, and the Google Glass and its associated cables and accessories.

The items and their costs can be found in Table 7 below. The only item that was over budget was the

Google Glass due to its accessories, but this extra cost was absorbed by not needing some of the other

items.

Table 7: Expenses

Page 44: Google Glass Project

35

Expenses

Item Price Category

Microphone $93 Microphone

Raspberry Pi (RPi) $35 Computing Platform

SD Card for RPi $7 Computing Platform

RPi Cables $16 Computing Platform

Google Glass w/Accessories $1,918 Google Glass/Display

Total $2,069

Page 45: Google Glass Project

36

9. PROGRAMMING

9.1. Programming for the Google Glass The Google Glass uses the android operating system which employs Java as its main programming

language. Google is the creator of the android operating system and provides a complete open-source

database of programming functions and development tools on the android developer website [43]. The

android developer website also has instructions on how to begin programming in android and free access

to the Android Development Tools (ADT) which is a plug-in for the Eclipse programming IDE

(integrated development environment) [44]. Google also has a complete online guide to creating

applications for the Google Glass on the Glass Developer website [45].

Three of the main structural components when programming an android application are activities, intents,

and processes. Just like their names suggest, activities are where the user does something, intents get

interpreted by the system and then perform an action, and processes are the execution structure of an

application. Activities have a graphical component or window in which to create an interface that the user

interacts with [46], whereas, intents are entirely behind the scenes calling on the system or other activities

(Figure 18) [47]. Whenever an application begins, it creates a Linux process with one thread of operation

that performs all of the application tasks in series [48]. To perform multiple tasks at the same time,

multiple threads need to be created.

Figure 18: Implicit intent [47]

When using these tools to program for the Google Glass, a programmer follows the same principles as

programming for a mobile device but with a few distinct differences. For instance, the main method for

input on mobile devices is the touch screen. However, the Google Glass does not have a touch screen;

instead it uses a combination of voice commands and touch gestures on a directional pad (D-pad), such as

swiping forward, backward, down, or tapping.

Another way the glass differs from normal mobile development is in its use of the timeline. When starting

the Glass, the first screen visible is the “Ok Glass” menu. This is where the user can start applications

with the voice commands by saying “Ok Glass, <voice trigger>” or touch gestures by tapping and

Page 46: Google Glass Project

37

swiping. When on the “Ok Glass” menu, swiping to the right will allow the user to see past conversations

or other usage history, and swiping to the left brings up current items or settings.

When coding for this type of structure the programmer has several decisions that need to be made. First is

whether to use the Glass Development Kit (GDK) or the Mirror API. Applications created with the GDK

are implemented on the Glass itself and use the Glass hardware. This allows applications to access the

hardware components on the Glass such as the built in microphone or the camera. GDK applications are

also stand alone and do not require access to a network in order to function. The Mirror API functions

quite different from the GDK. None of the code for applications made with the Mirror API runs on the

Glass. Instead, it runs on an external system and then communicates with the Glass to perform its

functions. The benefit of this is that the Mirror API has platform independence and more built in

functionality; however, a network connection is needed [49].

The GDK has several different options for application structure including static cards, live cards, and

immersions. Static cards show up to the right of the “Ok Glass” menu in the past section of the timeline.

This type of interface is not updated and remains unchanged over time. It is mainly used for things like

message updates, search results, or anything else that does not change after its initial function is

completed. The purpose of this project is to add live subtitles to conversation in order to help the hearing

impaired. This requires that the interface is updated continuously, which means that static cards are not a

viable option. Live cards and immersions, however, are updatable but function differently from each

other. Live cards are in the timeline to the left of the “Ok Glass” menu and can be navigated to and from

without closing them or impacting the use of the Glass as a whole. This is accomplished through the use

of a live card service that runs in the background, the results of which get printed to the card on the screen

(Figure 19). A service is similar to an activity except it does not have a visual component and runs

entirely behind the scenes.

Figure 19: Live Card Operation [50]

Live cards come in two different varieties: low frequency and high frequency (Figure 20). The low

frequency cards update on the order of seconds (e.g. 1 Hz) by using a remote view created by the service

and then printing the remote view to the live card. High frequency live cards use drawing logic that

refreshes many times a second (e.g. 60 Hz) instead of a remote view and then prints to a buffer, referred

to as a backing surface, before pushing the results to the card [50].

Page 47: Google Glass Project

38

Figure 20: Low Frequency (Left) and High Frequency (Right) Live Cards [X8]

Live cards do have some limitations, so a different format is needed for some of the more menu intensive and interactive applications. An immersion does not use the timeline; instead, it takes control of the Glass

and must be exited to access the other Glass functions (Figure 21) [51]. This allows for much more in

depth applications and complex functions.

Figure 21: Immersion [51]

9.2. Programming Process The following process was used when coding for the Google Glass.

1. Identify application needs

2. Research coding method

a. Find what functions are needed and where they are implemented

3. Write the code in Eclipse using the ADT

4. Revise and check for errors

5. Debug the code

a. Research the problem if necessary

6. Export the code to the Google Glass

a. Generates an Android Package (APK) that is installed on the Glass (Figure 22)

b. Repeat steps 4-5 as needed

Page 48: Google Glass Project

39

Figure 22: Exporting Code to the Glass [52]

9.3. Generated Applications Initially we tried using a low frequency live card as the structure of the Live Subtitles application, but it

had some limitations. It achieved the basic structure and function of the app, but it could only do single

iterations of the speech recognizer. Also, the WakeLock function that maintains control of the screen

could not be used in the live card. To resolve these issues a second application was created using an

immersion instead of the live card. The immersion solved the issues with the WakeLock and the non-

continuous problem.

9.4. Application Operation: From the main menu say “Ok Glass, Live Subs” to launch the application (Figure 23).

Figure 23: Launching Application

Page 49: Google Glass Project

40

The app launches and begins to listen to speech continuously printing the results to the screen (Figure 24).

Figure 24: App Listening

The user can swipe down on the touchpad to close the speech recognizer and review the conversation

after saying “history mode” in the speech recognizer (Figure 25).

Figure 25: Review of Conversation

The user can swipe down to enter the history mode (Figure 26)

Figure 26: Text History

Page 50: Google Glass Project

41

Tap to return the speech recognition mode

Figure 27: Return to Speech Recognition

Tap to pause or resume the speech recognition mode.

Figure 28: Tap to Enter Speech Recognition Mode

Say “delete file” in Speech recognition mode in order to the clear the text history.

Figure 29: Deleting Files

Page 51: Google Glass Project

42

9.5. Android Smartphone Programming

9.5.1. Speech to Text

The speech to text app was developed to allow testing of some of the speech-to-text and file generation

techniques, methods that are common between the cellphone and Glass. This allowed the programmers to

work in parallel with one another and test preliminary code without needing to be in possession of the

Glass. The code itself was taken from a tutorial on YouTube provided by AndroidDev101 [53] The

original code utilized the “Voice Typing” feature that Android introduced in 2011 [54] to continuously

listen to the speaker for voice commands, translate the voice commands to text, compare the text to

existing commands, and then perform the command’s action. The code was altered to remove the voice

commands and to simply listen for speech, convert that speech to text, and print the text to the screen.

Next a speech history feature was added to the program which continuously wrote the text from the

speech to a file which is stored on the device and can be read by the user on the device or taken off the

device via a USB cable. This was accomplished by using the FileWriter [55] and BufferedWriter [56]

methods that are built into Android. The android phone Speech-to-Text code can be found in Appendix H.

9.5.2. Filter Tester

At the time the filter research and coding began the Glass app had not been completed, so the need arose

to have an app for android that could test the implementation of the filters. From this need the Filter

Tester app was conceived. The app’s purpose was to allow for implementation and testing of filters that

could, in theory, be ported directly to the Google Glass to be implemented into the final Glass app. The

first goal was be able to capture and store audio in a .WAV form. This was accomplished by starting with

the open source code provided by Krishnaraj Varma through their blog [57] and Google Code [58]. Next

the code was modified to include a function for applying filters, the filters were based on the following

sources: Bristown-Johnson [17], Smith [15], and Fisher [20], which were discussed in the filtering

section. In addition to the filters, a simple gain algorithm, a scalar being multiplied by the signal, was

applied. The Filter Tester code can be found in Appendix I. An automatic gain control was coded in

MATLAB and provided in the Appendix J, but was not implemented into the code due to time

constraints. Although the purpose of this app was to be able to port the code to the Glass to be

implemented into the Glass app, the need to record the audio, however briefly, has made it incompatible

with the current speech to text.

9.5.3. Microphone Implementation

To implement an external microphone on Google glass requires 3 conditions to be met:

1. A USB microphone

This is easy to achieve, since USB powered microphones are readily accessible for computers and the

only item needed to connect it would be a USB to micro-USB converter.

2. Android 3.1 above with kernel compiled with USB-host support (Google Glass works on Android

4.0.3 and kernel comes with USB-host support) [59]

To make the microphone work with the Google Glass requires the Glass to act as a USB host, which

needs the Linux kernel to support it when the kernel is compiled. Android OS will also need to support

mounting USB device to the glass. According to the document, android 3.1 and above will do the job.

Google glass comes with Android 4.0.3, thus it also satisfy this condition.

Page 52: Google Glass Project

43

3. Driver of the microphone for android

This is the trickiest part of this topic. To make a hardware work, software is needed that supports it; in

this case, it is the driver. It seems that Google has done some modification on the Android OS on Google

glass [60], which makes driving an unknown USB device very difficult. It is very hard to tell if a specific

model of USB microphone will be supported unless you tried it. If it does not work out of box, heavily

modification to the Linux kernel (which is also the kernel of android) will be needed. This process needs

deep expertise in driver development and it can brick the whole device very easily.

9.6. Programming the RPi Linux is a very common open source OS and Raspbian, a modified form used on the RPi, is gaining

popularity. For this reason documentation on programming is relatively easy to come by. The first hurdle

for using the RPi was getting an USB microphone to work with it. This just required the changing of

some core setting. The next task was to install Carnegie Mellon University's (CMU) PocketSphinx

dictation software and luckily a basic guide was found which provided a step by step procedure. The last

task was to get the RPi to output its results to a remote display. This was accomplished through using

Secure Shell (SSH), a secure data communications program that uses the internet. The RPi natively

supports SSH and this was used to communicate with any other display/device that also can use SSH. A

RPi programming logbook was used to keep track of what was done and what sources were used and is

found in the appendix. Ultimately the RPi was eliminated as a computing platform due to the problem,

which are stated below.

9.6.1. Problems with Using RPi

9.6.1.1. Accuracy

Currently the dictation software uses CMU’s SphinxBase, a speech dictionary used by the dictation

software to know what sound relates to which word. SphinxBase is fairly limited and when the dictation

software hears words that are not in that dictionary it finds the next closest thing. This results is an

inaccurate output. This problem should be fixed by either using a more robust speech dictionary or

creating one through CMU’s SphinxTrain software. Another source of inaccuracy lies in the

settings. These setting would be optimized through trial and error.

9.6.1.2. Latency

The delay between the software hearing the words and outputting the text is also quite significant (often

15-20 seconds). This is a far cry from the project goal of “real-time.” To fix this some core setting of the

program would need to be optimized through trial and error, but due to the processor limitations the

latency will more than likely still be unacceptable.

9.6.1.3. Internet Dependency

Since the current system uses SSH to communicate with a remote display it is dependent on having an

internet connection. Although this does not violate any current requirements, it does limit the mobility of

the system and therefore will be addressed. The easiest remedy for this problem is to use a wired

connection. Other solutions that were under consideration are using Bluetooth and direct wifi.

Page 53: Google Glass Project

44

10. TESTING

10.1. Introduction In the Fall 2013 semester the team conducted 4 experiments: Read Time Research, Audio-to-Displayed

Text, L/R Comparison, and Dictation Software Comparison (Their procedures are found in Appendix X).

In Read Time Research, the team gained knowledge about the proper duration of time the text shown on

the display, which guarantees a good user experience. The team wanted our users to have comfortable

experience in conversing that the way of text showing on the display will not distract them. And the team

also gained the knowledge of how to use the graph to determine the range of true values.

In Audio-to-Displayed Text, we refined our project’s requirement in distance of the ability to capture

audio. It turned out that the initial requirement was difficult to meet with; we had to reduce the distance

requirement or seek solutions. Also, it was realized that the audio-filtering technology would help

promote the accuracy in dictation.

The L/R Comparison is an attempt to test the feasibility of using two microphones to determine a sound

source's location. Which has great significance in helping hearing-impaired people in communications.

The results confirmed our idea that the intensity in two microphones would be different due to the

different angles. The next step is to combine the L/R Comparison and the audio-filtering to promote to

accuracy in dictation performance which is the core part of the project.

The Dictation Software Comparison is to collect empirical data of different dictation software. We faced

many problems in this experiment due to the lack of the ideal apparatus (microphone and non-free

software). But we finished a complete criteria of dictation software which would help us pick out the

optimum software in next semester.

In Spring 2014 semester, the test group focused on test the performance of Google Glass. The test group

have conducted tests below to help other groups to improve their design and propel the project: Audio

Test for Google Glass, Performance of Live Card App and Performance of Immersive Card App. The

purpose of these tests was to test the performance the hardware and the application developed by

programming group. Also, test group would generate recommendation to help other group to promote

their design.

The Audio Test for Google Glass was the first test we conducted. This fundamental test gave the whole

team some basic ideas about the audio performance in Google Glass. The principle we set in this test (like

the criterion of test room, sample test sentences) was adopted in the other tests. From the result, the test

group decided not to use the headphone in later test for its failure in enhancing the audio.

The Performance of Live Card App was to test the performance of application Speech to Text, which

includes the launch time, working range, incremental test and ambient noise threshold. This Live Card

App was the first application used on Google Glass. It had good launch time but could only meet the

basic requirement in other test. Also, it couldn't work continuously which is crucial in real life use. Thus,

the application was required to be improved.

The Performance of Immersion Card App was to test the performance of improved application Live

Subtitles, which mainly focused on working range and ambient noise threshold. The Live Subtitles can

work continuously and has larger working rang and better performance in noisy condition than the Speech

to Text. However, it’s still not perfect. Test group recommended using filtering software or devices to

guarantee the usage in daily life.

Page 54: Google Glass Project

45

10.2. Praat Software For the L/R comparison experiments and future experiments that require the sound data to be analyzed,

the team used Praat. Praat was programmed and developed by Paul Boersma and David Veenink. This

program is designed to analyze the properties of speech and sounds, like the intensity, the pitch and the

spectrum.

10.3. Read Time Research

10.3.1. Results

Figure 30 Subtitle Research

�̅� = 2. 93; 𝜎 = 1.091; 𝑛 = 158; 𝛼 = 0.05

10.3.2. Discussion & Conclusion

From the Figure 30, the rates of the captions in different TV shows, movies and speech have no big

difference (2.4 words per second to 3.5 words per second). We are 95% confident that the true value lies

in the range of 2.76 - 3.1 seconds (2.93 +/- 0.17) from the data in statistics.

10.3.3. Recommendation

Average speed: 3 words/sec, which determine the latency of the text shown on the screen of the

Google Glass.

Amount of the words in caption: 1~16 (1~13 words within 2 lines suggested)

Duration of the caption: 0.98s~7.31s (1s~4.5s suggested)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Wo

rd p

er s

eco

nd

Page 55: Google Glass Project

46

10.4. Audio-to-Displayed Text

10.4.1. Result

Table 8: Audio-to-Display Text Results

distance (ft) total words correct words wrong words total accuracy (%)

0.5 108 103 5 95.4%

2.0 108 99 9 91.6%

2.5 108 88 20 81.5%

3.0 108 60 48 55.6%

3.5 108 40 68 37.0%

4.0 108 29 79 26.9%

Figure 31: Audio-to-Display Results

0

20

40

60

80

100

120

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Acc

ura

cy (

%)

Distance (ft)

Accuracy vs. Distance

Page 56: Google Glass Project

47

10.4.2. Discussion & Conclusion

From Figure 31, as the distance between microphone and speaker increases, the accuracy of the dictation

software drops significantly. In order to keep the accuracy, the distance shall be limited within 2.5 feet.

With this range, the software has the accuracy higher than 80%. Beyond 2.5 feet, the system is no longer

reliable.

10.4.3. Recommendation:

Change the microphone for the system: capable connect with smart phone via Bluetooth and can

capture the audio at a greater distance.

Change the dictation software: the performance of Dictanote is not good. While some dictation

apps on Android work well.

Reduce the audio-capture distance requirement from 15ft to approximately 2.5 ft.

10.5. L/R Comparison

Speaker: Shaopeng Han (voice recorded by phone)

Ambient noise: 30.0 dB

Analysis software: Praat

10.5.1. Result

Distance: 3ft

Figure 32: L/R Comparison Results Chart at 3 ft.

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180

Inte

nsi

ty D

iffe

ren

ce (

dB

)

Angle (Degree)

Corrected result(ABS)

Page 57: Google Glass Project

48

Table 9: L/R Comparison at 3ft

Distance: 3ft

angle

(degree)

intensity in left microphone

(dB)

Intensity in right

microphone (dB)

Difference L/R Correction

factor(based on 90)

Corrected result (ABS)

0 67.13 63.16 3.97 6.29 2.32

10 67.30 64.12 3.18 6.29 3.11

20 67.36 64.85 2.51 6.29 3.78

30 67.60 64.51 3.09 6.29 3.20

40 67.78 65.33 2.45 6.29 3.84

50 68.09 65.98 2.11 6.29 4.18

60 68.24 65.60 2.64 6.29 3.65

70 69.13 64.23 4.90 6.29 1.39

80 69.66 64.93 4.73 6.29 1.56

90 70.57 64.28 6.29 6.29 0.00

100 71.15 64.16 6.99 6.29 0.70

110 70.01 63.50 6.51 6.29 0.22

120 69.16 61.42 7.74 6.29 1.45

130 68.88 60.42 8.46 6.29 2.17

140 68.37 58.31 10.06 6.29 3.77

150 67.96 56.77 11.19 6.29 4.90

160 67.97 56.80 11.17 6.29 4.88

170 65.92 55.14 10.78 6.29 4.49

180 67.39 55.00 12.39 6.29 6.10

Distance: 2 feet

Figure 33: L/R Comparison Results Chart at 2 ft.

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

6.00

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180

Inte

nsi

ty D

iffe

rnce

(d

B)

Angle (Degree)

Corrected result(ABS)

Page 58: Google Glass Project

49

Distance: 2ft

angle

(degree)

intensity in left microphone

(dB)

Intensity in right microphone (dB)

Difference L/R Correction factor

(based on 90) Corrected

result (ABS)

0 53.43 67.83 14.40 9.69 4.71

10 53.13 67.99 14.86 9.69 5.17

20 53.48 67.93 14.45 9.69 4.76

30 53.86 68.24 14.38 9.69 4.69

40 54.15 68.08 13.93 9.69 4.24

50 55.04 68.22 13.18 9.69 3.49

60 55.58 68.42 12.84 9.69 3.15

70 56.65 68.39 11.74 9.69 2.05

80 57.94 68.55 10.61 9.69 0.92

90 59.03 68.72 9.69 9.69 0.00

100 58.84 68.77 9.93 9.69 -0.24

110 59.50 68.95 9.45 9.69 0.24

120 60.01 68.75 8.74 9.69 0.95

130 59.77 68.48 8.71 9.69 0.98

140 59.64 68.47 8.83 9.69 0.86

150 60.19 68.30 8.11 9.69 1.58

160 60.24 68.29 8.05 9.69 1.64

170 60.48 68.11 7.63 9.69 2.06

180 60.19 68.06 7.87 9.69 1.82

10.5.2. Discussion & Conclusion

From Figure 32 and Figure 33, the difference in distance to the 2 microphones varies with the angle: at

the 90 degrees, the distance between the sound source to two microphones are equal. By comparing L/R

intensity difference the location of the sound source could be determined. The closer distance between

sound source and microphone leads to greater intensity shown in the Praat. The difference in intensity

shown in Praat is related to the angle, which should be symmetric about 90 degrees: the intensity

difference reach its peak at 0 and 180 degrees. Before the correction factor is added, the intensity

difference in two microphones is quite different at 90 degree which means these two microphone are not

identical.

10.5.3. Recommendations

If the system needs the L/R comparison to determine the location, the related software must be

determined and shall work on the mobile platform. Try to use two identical microphones for the system to

promote the performance. Intensity difference between L/R microphones is also related with the distance

between the microphone and the sound source.

Table 10: L/R Comparison Results Chart at 2 ft

Page 59: Google Glass Project

50

10.6. Dictation Software Comparison

10.6.1. Discussion & Conclusion

The dictation apps on Android work better than those that work on Chrome. After testing the apps, we got

the data shown in Table 11: Dictation Software. “DictaNote-Speech Recognizer” is the best one of these

apps, because it has high accuracy and can work on Chrome. So we decided to use it to do the Audio-to-

Text experiment.

10.6.2. Recommendation

Dictation and Mail is recommended for the mobile platform, because of its accuracy and the ability to

display the whole text. DictaNote is recommended for the PC platform. This is based on the current

selection of dictation software tested.

10.6.3. Result

Product Name Multilingual Latency

Time(1~3) Accuracy (%)

Reviews by Users(1~5)

Speechpad.pw Select Languages 1 88 4.6

Voice Recognition English 1 91.7 2.75

DictaNote - Speech Recognizer

Select Languages 2 93.5 3.58

Simple Dictation Select Languages 1 85.2 3.38

VoiceNote-speech to text

multi-language 1 89.8 3.7

Dictation and mail multi-language 2 97.2 3.9

Speech to Text Translator TTS

English 2 92.6 4.4

ListNote English 1 95.4 4.4

Table 11: Dictation Software

Page 60: Google Glass Project

51

10.7. Audio Test for Google Glass

10.7.1. Results

Table 12: Microphone Range

Distance (feet) 3ft 6ft 9ft 12ft 15ft

Intensity With

(dB) 60.7 56.0 56.6 52.8 52.6

Intensity

Without (dB) 60.8 57.7 60.0 55.5 54.9

10.7.2. Discussion & Conclusion

From the data in Table 12, the intensity of audio which captured by Google Glass decreases as the

distance increases. Which conforms to our expectation. Comparing the data in table, there is no big

difference in audio intensity when using the headphone or not. It can be concluded that the headphone

cannot enhance the audio. The main function of the headphone is to works as speaker to help the user hear

the sound from Google Glass.

10.7.3. Recommendation

Due to the failure in enhancing audio, the headphone can be ignored in other test.

Performance of Live Card App

10.8. Launch Time of “Speech to Text”

10.8.1. Results

Table 13: Launch Time of “Speech to Text”

1st test 2nd test 3rd test Average

0.28s 0.44s 0.50s 0.41s

10.8.2. Conclusion

From the data in Table 13, the average launch time of Speech to Text 0.41 seconds, which is

excellent. The app works almost instantly after the click or voice commend.

Page 61: Google Glass Project

52

10.9. Working Range of “Speech to Text”

10.9.1. Result

10.9.2. Discussion and Conclusion

From the Figure 34for the simple sample sentence “it's nice to meet you”, the app can recognize it within

5 feet with 100% accuracy. Beyond 5 feet, the app cannot be triggered. This working range meet our

system requirement and is good enough for the initial prototype.

10.9.3. Recommendation

The filtering and sound enhancement technology can be equipped in Google Glass to broaden its working

range.

10.10. Incremental Test for “Speech to Text”

10.10.1. Result

100% 100% 100%

00%

20%

40%

60%

80%

100%

120%

3ft 4ft 5ft 6ft

Acc

ura

cy (

%)

Distance (ft)

Accuracy of "It's nice to meet you"

Accuracy

Figure 34: Speech to Text Accuracy

Page 62: Google Glass Project

53

Table 14: Accuracy of “Speech to Text” in Various Distance and Words

Sentence 1

(5 words)

Sentence 2

(7 words)

Sentence 3

(7 words)

Sentence 4

(13 words)

Sentence 5

(18 words)

Sentence 6

(20 words)

3ft 100% 100% 100% 84.6% 77.8% 45%

4ft 100% 71.4% 57.1% 30.8% / /

5ft 100% 14.3% 57.1% / / /

Figure 35: Accuracy vs. Distance & Words

10.10.2. Discussion & Conclusion

From Figure 35, the accuracy of the app Speech to Text decreases with the increase in words of the

sentences. Also, as the distance increases, the accuracy drops significantly. Because the app cannot work

continuously, the long sentence cannot be recognized completely.

10.10.3. Recommendation

To ensure the accuracy, the user should be close enough (4 feet) with the speaker. Also, the app need to

be improved to work continuously to guarantee the usage in real life.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Sentence 1 (5words)

Sentence 2 (7words)

Sentence 3 (7words)

Sentence 4 (13words)

Sentence 5 (18words)

Sentence 6 (20words)

Acc

ura

cy (

%)

Sentence #

Incremental Test

3 feet 4 feet 5 feet

Page 63: Google Glass Project

54

10.11. Ambient Noise Threshold Test for “Speech to Test”

10.11.1. Result

Table 15: Accuracy of “Speech to Text” in Various Environment & Distance

Music 20% (72dB) 40% (81 dB) 60% (85.2 dB)

3ft 0.67 1 0 0.67 0 0 0 0 0

4ft 0.67 0 0 0 0 0 0 0 0

5ft 0 0 0 0 0 0 0 0 0

Traffic 20% (63.2 dB) 40% (73.8 dB) 60% (79.5 dB)

3ft 1 0 0.67 0 0 0 0 0 0

4ft 0 0 0 0 0 0 0 0 0

5ft 0 0 0 0 0 0 0 0 0

Speech 20% (59.7 dB) 40% (69.7 dB) 60% (79.8 dB)

3ft 0 1 0 0.67 1 0 0.67 0 0.67

4ft 0 0 0 0.5 0 0 0 0 0

5ft 0 0 0 0 0 0 0 0 0

Tapping 20% (46 dB) 40% (52 dB) 60% (73.8 dB)

3ft 1 1 1 1 1 0.67 0 0 0

4ft 0 0 0 0 0 1 0 0 0

5ft 0 0 0 0 0 0 0 0 0

Table 16: Working Range of “Speech to Text” in Different Environment

Location Olin Lounge Outside Olin Restaurant Babcock St

Decibel

Reading

40 – 52 dB 55 – 62 dB 65 – 70 dB 79 – 82 dB

Critical Distant 4’2’’ 3’8’’ 2’10’’ 1’8’’

Page 64: Google Glass Project

55

10.11.2. Discussion & Conclusion

From the data in Table 15 we get the working range shown in Table 16, the performance of app Speech to

Text is very sensitive to the ambient noise. The accuracy drops significantly with the increase in distance

and the intensity of noise. Beyond 4 feet, the accuracy is very low. Also, the performance of the app is

related to the type of noise: the people’s talking reduce the accuracy more obviously than other type of

noise. However, these simulated noise from YouTube video seems too loud that the app cannot work

correctly.

10.11.3. Recommendations

Due to the too loud simulated noise, the test result may not show the real performance of Google Glass.

Thus, the test group need test the decibel reading from different real environment like restaurants, street,

cars, etc. For the app, programming group need improve it by trying using different way of coding.

Page 65: Google Glass Project

56

Performance of Immersion Card App

10.12. Ambient Noise Threshold Test for “Live Subs”

10.12.1. Result

Table 17: Accuracy of “Live Subs” in Various Environment & Distance

Olin Lounge

Distance 3ft 4ft 5ft 5ft 7in

Accuracy 1 0 1 1 1 1 0 0.67 1 1 0 0

Outside Olin

Distance 3ft 4ft 5ft 5ft 2in

Accuracy 1 1 1 1 1 0 0 0.67 1 1 0 0

Restaurant

Distance 3ft 3ft 6in 3ft 7in 3ft 8in

Accuracy 1 0.67 0 0 1 0 1 0 0 0 1 1

Babcock St.

Distance 2ft 10in 2ft 11in 3ft

Accuracy 0 0 0.5 0 0 0 0 0 0.67

Table 18: Working Range of “Live Subs” in Different Environment

Olin lounge Outside Olin lounge Restaurant Babcock St

40 – 52 dB 55 – 62 dB 65 – 70 dB 79 – 82 dB

5’7’’ 5’2’’ 3’8’’ 3’

10.12.2. Discussion and Conclusion

From Table 17, we get the working rang shown in Table 18. The app Live Subtitles has much

larger working range in different environment than the Speech to Test. In the relatively quiet environment,

the app can working within about 5 feet. As long as the app can capture the audio, it has the good

accuracy. With the ability of working continuously, the app is feasible and reliable in daily use. But the

accuracy is poor in noisy environment.

Page 66: Google Glass Project

57

10.12.3. Recommendations

The performance of the app in quiet conditions is better than the noisy conditions. Thus, try to use the

system in quiet conditions. Since the app can meet the system requirement, the team need to optimize the

app, such as ameliorating the User Interface, adding the function of storage the text history.

10.13. Accuracy of “Live Subs”

10.13.1. Results

Table 19: Accuracy of "Live Subs" in Different Environment

Location Publix US 192 Restaurant Food

Court Rest Area Car

Quiet

Room

dB 62.5 68-80 76-81 72-77 70 53-67 40

Accuracy 95% 90% 74% 62% 83% 95% 84%

10.13.2. Discussion and Results

Generally, the app Live Subtitles has good accuracy in different environment. From

Results

Table 19, the app has excellent accuracy in quiet environment. However, the intensity of the ambient

noise is not the only factor to affect the accuracy. The accuracy in quiet room is lower than in the road.

The category of noise and the structure of the building also have effects on the accuracy.

10.13.3. Recommendations

Due to the bad performance in the restaurant and food court, the test group recommend to use the filtering

technology to improve the accuracy in these conditions. Again, try to use the system in quiet conditions.

10.14. Polling

Table 20: Result of Polling

Question Easy to Easy to Comfortable Easy to read Useful? Likely used

Page 67: Google Glass Project

58

navigate the app?

open the app?

to wear? the text? in future?

10 10 10 10 10 5

7 8 3 3 10 10

7 9 9 10 9 10

9 8 8 10 6 10

9 9 7 7 8 9

8 10 8 10 7 9

7 8 5 8 10 10

10 9 8 6 8 9

9 10 8 6 8 10

9 10 10 10 10 10

8 9 8 8 10 10

9 10 8 9 10 10

9 10 8 6 10 10

10 10 10 10 10 10

10 10 5 10 9 10

Average 8.7 9.3 7.7 8.2 9 9.5

.

Page 68: Google Glass Project

59

11. CONCLUSIONS AND FUTURE WORK

11.1. Final State of the Project

Based on the research and programming that took place during the spring semester the final design

evolved to reflect what was feasible given the team’s programming know how, new information attained

and the platforms hardware limitations. The audio storage was eliminated because it was found that the

dictation software used is unable to accept recorded audio and that the dictation software could keep up

with the audio at normal speech speeds. The video capture system, as well as several audio filtering

techniques proved to be overly ambitious due to their inherent complexity and time constraints. However,

if the project were to continue filtering technology is still recommended since it will increase the devices

ability to handle speech to text in environments that have many speakers, something that the current app

is unable to handle. The figure below shows the actual completed system. Note that currently the audio

filters are researched, selected and some designed, but implementing them has proved to be infeasible for

the spring semester due to difficulties with the dictation software not being able to accept the filtered

audio.

Figure 36: Final State and Design

11.2. Major Accomplishments

Through the hard work of the team and advise of its advisors, the team has produced a core program that

has the ability to the receive speech from a speaker, translate it to text and display said text to the Google

Glass display. In addition to this app, the team also produced a comparable Android smartphone app that

has the most of the same features of the Glass app. Future teams can use the smartphone app as a

functioning base, if they wish to switch computing platforms. A simple yet powerful app for testing filters

and auto-gain systems was also created which future teams can use to enhance their own filtering and

auto-gain algorithms. Finally the group laid down firm groundwork for other teams to build on when it

Page 69: Google Glass Project

60

comes to filters, speech enhancement, and microphones by completing all the research and preliminary

design in those categories.

11.3. System Validation

Upon completion of the Live Subs app, system validation experiments were conducted to ensure that all

of the “shall” and most of the “should” requirements were met. For this section, the system requirement

will be represented in a checklist. In this checklist each requirement will have a statement indicating if it

was met or not, how it was validated and what the supporting evidence was that provided proof. Some

items could not be tested due to the possibility of destroying the Google Glass in the process or due to

time constraints. These requirements will have a “TBD” note in the “Met?” column indication they are to

be determined.

All of the “Shall” Requirements were met, and 8 of the 16 “should” requirements were met. The

requirements that were not met proved to be over ambitious for the time and knowledge of programming

that the team possessed. Although these requirements were not met, the current app design does not

prohibit their future implementation.

Page 70: Google Glass Project

61

Table 21: System Validation Checklist

Req.

Group Type

Req.

# Shall? Description Met?

Verification

Method Proof

Real

Time

Speech

to Text

Functional 1 Yes Capture audio human threshold Yes Test Based on ambient noise

threshold and accuracy test

yielding positive results

Functional 2 Yes Capture frequency 300 Hz to 3.4

kHz Yes Test

Based on ambient noise

threshold and accuracy test

yielding positive results

Functional 3 No Filter out ambient No Test Filtering was not implemented

Functional 4 Yes Display audio within 3 seconds Yes Test Based on device returning

audio as it is hearing it

Functional 5 Yes Capture audio nominal distance

of 3ft Yes Test

Based on ambient noise

threshold and accuracy test

results of 3 ft - 5ft

Mobile

Functional 6 Yes Under 8" x 3" x 1" Yes Observation Based on no hardware external

to Glass

Functional 7 No Battery life greater than 4 hrs No Test Based on battery life testing

and group use with an avg.

battery life of 2.5 hours

Functional 8 No Weather Resistant TBD Test Was not tested

Functional 9 Yes Durable enough to withstand

daily use Yes Observation

No formal test, based on

group's continuous usage

through the semester with no

visible wear on the device

Functional 10 No Weigh under 8 oz Yes Test Based on no hardware external

to Glass

Safety

Safety 11 Yes Not exceed 140F Yes Observation Based on device being pre-engineered by Google

Safety 12 Yes No Sharp Edges Yes Observation Based on there being no sharp

edges on the glass

Safety 13 Yes Electrically grounded Yes Observation Based on device being pre-

engineered by Google

Page 71: Google Glass Project

62

Safety 14 Yes Not reuse user's personal info. Yes Observation Based on the device not

utilizing stored data other than

text history feature

User

Friendly

Feel 15 Yes User Friendly Yes Test Average rate points in survey

Feel 16 No Comfortable to wear Yes Test 7.6 points in survey

Feel 17 No User selectable text formatting No Observation No feature present

Feel 18 No Aesthetically pleasing Yes Test Result from the survey

Feel 19 Yes Easy to control Yes Test 9.3 points in survey

Feel 20 No User indicates sound source No Observation No feature present

Feel 21 No Indicate sound level No Test No feature present

Feel 22 No Notify user when audio is

detected Yes Observation

Based on the app having a

feature where the microphone

icon turns red when it is

receiving audio and coming up

with a notification when it

does not

Feel 23 No Able to display to non-Glass

sources Yes Observation

Based on Glasses ability to

screen share with smartphone

via Glass app

Feel 24 No Different setting mode No Observation No feature present

Feel 25 No Indicate which source the sound

is coming from No Test No feature present

Feel 26 No Cold start-up under 3 mins Yes Test Based on Launch Time test

result of under 1 second

Feel 27 No Multi-language capability No Observation No feature present

Feel 28 No Data storage Yes Observation App has text history

Feel 29 No Notify speaker when unable to

keep up No Test No feature present

Page 72: Google Glass Project

63

11.4. Recommendations for Future Work

The team recommends that if another group continues this project that they should consider switching the

computing platform to an Android smartphone and have the text sent to the Glass from the phone. The

future team could use the current smartphone app, which has most of the same features as the Glass app

save for the menu and in app text history. Making this change would increase the microphone and

hardware options that can be implemented and would allow the team to use the latest Android API for the

programing. Both of these additions to the projects arsenal would aid in increasing the range and accuracy

of the app in adverse conditions. The inherent difficulty with this approach is getting the phone to send

data to the Glass. The team recommends using a Bluetooth connection to remain wireless and to attempt

to mitigate any potential lag in the communication time between the two devices. Based on the current

team’s expertise and composition, this could take anywhere from a couple of months to a semester.

Due to time constraints and technical difficulties mentioned in previous sections a filter and audio-gain

code were not implemented into the Glass app. It is recommended that the filter and gain code be

implemented for the future. The gain code would increase the range in environments without noise and, if

coupled, with the filter code, would increase the range and accuracy of the speech to text in environments

with non-speech noise. Both of these items have the preliminary design work done, and would need to be

tweaked based on the results of tests. The inclusion of these two features would expand the uses of the

app.

Although little work was done in the field of speech enhancement outside of the initial research for this

project, it is still recommended for future projects. The advantage that speech enhancement brings to the

app is the ability to handle the presence of multiple speakers, which the filters are unable to address well.

Incorporating robust speech enhancement will be very difficult without the proper expertise in the group

since it is near the cutting edge of the signal processing and speech recognition field. Inclusion of this

attribute will undoubtedly be a modification that will set the project apart from other similar projects.

Similar to the advantages of adding the filter and gain on the software end, adding a microphone that

meets the team-defined specifications (such as the ones purchased for this project) would also increase the

range of the app in clean environments. This was an impossibility for this iteration based on the current

hardware available to the Glass and the current team’s expertise. If, however, the Glass gains popularity

there is a good possibility that a microphone that is compatible with the Glass will be developed, lowering

the programming hurdle to implement it.

In addition to the above hardware and software changes, it is recommended that any future groups be

composed with a multidisciplinary team of mechanical engineers, electrical engineers and computer

science majors. If this cannot be done, then the future team should seek strong advisors that have

expertise in aspects that the group is lacking. Making this change would result in less researching time

and quicker development times.

To increase the functionality of the app it is advised that in the future it will include some of the user

friendliness requirements that were not included in the current Glass app. Most notable of these futures

would be the addition of Multilanguage capability that could handle different languages both for the

current direct speech to text and for translating the incoming language into the user’s native tongue. This

would expand its potential user base dramatically by appealing to people from all over the world as well

as people who are not hearing impaired, but are traveling abroad. A less complex endeavor involving the user friendliness requirements would be to include user settings to allow the user to change font type, size

and other cosmetic items. Although it may seem silly to worry about such a simple issue, users will get

Page 73: Google Glass Project

64

more use and enjoy their experience with the app more if they have more control over it. To achieve some

of the other user friendliness requirements, it may prove necessary to change to a custom-built speech

recognizer. This, similar to the speech enhancement, would be a monumental undertaking, but a future

team could use Carnegie Mellon Universities’ open source Sphinx or PocketSphinx as a base to start off

of.

Page 74: Google Glass Project

65

APPENDICES

Page 75: Google Glass Project

66

Appendix A Meeting Minutes

Date 8/23/2013 Time 2:00 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Set Milestones

Split into groups

Assign Tasks

Generate initial requirements

Points discussed

Discussed meeting with Dr. Morkos and GSAs

Created initial system requirements

Generated Milestones to capture audio and then display it to screen

Action points

Issue Action required Responsible (Who) Deadline (By When)

Microphones Research Microphones

Attributes

Constantin, Brandon 8/30

Programming Research Programming

Methods

Luke 8/30

Dictation Research Dictation

Software

Yanglong Lu 8/30

Google Glass Research Google Glass Han, Cheng Lu 8/30

Open Issues

None.

Page 76: Google Glass Project

67

Date 8/26/2013 Time 4:50 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Explain how the Google Drive works

Establish executive summary procedure

Establish who will be in charge of starting Google Hangout meeting

Discuss Project Statement and Requirements

Points discussed

Explained how to use the Google Drive and where everything was on it.

Kyle was named temporary custodian for the executive summaries

Establish Fridays at 5 pm for informal meetings to answer any question the group has on their tasks.

Established project statement and requirements.

Open Issues

None

Page 77: Google Glass Project

68

Date 8/30/2013 Time 5:00 PM Place Google Hangout

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Try out Google Hangouts

Announce new member

Discuss executive summaries

Points discussed

Group a5oved of Google Hangouts

Announced new member (Darius) who will be joining the group next semester

New method of doing the executive summaries was decided on where each member writes their own

portion that will be combined

Microphone testing was put on temporary hold to work on researching more

Open Issues

None.

Page 78: Google Glass Project

69

Date 9/4/2013 Time 5:30 PM Place Evans Library Room 410

Called by Mills, Brandon Minutes prepared by First M. Last

Attended by

Team Members:

Brandon Mills

Luke Glesener

Yanglong Lu

Constantin Jessen

Shaopeng Han

Cheng Lu

Sponsor (Advisors/Others (Specify)):

None

Agenda:

Review changes to the Google Drive and new Drive apps.

Discuss workshops.

Discuss change in adviser meeting time

Review all project statements and combine elements to create a group project statement

Discuss what makes a good system requirement.

Create a master systems requirements list.

Begin brainstorming solutions.

Points discussed

Record the discussion for each agenda item. Mention who said what, what were the agreements and

disagreements, and how did the issue come to closure (resolved, still open due to disagreement, discarded

because lost relevance, etc.).

ENSURE that for all issues that yield to a future action, entries are made to the Action Points table. This

table will be the reference point for the next meeting where the status of the project will be followed up.

Open Issues

Write the open issues here and why they are open.

Overall, the purpose of a MoM is twofold:

To record all the proceedings of a meeting so that the document can be referred back later when memories

about the specific discussions start to fade.

To assign responsibility to stakeholders (project team, sponsor, client, suppliers) and track their

performance against those commitments.

Page 79: Google Glass Project

70

Date 9/6/2013 Time 5:00 PM Place Google Hangout

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Yanglong Lu

Shaopeng Han

Cheng Lu

Kyle Coleman

Darius Danin

Sponsor (Others (Specify)):

None

Agenda:

Discuss Wednesdays’ workshop.

Assign 2 people for project planning form

Assign 2 people for object/requirement tree

Assign 2 people for requirement checklist

Assign 1-2 people to finalize the project statement

Discuss any questions for Dr. Morkos to be brought up in the executive summary

Next in person meeting will be Monday after the adviser meeting in the library

Points discussed

Luke and Kyle volunteered for doing the objective tree/requirement tree.

Alex and Lu volunteered for the requirements checklist

Brandon and Cheng volunteered for the Project planning.

Brandon volunteered for project statement and assigned Constantin.

For executive summary, the what the group is doing next is brainstorming and if time permits

comparison.

Action points

Issue Action required Responsible (Who) Deadline (By When)

Project Planning Fill out form Brandon and Cheng 9/9/2013

Objective/Requirement

Tree

Create objective tree

with weights

Kyle and Luke 9/9/2013

Requirement Checklist Alex and Lu 9/9/2013

Project Statement Finalize Project

statement

Brandon and Constantin 9/9/2013

Page 80: Google Glass Project

71

Date 9/16/2013 Time 4:00 PM Place OEC 273/275

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Needed a detailed description on how microphones work

Discussed the possibilities of Bluetooth

Group needs to look into how to connect different microphone types

Group needs to set up criteria's for first test.

Open Issues

None.

Page 81: Google Glass Project

72

Date 9/16/2013 Time 5:00 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Discusse Microphones and research

Points discussed

Discussed microphone and microphone research

Luke volunteered to contact the Electronics Support Lab about microphones

Darius, Brandon and Kyle volunteered to help with the Microphone Research

Brandon will post current research and sources on the Google Drive

Action points

Issue Action required Responsible (Who) Deadline (By When)

Microphone Microphone Research Brandon, Darius, Kyle,

Constantin

9/30

Microphone Search for governing

equations

Luke, Cheng Lu, Han,

Yanglong Lu

9/30

Open Issues

None.

Page 82: Google Glass Project

73

Date 9/23/2013 Time 4:00 PM Place OEC 273/276

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Testing must occur in same room.

Explain polar patterns

Find human threshold for hearing.

Design chart needs update

Open Issues

None

Page 83: Google Glass Project

74

Date 10/8/2013 Time 4:00 PM Place OEC 273/276

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Possible control settings for Sensitivity for microphones

Possible option for detachable microphone

L/R comparison test

Acquire less sensitive microphones

Open Issues

None

Page 84: Google Glass Project

75

Date 10/11/2013 Time 5:00 PM Place Google Hangout

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Yanglong Lu

Shaopeng Han

Cheng Lu

Kyle Coleman

Darius Danin

Sponsor (Others (Specify)):

None

Agenda:

Get updates from members

Discuss Task List (10/7-10/14)

Get volunteers/assign people to tasks

Talk about PDR

Points discussed

Microphones have been ordered/ in the works

Luke spoke with Phyo about PDR and our workshop documents

Luke has spoken with Dave about a decibel meter

See Task List (10/7-10/14)

Page 85: Google Glass Project

76

Date 10/18/2013 Time 5:00 PM Place Google Hangout

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Yanglong Lu

Shaopeng Han

Cheng Lu

Kyle Coleman

Darius Danin

Sponsor (Others (Specify)):

None

Agenda:

Get updates from members

Points discussed

Luke spoke with Dave about microphones and decibel reader and that Dave said he would get back with

us on Monday (10/21)

Cheng and Alex have uploaded summaries of the research they have done into acoustic filtering.

Lu added more data points to the read time research

Darius also collected more data for the read time research.

Brandon did research into noise reduction, signal separation and dereverberation.

Constantin and Kyle have been researching into dictation software, and had no problems to report.

Luke has been researching into programming and also has no problems to report.

Action points

Issue Action required Responsible (Who) Deadline (By When)

Req. for Filter Generate Req. for Filter Alex and Brandon 10/21

Read Time Conduct Analysis on

Data

Cheng and Lu 10/21

Programming Conduct more research Luke and Darius 10/21

Req. for Dictation Generate Req. for

Dictation Software

Kyle and Constantin 10/21

Page 86: Google Glass Project

77

Date 10/21/2013 Time 4:00 PM Place OEC 273/276

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Finding a dB meter

Determine the dictation software and microphone

Create test protocols for the tests

Open Issues

None

Page 87: Google Glass Project

78

Date 10/21/2013 Time 6:00 PM Place Olin Lounge

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed

Discussed advisory meeting

Assigned tasks

Action points

Issue Action required Responsible (Who) Deadline (By When)

Microphones Talk to Dave in ESL Luke 10/27

Research Dictation SW Evaluate Current List Dictation Group 10/27

Research Filtering Continue Looking at

different methods

Filter Group 10/27

Open Issues

None

Page 88: Google Glass Project

79

Date 10/28/2013 Time 6:00 PM Place Olin Lounge

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed

Discussed advisory meeting and defined action points (tasks)

Assigned people to the tasks.

Action points

Issue Action required Responsible (Who) Deadline (By When)

Dictation

Research/Selection

Conduct tests on

software

Dictation Group 11/1

Filtering Research Continue research Han 11/1

Programming Research SW and

program for the

platforms

Luke and Brandon 11/1

Open Issues

None.

Page 89: Google Glass Project

80

Date 11/4/2013 Time 4:00 PM Place OEC 273/276

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Darius Danin

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Brandon needs to try to contact Will

Acquire any microphones

Luke, Yanglong need to look for alternative option

Open Issues

None

Page 90: Google Glass Project

81

Date 11/4/2013 Time 6:00 PM Place Olin Lounge

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed

Advisory meeting was discussed

Discussed what needed to be done to catch up

Assigned Tasks

Action points

Issue Action required Responsible (Who) Deadline (By When)

Contact Will Powell Email Will Powell Brandon 11/10

Obtain and work on RPi Buy RPi and begin

programming

Brandon 11/10

Find Alternatives to

Google Glass

Research alternatives to

GG

Luke, Yanglong Lu 11/10

Experiments Conduct tests Yanglong Lu, Cheng Lu,

Han

11/17

Microphones Purchase Temp.

Microphones

Kyle, Constantin 11/7

Dictation Work on dictation SW Kyle, Constantin 11/10

Open Issues

None

Page 91: Google Glass Project

82

Date 11/8/2013 Time 5:00 PM Place Google Hangouts

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed

Discussed progress made on tasks

Assigned new tasks

Action points

Issue Action required Responsible (Who) Deadline (By When)

Experiments Continue testing Yanglong Lu, Cheng Lu,

Han

11/17

RPi Programming Get PocketSphinx

Working

Brandon 11/17

Android Programming Continue Programming Luke 11/17

Patents Read and Summarize

Patents

Kyle 11/17

Microphones Determine how to build

microphone

Constantin 11/17

Open Issues

None.

Page 92: Google Glass Project

83

Date 11/18/2013 Time 6:00 PM Place Olin Lounge

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed

Advisory meeting was discussed

Found errors in testing and decided to rerun some experiments

Assigned Tasks

Agreed to meeting on Wednesday (11/20) to begin compiling Interim Report

Action points

Issue Action required Responsible (Who) Deadline (By When)

RPi Research how to

improve PocketSphinx

performance

Brandon 11/25

Experiments Rerun L/R and Audio-to-

Text experiments

Yanglong Lu, Cheng Lu,

Han

11/25

Android Continue Android

programming

Luke 11/25

Microphones Talk to ESL about

making circuit found

Constantin 11/25

Open Issues

None.

Page 93: Google Glass Project

84

Date 1/8/2014 Time 4:50 pm Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda:

Discuss Team Organization

Assign Leads for Teams

Microphone Purchasing

Points discussed

Team decided to split into three groups: Experimenting/Testing, Programming and Speech Enhancement

Luke Glesener was assigned as lead for programming group based on his programming knowledge

Kyle and Cheng Lu were put on the programming group.

Constantin Jessen volunteered to lead the testing group

Yanglong Lu was put on the testing group

Brandon and Han were put on the speech enhancement group

Open Issues

None

Page 94: Google Glass Project

85

Date 1/13/2014 Time 3:00 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Sponsor (BMW/Others (Specify)):

Agenda: Create tasks for each of the groups

Points discussed

Speech enhancement group was assigned to discover OTS software for speech enhancement on the

android

Testing group was assigned with testing the capabilities and limitations of the Google Glass

Programming group was tasked with researching Glass programming

Action points

Issue Action required Responsible (Who) Deadline (By When)

OTS Software Create List and Contact Suppliers

Speech Enhancement Group

1/19

Test Google Glass Preform Experiments on the Google Glass

Testing Group 1/19

Research Glass Programming

Research Glass Programming

Programming Group 1/19

Open Issues

None.

Page 95: Google Glass Project

86

Date 1/29/2014 Time 3:00 PM Place Evans Study Room

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Discuss programming progress

Run microphone experiments with new components

Do some programming

Points discussed

Decided to break the programming group into three subgroups. One to make the app for the glass, one to

implement the “voice typing” on the android, and one to use the basic speech recognition on the android.

Test group tested the microphone

Assigned tasks (see Task List 1/27-2/2)

Open Issues

None.

Page 96: Google Glass Project

87

Date 1/29/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Appropriate dress code

Plans for the next 2 weeks.

Open Issues

None

Page 97: Google Glass Project

88

Date 2/3/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

If the app can just run 15 min, that’s fine

Possible splitting into 3 groups to decide which app is the best

Shirt prices

Decide on the filter concept

Open Issues

None

Page 98: Google Glass Project

89

Date 2/5/2014 Time 3:00 PM Place Evans Study Room

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss progress and tasks for the week

Discussed selecting one program

Points discussed

Advisory meeting was discussed

Luke said the only thing left is to add in the live cards

The group discussed selecting one program and decided to combine the unique elements of each of the

Glass programs when both are finished

The experiment group will be testing the current limitations of the Glass this week to determine its

validity as a stand-alone device for the speech to text.

Action points

Issue Action required Responsible (Who) Deadline (By When)

Glass Programs Complete them Programming Group 2/10

Glass Tests Test current Glass app Test Group 2/9

Open Issues

None

Page 99: Google Glass Project

90

Date 2/10/2014 Time 3:00 PM Place Library Room 217

Called by Brandon Mills Minutes prepared

by

Brandon Mills

Attended by

Team Members:

Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discuss Tasks

Points discussed Determined the state of the programming

Live Cards near completion

Alex’s group currently working on debugging

Discussed creating an experiment to assess the need for filtering software by determining the ambient

noise threshold.

Action points

Issue Action required Responsible (Who) Deadline (By When)

Assess filter need Create Experiment Experiment Group 2/19

Open Issues None

Page 100: Google Glass Project

91

Date 2/17/2014 Time 3:00 PM Place 1st Floor of Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discussed Progress of Subgroups

Group discussed a possible meeting with Dr. Crawford

Points discussed

Filter group discussed some challenges to programming a custom filter and that none of the companies

have contacted the group back about trial versions of filters. They also contacted a couple more

companies

The Programming subgroup discussed their progress.

The group discussed the move to an immersion card system

The group decided to set up a meeting with Dr. Crawford with Dr. Morkos’ permission

Open Issues

None

Page 101: Google Glass Project

92

Date 2/24/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Adding Dr. Crawford as an advisor

Pick more reasonable numbers for testing

The range of 3 to 5 ft. is perfectly suitable

Different environments with different sound intensities

Possible usage of a secondary microphone

Open Issues

None

Page 102: Google Glass Project

93

Date 2/26/2014 Time 6:45 PM Place 1st Floor of Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Address any questions

Discussed Progress of Subgroups to get approximate times of completion

Discuss time remaining with emphases on deadlines

Discuss logo and shirt options

Points discussed

Discussed that after spring break their is only 5 weeks left

Want to keep 1 week to complete report, run final test, and address any last minute issues.

Programming group is split into two sub groups

a. The first is to get the glass app working

b. the second is to set up a immersion card skeleton

c. If either/both subgroups get done early they will work on the multithreading

Filter group will continue attempting to implement filters

Test group to redo portions of the ambient noise threshold test with emphases on the context the glass is

to be used.

The group decided on a black shirt with a white logo

Members willing to pay the extra $20 out of their own pockets to get them customized.

Logo selected is a 3d cube with G’s on the visible surfaces

Brandon will get a digital logo created to be approved by the group and for Constantin to use to get the

shirts created

Open Issues

None

Page 103: Google Glass Project

94

Date 3/10/2014 Time 3:00 PM Place 1st Floor of Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Discuss what is expected for the last 5 weeks of the semester.

Discuss the logo for the T-Shirts

Finish and print the PP for today’s advisory meeting

Discuss penalties for failing to meet deadlines

10 pt deduction per week for major tasks

5 pt deduction for failing to upload PP and executive summaries on time

Executive summary and PP deadlines will remain constant for the rest of the project

Weekly meetings will remain constant for the rest of the semester

If no email is sent that says otherwise, all meetings will be in the 1st floor of the library

Discuss next meeting with Dr. Crawford

Points discussed

Discussed the above topics

Decided to set up another meeting with Dr. Crawford should any programming problem persist at the end

of the week.

The group has agreed with the the game plan for the last 5 weeks

a. 2-3 weeks getting app working

b. 2 weeks to implement the multithreading and filter

c. 2 weeks to get filters working (not on Glass) so that it can be later implemented on the glass

d. 1 week for new glass threshold tests

Group agreed with the logo, Constantin is looking into ordering the shirts

Action points

Issue Action required Responsible (Who) Deadline (By When)

Need cost of shirts Get quote from company Constantin Monday (2/17)

Open Issues

None

Page 104: Google Glass Project

95

Date 3/10/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Performing a test inside the car with radio off and windows closed

Finding places at which the App works

Open Issues

None

Page 105: Google Glass Project

96

Date 3/17/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Performing test on new prototype

Open Issues

None

Page 106: Google Glass Project

97

Date 3/19/2014 Time 3:00 PM Place 1st Floor of Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Monday advisory meeting

New todo list

Glass App

Shirts

Address questions, comments and concerns

Points discussed

Topics related to the agenda items that were mentioned in the advisory meeting was discussed

Introduced a “Todo List” basically a checklist of things that need to be done for the project to be

considered complete.

Discussed how the app preforms and how it can be enhanced

a. Mainly enhancing the interface

b. Try to implement filter

Discussed shirt cost and wether the team would spend $40 out of their own pocket if the necessary

a. Group agreed to pay if necessary

b. Still undecided on last name or first name on shirts

There were no questions, comments or concerns

Open Issues

None.

Page 107: Google Glass Project

98

Date 3/24/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Performing test on new prototype

Test the app under loud noise conditions

Create a user manual

Test how well the app performances over distance

Finishing the microphones

Putting Tong on the microphone team

Open Issues

None

Page 108: Google Glass Project

99

Date 3/26/2014 Time 3:00 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda: Discuss filter possibility

Discuss microphone capability

Discuss current state of the project

Discuss task list

Points discussed

The microphone option for this semester was found to be an impossibility based off of Tong’s research

into it.

Tong suggests applying a gain via a filter to the signal to increase the range

The filtering was selected, but will more than likely no be able to be implemented this semester due to the

bugs in it

Luke suggested multiplying the signal by a scaler to act as a gain

The project is on track for the speech to text app meeting most of our system requirements

Discussed a new “Items to do” list which includes the tasks for the app, presentation, poster and report. It

can be found in the project directory in group management.

Open Issues

None

Page 109: Google Glass Project

100

Date 3/31/2014 Time 3:00 PM Place Library

Called by Brandon Mills Minutes prepared by Brandon Mills

Attended by

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Sponsor (BMW/Others (Specify)):

Agenda:

Discuss filter update

Discuss Glass app updates

Discuss task list

Meeting on Wednesday to work on the poster

Points discussed

For the filter it was determined that the problem should lie in the way it was implemented and not in the

equations used

Luke’s idea of how to increase the gain worked

A method of doing the text history for the app was discussed, when the user exits the speech to text

portion of the app they will have access to the history view scrolling

Discussed updates to the task list on the drive

Wednesday’s meeting was moved to 4 pm. Will not have meeting minutes since the poster will be all that

is covered.

Open Issues

None.

Page 110: Google Glass Project

101

Date 3/31/2014 Time 4:00 PM Place SKU 116

Called by Dr. Morkos Minutes prepared by Constantin R. Jessen

Attended by Dr. Morkos; Mr. Menon

Team Members: Brandon Mills

Luke Glesener

Kyle Coleman

Cheng Lu

Yanglong Lu

Constantin Jessen

Shaopeng Han

Xinyan Zhu

Fengyuan Tong

Sponsor (BMW/Others (Specify)):

Agenda:

Advisor Meeting

Points discussed

Possible making the microphones work without going in the main code of the glasses

Record a video of the app

Perform testing in real world environments

Open Issues

None

Page 111: Google Glass Project

102

Appendix B Emails

Executive Summary #1 (8/30/13) 3 messages

FIT Google Glass Group <[email protected]> Fri, Aug 30, 2013 at 12:38 PM To: Beshoy Wahib Morkos <[email protected]>

Dr. Morkos

Attached is the Google Glass Group's First executive summary.

~ G^3

ME402 G3 Executive Summary Week01.docx 334K

FIT Google Glass Group <[email protected]> Fri, Aug 30, 2013 at 3:35 PM To: Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Luke Glesener

<[email protected]>, Kyle Coleman <[email protected]>, Cheng Lu <[email protected]>,

Yanglong Lu <[email protected]>, Shaopeng Han <[email protected]>

Sorry about that...just realized I forwarded you all the wrong email.

This is the correct one with a copy of our executive summary. [Quoted text hidden]

ME402 G3 Executive Summary Week01.docx 334K

Beshoy Wahib Morkos <[email protected]> Wed, Sep 4, 2013 at 3:13 PM To: "FIT Google Glass Group ([email protected])" <[email protected]>

Cc: "Varun Menon <[email protected]> ([email protected])" <[email protected]>

Team,

Make sure you forward this to the rest of your advisory committee (Varun) as well.

BM

From: FIT Google Glass Group [mailto:[email protected]]

Sent: Friday, August 30, 2013 12:39 PM To: Beshoy Wahib Morkos Subject: Executive Summary #1 (8/30/13)

Dr. Morkos

Page 112: Google Glass Project

103

Attached is the Google Glass Group's First executive summary.

~ G^3

Executive Summary #2 (9/8/13) 1 message

FIT Google Glass Group <[email protected]> Sun, Sep 8, 2013 at 2:45 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Brandon Mills <[email protected]>, Luke Glesener <[email protected]>, Kyle Coleman

<[email protected]>, Yanglong Lu <[email protected]>, Cheng Lu <[email protected]>,

[email protected], Shaopeng Han <[email protected]>

Dr. Morkos

Attached is the Google Glass Group's second executive summary.

~ G^3

Executive Summary #3 (9/15/13) 1 message

FIT Google Glass Group <[email protected]> Sun, Sep 15, 2013 at 3:28 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected], Brandon Mills

<[email protected]>, Luke Glesener <[email protected]>, Kyle Coleman

<[email protected]>, Yanglong Lu <[email protected]>, Cheng Lu <[email protected]>,

Constantin Jessen <[email protected]>, Shaopeng Han <[email protected]>

Dr. Morkos

Attached is the Google Glass Group's third executive summary.

~ G^3

Executive Summary #4 (9/22/13) 1 message

FIT Google Glass Group <[email protected]> Sun, Sep 22, 2013 at 3:45 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos

Attached is the Google Glass Group's fourth executive summary.

~ G^3

Page 113: Google Glass Project

104

Executive Summary #5 (9/29/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Sep 29, 2013 at 3:51 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos

Attached is the Google Glass Group's fifth executive summary.

~ G^3

Executive Summary #6 (10/6/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Oct 6, 2013 at 3:10 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's sixth executive summary.

~ G^3

Executive Summary #7 (10/20/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Oct 20, 2013 at 3:10 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's seventh executive summary.

Also, we heard you were going to be out of town next week and want to know if you are going to be on campus

tomorrow for our advisory meeting?

~G^3

Executive Summary #8 (10/20/2013)

Page 114: Google Glass Project

105

1 message

FIT Google Glass Group <[email protected]> Mon, Oct 28, 2013 at 2:27 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's eighth executive summary.

~G^3

Executive Summary #9 (11/3/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Nov 3, 2013 at 3:01 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's ninth executive summary.

~G^3

Executive Summary #10 (11/11/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Nov 10, 2013 at 7:48 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's tenth executive summary.

~G^3

Executive Summary #11 (11/17/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Nov 17, 2013 at 3:53 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Page 115: Google Glass Project

106

Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's eleventh executive summary.

~G^3

Senior Project 2 messages

FIT Google Glass Group <[email protected]> Wed, Nov 20, 2013 at 12:02 PM To: Beshoy Wahib Morkos <[email protected]>

Good afternoon,

Congratulations on your new baby. I am sorry to bother you since I am sure you are busy. We have a working

prototype, albeit slow and inaccurate at the moment. So I am wanting to dial down the research and testing and

ramp up the documentation compiling and report writing, if that is all right with you. I was also wondering,

since the PDR is contained in the interim report, if we can just submit the interim report and do the PDR in

power point form?

For the interim report, what all would you like to see in it besides the project description, goals, requirements,

background and research, testing, decisions and Global, Social, and Contemporary Issues? Would it be possible for some examples and/or guideline to be uploaded to angel?

Thank you for your time,

~Google Glass Group

Beshoy Wahib Morkos <[email protected]> Wed, Nov 20, 2013 at 12:52 PM To: FIT Google Glass Group <[email protected]>

Page 116: Google Glass Project

107

Team,

Hope all is well. I apologize we did not get the opportunity to meet yesterday. It may be hectic for me the rest of the semester. I would plan for one last meeting (interim presentation) to occur during finals week. Working prototype sounds fantastic. Keep working on improving it for demonstration purposes. It does not matter if it is slow and inaccurate, it is functioning! I agree with your recommendation for ramping down research. However, I would continue to work on testing and prototyping. You still have about a week’s worth of time (that’s 100 man hours you can utilize).

The interim will serve as a PDR for your team. Your presentation should suffice as both a PDR built into an interim presentation. You should include everything you described. Use the project template online as a guideline. If you need support, send Phyo your report and he can help.

Hope that helps. Let me know if you need any additional information.

BM

From: FIT Google Glass Group [mailto:[email protected]]

Sent: Wednesday, November 20, 2013 12:02 PM To: Beshoy Wahib Morkos Subject: Senior Project

Executive Summary #12 (11/25/2013) 1 message

FIT Google Glass Group <[email protected]> Sun, Nov 24, 2013 at 2:53 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Cc: Brandon Mills <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han

<[email protected]>, Luke Glesener <[email protected]>, Constantin Jessen

<[email protected]>, Yanglong Lu <[email protected]>, Cheng Lu

<[email protected]>

Dr. Morkos,

Attached is the Google Glass Group's twelfth executive summary.

~G^3

Interim Report 5 messages

FIT Google Glass Group <[email protected]> Mon, Nov 25, 2013 at 7:50 PM To: Beshoy Wahib Morkos <[email protected]>

Page 117: Google Glass Project

108

Hello Dr. Morkos,

We were wondering when our interim report is due? We were planning on having it done during finals week, if

thats alright.

~G^3

Beshoy Wahib Morkos <[email protected]> Mon, Nov 25, 2013 at 8:01 PM To: FIT Google Glass Group <[email protected]>

Team,

It is due the day of your interim presentation, December 2nd.

BM

Sent from my HTC EVO 4G LTE exclusively from Sprint [Quoted text hidden]

FIT Google Glass Group <[email protected]> Tue, Nov 26, 2013 at 9:59 AM To: Beshoy Wahib Morkos <[email protected]>

Good morning,

How do we go about using our team budget to print out the interim report?

~G^3 [Quoted text hidden]

Beshoy Wahib Morkos <[email protected]> Tue, Nov 26, 2013 at 12:11 PM To: FIT Google Glass Group <[email protected]>

Team,

I notified the class of this deadline yesterday and all documentation required during interim presentations. It is important your team comes to class. I saw only 2 members of your team in class yesterday. This is not acceptable.

bm

From: FIT Google Glass Group [mailto:[email protected]]

Sent: Monday, November 25, 2013 7:50 PM To: Beshoy Wahib Morkos Subject: Interim Report

Hello Dr. Morkos,

We were wondering when our interim report is due? We were planning on having it done

Page 118: Google Glass Project

109

during finals week, if thats alright.

~G^3

Beshoy Wahib Morkos <[email protected]> Tue, Nov 26, 2013 at 12:14 PM To: FIT Google Glass Group <[email protected]>

Team,

I would first recommend you put your report together. You may be able to print it out here on campus. I recommend you make color prints of only those pages that require color. Otherwise, black/white will suffice. If you find you need to print off campus, make sure it is not over $50 and we will reimburse.

BM

From: FIT Google Glass Group [mailto:[email protected]] Sent: Tuesday, November 26, 2013 9:59 AM

To: Beshoy Wahib Morkos Subject: Re: Interim Report

Executive Summary #13 (1/12/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Jan 13, 2014 at 1:52 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's thirteenth executive summary. ~G^3

(1/13) Advisory Meeting PP

1 message

FIT Google Glass Group <[email protected]> Mon, Jan 13, 2014 at 7:10 PM To: [email protected]

Good evening,

Page 119: Google Glass Project

110

Here is a copy of the presentation for today's advisory meeting. ~G^3

Executive Summary #14 (1/19/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Jan 20, 2014 at 3:56 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, "ddinan12 ." <[email protected]>

Dr. Morkos, Attached is the Google Glass Group's fourteenth executive summary. ~G^3

Interim Report

2 messages

FIT Google Glass Group <[email protected]> Tue, Jan 21, 2014 at 9:03 AM To: Beshoy Wahib Morkos <[email protected]>

Good morning, Would it be possible to arrange a time for a couple members of the Google Glass Group to meet with you tomorrow after the senior design class to discuss the interim report? (I.e. What we did well, what we did bad, etc) Thank you for your time, ~G^3

Beshoy Wahib Morkos <[email protected]> Tue, Jan 21, 2014 at 12:45 PM To: FIT Google Glass Group <[email protected]>

Team,

You may drop by anytime this afternoon. Please review my office hours posted on the door

bm

From: FIT Google Glass Group [mailto:[email protected]] Sent: Tuesday, January 21, 2014 9:03 AM

To: Beshoy Wahib Morkos

Page 120: Google Glass Project

111

Subject: Interim Report

[Quoted text hidden]

Item in. Please pick up from my office. 1 message

Beshoy Wahib Morkos <[email protected]> Thu, Jan 23, 2014 at 3:24 PM To: "FIT Google Glass Group ([email protected])" <[email protected]>

______________________________________________ Beshoy W. Morkos, Ph.D.

Assistant Professor

Department of Mechanical and Aerospace Engineering

Systems Research in Intelligent Design and Engineering (STRIDE) Research Lab

Florida Institute of Technology

150 West University Blvd., Melbourne FL 32901

F.W. Olin Engineering Complex, Office 245

E-mail: [email protected]

p:(321)674-8780 f:(321)674-8813

image001.jpg 34K

Executive Summary #15 (1/26/2014)

1 message

FIT Google Glass Group <[email protected]> Mon, Jan 27, 2014 at 1:54 AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>

Dr. Morkos,

Page 121: Google Glass Project

112

Attached is the Google Glass Group's fifteenth executive summary. ~G^3

Executive Summary #16 (2/2/2014)

1 message

FIT Google Glass Group <[email protected]> Mon, Feb 3, 2014 at 1:46 AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's sixteenth executive summary. ~G^3

Executive Summary #17 (2/9/2014)

3 messages

FIT Google Glass Group <[email protected]>

Mon, Feb 10, 2014 at 2:44 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's seventeenth executive summary. ~G^3

ME402 G3 Executive Summary 2_5.docx 46K

Beshoy Wahib Morkos <[email protected]> Tue, Feb 11, 2014 at 1:52 AM To: FIT Google Glass Group <[email protected]>

Team, Can we meet on Wednesday instead? I am sick and am not in today. Let me know. BM ________________________________ From: FIT Google Glass Group [[email protected]] Sent: Sunday, February 09, 2014 1:44 PM

Page 122: Google Glass Project

113

To: Beshoy Wahib Morkos; [email protected] Cc: Cheng Lu; Brandon Mills; Constantin Jessen; Kyle Coleman; Shaopeng Han; Luke Glesener; Yanglong Lu; [email protected] Subject: Executive Summary #17 (2/9/2014) [Quoted text hidden]

FIT Google Glass Group <[email protected]>

Tue, Feb 11, 2014 at 2:30 AM

To: Beshoy Wahib Morkos <[email protected]>

That works for us, when would like to have the meeting Wednesday?

Thank you for contacting us

1 message

bdSound <[email protected]> Tue, Feb 11, 2014 at 7:48 PM Reply-To: bdSound <[email protected]> To: [email protected]

This message is a confirmation that your information has been successfully submitted and will be processed as soon as possible. *Your name* Brandon Mills *Your email* [email protected] *Company Name* Florida Institute of Technology *Company Website Address* www.fit.edu *Phone Number* *Country* *How did you hear about us?* *How do you wish to be contacted?* Email *Describe your request* Hello, I am an engineering student at the Florida Institute of Technology in United States and am working on a capstone project that requires clear speech from a sound source approximately 2-10 feet away from the microphone in environments that may have ambient noise for a speech to text application on an android platform. I was wondering if bdSES would be able to enhance the audio being captured under this condition and if I would be able to receive a trial version for my project.

Page 123: Google Glass Project

114

Thank you for your time, ~Brandon Mills *I agree with your privacy policy and to be contacted back* Yes ------ bdSound

Noise Reduction Software

1 message

FIT Google Glass Group <[email protected]> Wed, Feb 12, 2014 at 10:49

AM To: [email protected]

Hello, I am an engineering student at the Florida Institute of Technology in United States and am working on a capstone project that requires clear speech from a sound source approximately 2-10 feet away from the microphone in environments that may have ambient noise for a speech to text application on an android platform. I was wondering if your noise reduction software would be able to enhance the audio being captured under this condition and if I would be able to receive a trial version for my project. Thank you for your time, ~Brandon Mills

Advisory Meeting (2/12)

1 message

FIT Google Glass Group <[email protected]> Wed, Feb 12, 2014 at 1:59 PM To: Beshoy Wahib Morkos <[email protected]>

Good afternoon Dr. Morkos, Are we still going to have our advisory meeting today, or would you like us to make a PP video of our presentation and send it to you? ~G^3

Executive Summary #18 (2/16/2014)

1 message

FIT Google Glass Group <[email protected]> Mon, Feb 17, 2014 at 3:00

AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen

Page 124: Google Glass Project

115

<[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's eighteenth executive summary. ~G^3

Advisory Meeting Tomorrow

6 messages

FIT Google Glass Group <[email protected]> Tue, Feb 18, 2014 at 5:19

PM To: Beshoy Wahib Morkos <[email protected]>

Dr. Morkos, We noticed that you sent an email about having class tomorrow at 3:00 pm. We were under the impression that we had an advisory meeting tomorrow since we were unable to have one last week. Are we presenting tomorrow? If not, when could we reschedule? ~G^3

Beshoy Wahib Morkos <[email protected]> Wed, Feb 19, 2014 at 6:37 AM To: FIT Google Glass Group <[email protected]>

Team, I will try to squeeze you guys in tomorrow after the presentation. We may only have 20 minutes. BM Sent from my HTC EVO 4G LTE exclusively from Sprint [Quoted text hidden]

FIT Google Glass Group <[email protected]> Wed, Feb 19, 2014 at 8:25 AM To: Luke Glesener <[email protected]>

Do you reckon he means today? [Quoted text hidden]

Luke Glesener <[email protected]> Wed, Feb 19, 2014 at 9:14 AM To: FIT Google Glass Group <[email protected]>

I think yes. -------------------------------------------- Luke Glesener

Page 125: Google Glass Project

116

Florida Institute of Technology Office of Sponsored Programs FITSSFF President Phone: (651) 295-4528 -------------------------------------------- [Quoted text hidden]

FIT Google Glass Group <[email protected]> Wed, Feb 19, 2014 at 11:46 AM To: Beshoy Wahib Morkos <[email protected]>

Good Morning/Afternoon, Sorry to bug you, but you mean today after class right? ~ G^3 [Quoted text hidden]

Beshoy Wahib Morkos <[email protected]> Wed, Feb 19, 2014 at 11:46 AM To: FIT Google Glass Group <[email protected]>

Yes, we can meet after class today.

BM

From: FIT Google Glass Group [mailto:[email protected]] Sent: Wednesday, February 19, 2014 11:47 AM To: Beshoy Wahib Morkos

Subject: Re: Advisory Meeting Tomorrow

[Quoted text hidden]

Executive Summary #19 (2/23/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Feb 24, 2014 at 3:43 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's nineteenth executive summary. ~G^3

R: Information request from your website

Page 126: Google Glass Project

117

3 messages

Fabio Cagnetti | bdSound <[email protected]> Mon, Feb 24, 2014 at

3:25 PM To: Brandon Mills <[email protected]>

Hi Brandon, sorry for my late reply, we were very busy during this period also to let the android version ready for download from the website. The bdSES has a noise reduction processing block but for your application purpose this kind of NR may not be the most performing solution. However we will be more than happy if you would like to report us your results; if possible we will try to help you for your project. Regards, Fabio bdSound srl via Bernardino Verro, 33/6 20141 Milano (MI) Tel: +39 02 899 50 230 Fax: +39 02 899 50 231 Mob: +39 3294684862 www.bdsound.com [email protected] Follow us: www.bdsound.com/linkedin www.bdsound.com/googleplus www.bdsound.com/twitter www.bdsound.com/facebook www.bdsound.com/blog ================================================ IMPORTANT ADVICE This e-mail (and any attachment(s)) is strictly confidential and for use only by intended recipient(s). Any opinions therein expressed are those of the author. Therefore its content doesn't represent any commitment between bdSound srl and the recipient(s) and no liability or responsibility is accepted by bdSound srl for the above mentioned content. INFORMAZIONE IMPORTANTE Il contenuto e gli allegati di questo messaggio sono strettamente confidenziali, e ne sono vietati la diffusione e l'uso non autorizzato. Le opinioni ivi eventualmente espresse sono quelle dell'autore: di conseguenza il messaggio non costituisce impegno contrattuale tra bdSound srl ed il destinatario, e bdSound srl non assume alcuna responsabilità riguardo ai contenuti del testo e dei relativi allegati, né per eventuali intercettazioni, modifiche o danneggiamenti. ================================================ -----Messaggio originale----- Da: BDSOUND CONTACT [mailto:[email protected]] Inviato: mercoledì 12 febbraio 2014 01:49 A: [email protected] Oggetto: Information request from your website

Page 127: Google Glass Project

118

*Your name* Brandon Mills *Your email* [email protected] *Company Name* Florida Institute of Technology

*Company Website Address* www.fit.edu *Phone Number* *Country* *How did you hear about us?* *How do you wish to be contacted?* Email *Describe your request* Hello, I am an engineering student at the Florida Institute of Technology in United States and am working on a capstone project that requires clear speech from a sound source approximately 2-10 feet away from the microphone in environments that may have ambient noise for a speech to text application on an android platform. I was wondering if bdSES would be able to enhance the audio being captured under this condition and if I would be able to receive a trial version for my project. Thank you for your time, ~Brandon Mills *I agree with your privacy policy and to be contacted back* Yes bdSound - http://www.bdsound.com/contacts/foxcontact.html Client: 163.118.245.145 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11

[email protected] <[email protected]> Mon, Feb 24, 2014 at 3:58 PM Reply-To: [email protected] To: [email protected]

this week 5 screen Sent through Glass

FIT Google Glass Group <[email protected]> Mon, Feb 24, 2014 at 9:08

Page 128: Google Glass Project

119

PM To: Fabio Cagnetti | bdSound <[email protected]>

Hello Fabio, Thank you for your response, my team and I will look into incorporating your processing block. If it works we will report its results. Sorry for the previous email, it was accediently sent from our Google Glass. Thank you for your time, ~Brandon [Quoted text hidden]

Proposed Advisory Status

3 messages

FIT Google Glass Group <[email protected]> Thu, Feb 27, 2014 at 5:40

PM To: "[email protected]" <[email protected]> Cc: Beshoy Wahib Morkos <[email protected]> Bcc: Brandon Mills <[email protected]>

Dr. Crawford (this is Luke from the Google Glass Group email), We discussed it with our advisor Dr. Morkos, and were wondering if you would be interested in joining the advisory committee for our senior design team? We have presentations every Monday at 4:30 pm in the Skurla lecture hall. However, these meetings are not a requisite for being on the advisory committee. It is kind of last minute, and we understand if you are too busy. If you have more questions feel free to contact Dr. Morkos. Also, attached is a few code samples we have generated. If you could look at them and give us feedback on what we could improve, it would be a great help. (Just be aware though that we are mechanical engineers, so it may not be up to your normal standards.) Ep8.zip is an android app we created that performs the gist of what we want to accomplish. Stopwatch1-2.zip is an app we created using the structure laid out in the Google Glass sample stopwatch app (It's not a stopwatch anymore). It is not completed yet, but it does one iteration of what we want it to do. Thank you for your time. ~G^3

2 attachments

Ep8.zip 1824K

Stopwatch1-2.zip 134K

Heather Crawford <[email protected]> Fri, Feb 28, 2014 at 9:01 AM

Page 129: Google Glass Project

120

To: FIT Google Glass Group <[email protected]> Cc: Beshoy Wahib Morkos <[email protected]>

Hi Luke, Thank you very much for asking me to be on your advisory committee. Unfortunately, I have to decline as my semester is already quite full. I will have a look at your code, but I won't be able to get to it until sometime late next week or early the week after. Is that too late to be helpful? Regards, Heather Crawford, Ph.D. Assistant Professor Department of Computer Sciences and Cybersecurity Florida Institute of Technology Ph: (321) 674-7559 Fax: (321) 674-7046 [email protected] On 2/27/14, 5:40 PM, "FIT Google Glass Group" <[email protected]> wrote: [Quoted text hidden]

FIT Google Glass Group <[email protected]> Sun, Mar 2, 2014 at 2:50 PM To: Heather Crawford <[email protected]> Cc: Beshoy Wahib Morkos <[email protected]>

No that is not too late. Any help you are willing to give is appreciated. Thanks again! [Quoted text hidden]

Executive Summary #20 (3/2/2014)

1 message

FIT Google Glass Group <[email protected]> Mon, Mar 3, 2014 at 4:18 AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, [email protected], Yanglong Lu <[email protected]>

Dr. Morkos, Attached is the Google Glass Group's twentieth executive summary. ~G^3

Executive Summary #21 (3/9/2014)

1 message

FIT Google Glass Group <[email protected]> Mon, Mar 10, 2014 at 3:20

Page 130: Google Glass Project

121

AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Cheng Lu <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's twenty-first executive summary. ~G^3

Executive Summary #22 (3/16/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Mar 17, 2014 at 1:05 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, [email protected], Yanglong Lu <[email protected]>

Dr. Morkos, Attached is the Google Glass Group's twenty-second executive summary. ~G^3

T-Shirt Final Quote

2 messages

FIT Google Glass Group <[email protected]> Wed, Mar 19, 2014 at 5:02 PM To: Beshoy Wahib Morkos <[email protected]>

Good Afternoon, The final cost of the shirts are as quoted below,

Quote #1

1 Color Front | 0 Color Back

Item 1. Ultra Club Pique Polo -- Black

10 S-XL

with 10 Names

10 @ $40.56 each

Page 131: Google Glass Project

122

Total for 10 pieces $405.60

Free shipping to Melbourne, FL (32901)

We would also like to know what shirt size you would like and how you would

like us to purchase them.

Thank you for your time,

~G^3

Beshoy Wahib Morkos <[email protected]> Thu, Mar 20, 2014 at 11:26 PM To: FIT Google Glass Group <[email protected]>

Team,

I asked about this and we are not approved to spend money on T-Shirts from senior design funds. These funds are specifically earmarked for equipment and material only. Maybe on some of the competition based projects, but not noncompetitive projects. Sorry about this gentlemen.

BM

From: FIT Google Glass Group [mailto:[email protected]] Sent: Wednesday, March 19, 2014 5:02 PM

To: Beshoy Wahib Morkos Subject: T-Shirt Final Quote

[Quoted text hidden]

Executive Summary #23 (3/23/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Mar 24, 2014 at 1:58 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's twenty-third executive summary. ~G^3

Page 132: Google Glass Project

123

Executive Summary #24 (3/30/2014)

1 message

FIT Google Glass Group <[email protected]>

Mon, Mar 31, 2014 at 1:10 AM

To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's twenty-fourth executive summary. ~G^3

FW: Showcase Poster Template

3 messages

Beshoy Wahib Morkos <[email protected]> Wed, Mar 26, 2014 at 3:16 PM To: "Kristen Brockway ([email protected])" <[email protected]>, "Jonathan Kvasnok ([email protected])" <[email protected]>, "Randy Arjunsingh ([email protected])" <[email protected]>, "[email protected]" <[email protected]>, "Francisco Feliciano ([email protected])" <[email protected]>, "[email protected]" <[email protected]>, "FIT Google Glass Group ([email protected])" <[email protected]>, "Kurt Bostelaar ([email protected])" <[email protected]>

FYI

From: Ronnal Reichard Sent: Monday, March 17, 2014 3:35 PM To: Beshoy Wahib Morkos; Razvan Rusovici; Barry Grossman; Philip Chan; Jonathan Whitlow;

Ralph Locurcio; Howell Heck; Stephen Wood; Kevin Johnson Cc: Martin Glicksman Subject: Showcase Poster Template

All:

Attached is the showcase poster template. Please submit electronically in PPT

format. The deadline for submission of posters is 4 April 2014. Submissions after that

date may or may not be printed in time for the Showcase or for inclusion in the Showcase

E-book.

Thanks,

Ron

Page 133: Google Glass Project

124

Ronnal P. Reichard, Ph.D.

Professor of Engineering

Coordinator, Senior Design

College of Engineering

Florida Tech

Office: 321-674-7349

Cell: 321-432-8076

Showcase2012 Poster Template 3-15-2012.ppt 565K

FIT Google Glass Group <[email protected]> Thu, Mar 27, 2014 at 9:09 AM To: "[email protected]" <[email protected]>

[Quoted text hidden]

Showcase2012 Poster Template 3-15-2012.ppt 565K

FIT Google Glass Group <[email protected]> Wed, Apr 2, 2014 at 5:25 PM To: Kyle Coleman <[email protected]>

---------- Forwarded message ---------- From: FIT Google Glass Group <[email protected]> Date: Thursday, March 27, 2014 Subject: FW: Showcase Poster Template [Quoted text hidden]

Showcase2014 Poster MAE Google Glass Group

1 message

FIT Google Glass Group <[email protected]> Thu, Apr 3, 2014 at 9:05 PM To: Beshoy Wahib Morkos <[email protected]>, [email protected]

Dr. Morkos, Please find our group's showcase poster attached.

Page 134: Google Glass Project

125

~G^3

Executive Summary #25 (4/6/2014)

2 messages

FIT Google Glass Group <[email protected]> Mon, Apr 7, 2014 at 2:03 AM To: Beshoy Wahib Morkos <[email protected]>, [email protected] Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, [email protected]

Dr. Morkos, Attached is the Google Glass Group's twenty-fifth executive summary. ~G^3

ME402 G3 Executive Summary 2_13.docx 53K

Beshoy Wahib Morkos <[email protected]> Wed, Apr 9, 2014 at 11:35 AM To: FIT Google Glass Group <[email protected]>, "[email protected]" <[email protected]> Cc: Cheng Lu <[email protected]>, Brandon Mills <[email protected]>, Constantin Jessen <[email protected]>, Kyle Coleman <[email protected]>, Shaopeng Han <[email protected]>, Luke Glesener <[email protected]>, Yanglong Lu <[email protected]>, "[email protected]" <[email protected]>

Team,

As punishment for your puerile behavior on your executive summaries, I am requiring you follow through on your documents. Thus, I am asking that you bring in the following to class tomorrow, as sited in your executive summaries:

1. Mud Pie (closest cake related to a pig)

2. Portal Cake (as close as you can get)

3. Key Lime Pie

Make sure you bring plates, forks, and napkins as well. The executive summary portion of your final grade will depend on your ability to deliver on the items sited on your executive summaries.

Lesson learned. Thanks gentlemen.

Page 135: Google Glass Project

126

BM

From: FIT Google Glass Group [mailto:[email protected]] Sent: Sunday, April 06, 2014 2:04 PM To: Beshoy Wahib Morkos; [email protected]

Cc: Cheng Lu; Brandon Mills; Constantin Jessen; Kyle Coleman; Shaopeng Han; Luke Glesener; Yanglong Lu; [email protected] Subject: Executive Summary #25 (4/6/2014)

Dr. Morkos,

Attached is the Google Glass Group's twenty-fifth executive summary.

~G^3

Advisory Meeting - Out of Town on 4/7/14

6 messages

Beshoy Wahib Morkos <[email protected]> Mon, Apr 7, 2014 at 11:22 AM To: "FIT Google Glass Group ([email protected])" <[email protected]>

Google Glass Team,

I will be out of town on Monday, April 7th. We will reschedule your weekly advisory

meeting to Wednesday, April 9th. Please send me your executive summary by Tuesday.

BM

______________________________________________ Beshoy W. Morkos, Ph.D.

Assistant Professor

Department of Mechanical and Aerospace Engineering

Systems Research in Intelligent Design and Engineering (STRIDE) Research Lab

Florida Institute of Technology

150 West University Blvd., Melbourne FL 32901

F.W. Olin Engineering Complex, Office 245

E-mail: [email protected]

p:(321)674-8780 f:(321)674-

Page 136: Google Glass Project

127

8813

image001.jpg 34K

FIT Google Glass Group <[email protected]> Wed, Apr 9, 2014 at 8:45 AM To: Beshoy Wahib Morkos <[email protected]>

Good Morning Dr Morkos, When would you like the Google Glass Group to do our mock presentation today? ~G^3 [Quoted text hidden]

Beshoy Wahib Morkos <[email protected]> Wed, Apr 9, 2014 at 1:48 PM To: FIT Google Glass Group <[email protected]>

Team,

I will try to squeeze you guys in after the two SAE teams today. We will have to accelerate it though.

BM

From: FIT Google Glass Group [mailto:[email protected]] Sent: Wednesday, April 09, 2014 8:45 AM To: Beshoy Wahib Morkos

Subject: Re: Advisory Meeting - Out of Town on 4/7/14

[Quoted text hidden]

FIT Google Glass Group <[email protected]> Wed, Apr 9, 2014 at 1:50 PM To: Brandon Mills <[email protected]>, Kyle Coleman <[email protected]>, Constantin Jessen <[email protected]>, Cheng Lu <[email protected]>, "Shaopeng Han (Alex)" <[email protected]>, Yanglong Lu <[email protected]>, Xinyan Zhu <[email protected]>

[Quoted text hidden]

FIT Google Glass Group <[email protected]> Wed, Apr 9, 2014 at 1:52 PM To: Beshoy Wahib Morkos <[email protected]>

To clarify, they have their meetings at 3 and 3:30 pm, so we would be presenting around 3:45 to 4pm? Thanks again, ~G^3 [Quoted text hidden]

Page 137: Google Glass Project

128

Beshoy Wahib Morkos <[email protected]> Wed, Apr 9, 2014 at 1:53 PM To: FIT Google Glass Group <[email protected]>

Yes,

That is correct. I will try.

From: FIT Google Glass Group [mailto:[email protected]]

Sent: Wednesday, April 09, 2014 1:53 PM

[Quoted text hidden]

Page 138: Google Glass Project

129

Appendix C Executive Summaries

Week 1

This Week’s Plan:

The plan for this stage in the project was to split into individual groups and start research on specific

fields.

This Week’s Progress:

Brandon Mills focused on researching and compiling information on how to evaluate a microphone,

addressing what parameters the microphone must meet in order to be used for our purposes. Brandon has

also been continually updating and manipulating the Google drive to better suit our needs.

Constantin Jessen is working on physically mounting a microphone to a portable device such as a cell

phone and eventually the Google Glass. Constantin has purchased an inexpensive USB microphone that

he hopes to use to prove the concept before we move on to attaching the actual microphone. A decision

matrix will be created and utilized in the following week, once enough information on microphones is

compiled, in order to decide which microphone is best suited for the task at hand. Multiple iterations of

this decision matrix are expected in order to get the best product, and as new information is acquired.

Luke Glesener has been working on manipulating the open source code of an Android smart phone.

Adapting the source code will allow our group to possibly use the Android smart phone as a mobile

platform for the microphone assembly. Luke believes that he may be able to gain access to the Android’s

on-board speech to text software, used by the device to create text messages from audio input. This may

allow our group to circumvent using more expensive stand-alone dictation software. The initial reason we

have decided to focus our efforts on the Android is because, Android devices can connect to and interface

with Bluetooth enabled devices. This is known to work with hard drives, keyboards, and such. Also

Android apps can be programmed to interface and share data with each other; which means we can get

data from whatever dictation software we end up using and implement it in an app if we so choose. Also,

three quarters of the team have Android smart phones and it is what we are most familiar with. Future

research may entail looking into other mobile platforms.

Kyle Coleman and Yanglong Lu have been researching the previously mentioned software. There are

multiple software packages out there; Lu has been compiling a list of dictation software and their specific

abilities in order to facilitate a future decision matrix. Some important factors Lu has been focusing on

are; cost, or how much of the budget will we have to reserve to purchasing the software, effectiveness of

the dictation itself, mobility or what platforms can the program be run on, and usability or how easy it is

to use and manipulate the program to address our needs. Coleman has taken to researching specific

dictation software, primarily the most prominent speech to text company Dragon. Dragon has two major

capabilities that might be advantageous to exploit. The Dragon Nuance software already has the

capability of using a mobile microphone, such as the one in an android device, working in tandem with a

Dragon app to pick up and send audio from the Android to a computer to be translated and output on the

screen. Additionally, Dragon Nuance can provide a developer’s kit that will allow the team to create

Dragon mobile apps using the Dragon Nuance source codes. This developer’s kit working alongside

Page 139: Google Glass Project

130

Luke’s work with the Android device may be what we need to complete the project. Coleman hopes to

find similar developer’s kits provided by other companies, possibly even Google itself, in order to have a

wider variety of software to choose from, and to see if it might be easier to get a hold of the Kit itself.

Shaopeng Han and Cheng Lu have been focusing on the Google Glass itself. They hope to find more

information on how the glasses can be improved and implemented. As of yet the Google Glasses are

rather limited in what they can do. It is our hope that once we successfully complete our current project

with the speech to text, we can move on to additional functions.

Issues: The biggest issues we are having relate to: 1) Not having the Google Glass. 2) Finding specific

product information for software from general sales pitches. 3) Finding specific cost of products. ( 2 and 3

may be solved by contacting companies directly) 4) Working with Google to possibly get their software and developer’s kit at a reduced price.

Week 2

This Week’s Plan:

The plan for this stage of the project was to step back and work on our group’s project statement and

requirements, both individually and as a whole. In addition to this further research was to be emphasized

and a brainstorming session was to be conducted. however proper documentation and formatting of the

Requirements and Project statement took paramount.

This Week’s Progress:

This week the group came together in an attempt to refocus what we were doing on the documentation

side of the project. Previously, before being told to in class, the group compiled a group Project Statement

and generated a requirements worksheet. However in class, it was noted that each individual would need

to complete their own version of these documents before compiling a universal copy. As a result the team

split up with each person being assigned to produce their own versions. Much of the time spent this week

was working on these individually. Research was being conducted into what each member would like to

see this project produce as an end product.

In addition to the individual work, a partial brainstorming session was conducted with the majority of the

members in attendance. It is expected that a more in depth brainstorming session will take place in the

following week as the preliminary one has only set up the ground rules for how we may attack our

problems. The brainstorming session will focus on generating functional diagrams and concepts of

operation.

Brandon Mills has continued his work pertaining to the maintaining and updating the Google drive while

setting up meeting times and schedules. Constantine has continued his research in sector of microphone

technology. The rest of the team is similarly continuing to focus on their individual areas of interest;

however it appears the team is running into some difficulty moving forward as we in general do not know

what the next step needs to be. In the coming week, the group will have to primarily decide on a course of

action to take in order to move out of our current stalemate.

On another note, every member of the team was present for the workshop presented by Phyo. The

resultant homework assigned has been divided up among the team, with each document having at least

two personnel working on it.

Page 140: Google Glass Project

131

Project planning form: Brandon and Shaopeng

Objective requirement tree: Luke and Coleman

Requirements document (based upon requirements checklist): Cheng and Yanglong

Week 3

This Week’s Plan:

The plan for this stage of the project was to conduct a brainstorming session in order to choose a modular

system design. Each member of the team was to create their own ideas for the systems design and then

present them at the brainstorming meeting. A Gant chart was to be created and preliminary experimental

procedures for the testing of microphones were to be created.

This Week’s Progress:

Early this week the group completed and handed in copies of the working documents for the; project

planning form, objective requirement tree, and requirements document. Producing these documents was

an assignment from the previous week’s workshop.

After the meeting with our advisors on Monday, the team split up to work on individual ideas for how

they believed the system design should be laid out. On the following Wednesday, a brainstorming session

was conducted to compile everyone’s ideas and come up with one that encompasses the aspects that the

group as a whole decided would give us the most options.

In addition to the brainstorming session, team members either continued their efforts in researching

microphones and programs, or started researching how the use of video devices can assist in the process

of separating sources. The concept, generated during the brainstorming session, was that video can be

used to compare the text being sent to the screen to the apparent speech facial patterns of the target

source. This application might latter be expanded into the use of “lip reading” when the target source is

too far away, or the ambient noise, makes it impossible for audio to be separated out. Problems predicted

about the use of a video device include; storage issues, size and weight requirements, mounting issues,

compatibility with speech software, do “lip reading” algorithms even exist, power supply issues.

Source

•Person

Input Device

•2x Mic.

•Video

Storage

•Optional

Filter

•S.S.S.

•L/R Comp.

•Est. Distance

•Calc. Expected dB

•Oscillate

•Order Liklieness

•Face Recog.

Dictation

•Software

Display

•Text

Page 141: Google Glass Project

132

Similar issues must be addressed when talking about the use of two microphones. The group is still not as

of yet sure how the microphones will be mounted to the Google Glass frame, or where they will draw

their power and store their data. The reasoning behind the dual microphone system stems from research

into how audio sources can be separated, identified, and located. Two microphones allow for more data to

be collected from the source and comparisons can be made between the two. For example if one

microphone picks up a source sooner and with a higher volume than the second microphone, it can be

assumed that the source is closer to one than the other, and depending on where the devices are mounted,

it can be mathematically calculated precisely how far away the source is and its direction. There was

some minor discussion on what mode of communication should be used between the recording devices

and the platforms. Hard-wire connections would be the most reliable; however the wires themselves may

get in the way. Blue-Tooth is moderately reliable as well; however the specifics of the actual utilization of

BT devices will need to be researched. It is expected that a decision matrix will need to be created to

decide how to a5oach this problem, it may even be possible to use a gradual evolution process, building

off the success of one method to advance to more complicated designs (e.g. get hard-wire to work, before

testing BT). Another recent discovery, with regards to microphones is the use of fiber optics; Constantine

in particular is very excited about this discovery and will be compiling more information about it.

With regards to microphone testing, a document detailing the official Protocol of how the group will be

conducting specific experiments is in the works. This document will address specific testing procedures,

data collection, analysis, documentation, and reporting requirements. Once the details on how the

prototype will be set up are in place, this document will be refined and placed in a proposal before testing

begins.

For the future, the group, we will be coming up with strategies to partition the system level design and

allocate resources to the partitions. Additional requirements for the component designs are to be

generated as they become necessary. A portion of the group will be generating the testing procedure for

the audio-to-on screen text in a clean room test and formalizing it into a written proposal, in addition to

the microphone testing proposal previously stated.

Week 4

This Week’s Plan:

The plan for this stage of the project was to split the group into two research teams. One team, headed by

Luke, was to focus on studying up on the different aspects of acoustics by reading assigned chapters of

“Engineering Acoustics” and “Handbook of Engineering Acoustics.” These are textbooks the group found

that talk about many of the things we will need to know in order to understand how to properly test our

microphones. The second team, headed by Constantin, was to refine the selection of microphones the

team is going to use in the tests.

Major Issues:

Major issues were encountered when trying to find specific product information on microphones. The

best cites for getting detailed information were most commonly musical acoustic cites. Even with the

specifics for each microphone, it is expected that the primary way the team will figure out which

microphone the project requires will be through testing, but we have to purchase the microphones in the

first place in order to test them.

One possible solution the microphone team has been focusing on is to take a microphone with known

abilities that we already have and obtain its specifications and compare them to those we found in our

Page 142: Google Glass Project

133

research. The team lead is attempting to get the specifications for I-pad microphones and Android

microphones directly from the manufactures due to their specifications not being publicized.

This Week’s Progress:

This week, after splitting into respective teams, a good amount of work got done. For the textbook

research team, most of the work was brushing up individually on what acoustics are and how they can be

applied to Engineering. Each team member received and read their respective sections. The microphone

research team’s primary mission was to come up with a list of microphones that the group can look at and

decide if they can be used for our project.

The research team’s work done this week consisted of research on sound analysis software, finding an

open source program called “Praat” that appears to be capable of all the analysis needed for our project.

The team will be looking into this software and attempt to integrate it into the design. Additional info was found regarding; bandpass filtering, calibration and calculations for microphones, noise emission

characteristics of machines, as well as some research being done on shapes of microphone input and

sounds in buildings and rooms.

For the microphone team, a list of possible microphones and their specifications was created. The list is

currently being refined by the team head and will be presented to the group as a whole at the next meeting

to make an official decision on which microphones the team will focus on testing.

The focus for this upcoming week will be to refine our microphone choices based on the benefits

discussed in the acoustic readings. This will result in knowing which microphones should be purchased.

In addition to microphone selection, this week as a group we will refine the testing, recording and

analysis procedure for the microphones. The group will be looking into possibly ordering selected

microphones through the electronics support lab. We will be designing the experiment from the previous

paragraph and work on requirements for the other components. If time permits, begin/continue the

research how to automate and display the speech to text process, otherwise this will be moved to the next

week.

Week 5

This Week’s Plan:

This week’s plan for our project was to re-organize the teams and continue focusing on two different

areas. Team 1(Praat/Experiments Group) focused on writing experiments for the group to perform and

researching the Praat software. While team 2 (Microphone Group) focused on filling in numerical data for

the microphone criteria. This included finding the a Hz range for the microphones for a conversation

between two people over a normal distance (2-5 ft). Additionally, this team was tasked with finding a

definition for sensitivity, and a standard for the sensitivity the microphone we choose will need to meet.

By the end of the week, the goal was to specify what was critical in the criteria table while expanding the

list of microphones we have to choose from.

Major Issues: The largest issues we encountered this week was with the research into microphone sensitivity: When

researching the meaning of “microphone sensitivity”, it was difficult to find a physical meaning for the

given specifications of microphones. The negative decibel numbers correspond to the amount of gain

required by the preamplifier before it gets converted in the analog to digital converter (ADC). The

difficult part was determining what that means for the project.

Page 143: Google Glass Project

134

After discussing this dilemma with a computer science major and percussion expert (Shaun Davenport), it

was found that the sensitivity number corresponds to the minimum decibel reading that the microphone

can pick up. This information should help us determine the sensitivity standards the microphones need to

meet. However, no literature documentation was discovered that backs up this interpretation.

This Week’s Progress:

Team 1 created three new experiments; L/R Comparison, Read Time and Speech Time: L/R Comparison

is designed to check the feasibility of using the microphones to determine the relative location of the

sound source. It will be done by having two microphones set on either side of a sound proof block and

recording a sound that is positioned at set angle and distance from the block. The sound will be recorded

and compared using the Praat software to compare their average intensity levels.

Read and Speech Time research was used to determine how long the system should take to receive audio

and convert it into text for the display. The “Read Time” determines how long the text will remain on the

display. This test was based off of the concept Mr. Menon suggested at the last advisor meeting. The test

is conducted by watching shows or movies, and recording the data from the subtitles. Data recorded

included; how long the test was displayed and how many words were displayed. The “Speech Time”

research was used to measure how long it takes the average person to speak sentences of different lengths.

The data pulled from these two tests will factor into the latency time allotted for the program.

Team 1 also researched the Praat software learning how to create and interpret spectrograms, intensity

graphs, and pitch. The team learned how to write scripts in the Praat environment to introduce sound clips

to the program, allowing for automatic sound processing. The team determined how to divide the sound

between general and silent (blank) sound to increase the accuracy of the readings.

This week team 2 created the criteria for the sensitivity and the Hz range of the microphones. The best Hz

range for our purposes was set to a minimum from 60Hz to 15,000Hz. This frequency found to be the

range of the average humans’ hearing capabilities. As for the sensitivity, it was found that the “minus” dB

for the microphone specifications stands for the minimum required dB to be able to convert the audio

waves into electric output. After some consideration of the reduction of the sound pressure level over

distance, the group set the minimum sensitivity for microphones to -40 dB.

Next Week:

For this coming week, the group will finish and refine the experiment procedures. Conduct experiments

that do not require the microphones. Attempt to start ordering microphones through the electronic support

lab. In addition the group will start doing research into requirements other components. Also, in lieu of

the free dB meter applications on a cell-phone, which tend to be inaccurate, we will try to acquire the use

of a sound level meter from either the Gleason Center or the WFIT radio station for use in the

experiments.

Week 6

This Week’s Plan:

The plan for this week was to finalize microphone requirements, search for microphones within specific

requirements and contact the electronic support lab about ordering them. More data was also going to be

collected on the read and speech time research/experiments. The group also planned to learn more about

the Praat software, namely spectrograms and intensity analysis, and how programming worked on the

android and computer platforms. Finally the group planned to update the Gantt chart and plan future

tasks.

Major Issues:

Page 144: Google Glass Project

135

Microphones were not ordered this week due to the fact that the group wanted to expand the list of

microphones that met last week’s specifications before moving forward. This resulted in having to push

back the read/speech time research in order to get back on track with the microphones.

This Week’s Progress:

During this week the Microphone group focused on multiple tasks including; turning the microphone

specs into requirements, researching programming for android and computers for future audio recording

and translation, finding more microphones that adhere to the specifications created in the previous week,

and worked on creating a decision matrix for the microphones. The group worked on updated the Gantt chart to reflect what the team has done, and revising it to

compensate for new experiments that were created. Additionally individual components of the project

were further broken down in order to make them easier to understand and work on. The updated chart

now goes to November 4th. A task list was also created to help organize the team and to provide more

guidance on what needed to be done.

It was found that spectrograms are a great way to visualize sound. It shows frequency, amplitude (energy)

and the time all in one convenient graph. The group will primarily use spectrograms to examine the sound

filters effectiveness in the future. The group also learned how to generate the spectrograms in the Praat

software. Praat software intensity graphs were also researched with more of a focus on generating the

graph and specific procedure of get the max, min or average by select the time steps. Praat also allows the

use of “silent threshold” which eliminates ambient noise, such as the microphone’s internal acoustics, that

would otherwise interfere with the sound quality.

Next Week:

For this coming week the group will be researching the dictation and filtering software and will generate a

list of requirements for each. The group will also be ordering the microphones and searching for a venue

for the microphone experiments. If there is time, the group will also start looking for software that meets

the requirements generated. If the software does not advertise the desired information, the company will

be contacted through phone or email for more specifics. If the information still cannot be obtained

experiments will be created to find the information empirically.

Week 7

This Week’s Plan:

The goal for the last two weeks was divide the group into three research teams; one focusing on filtering

software another on dictation software, and the last looked into wrapping up the speech to text latency

testing. The filter research group, comprised of Brandon, Cheng Lu and Shaopeng Han, was specifically

tasked with doing research into acoustic filtering and generate a requirements list for the filtering

software. The dictation software group comprised of Coleman and Jessen, was tasked with coming up

with criteria for the software and compiling a list of possible providers. The last group was headed by

Yanglong Lu, with assistance from Darius. In addition to these new teams, the microphone group was

carried over to finish up ordering devices.

Major Issues:

Finding requirements for dictation software is rather difficult due to the absence of numerical values.

Therefore the sorting for the dictation software has to be evaluated with a side by side comparison.

Testing or discussion may be required to create hard benchmarks in this regard.

This Week’s Progress:

In the last two weeks, the filtering team researched specific filtering techniques and found that the main topic of interest was called “Speech Enhancement”. This field of signal processing consists of three main

branches; (1) noise reduction, (2) dereverberation, and (3) source separation. All of these branches have a

Page 145: Google Glass Project

136

tradeoff between speech intelligibility and sound quality. Furthermore, research into different methods of

source extraction for enhancing the speech signal in a noise rich environment provided multiple results.

Which method is the most suited for our needs should be taken into consideration in our next meeting.

From this initial research requirements were generated for the filtering software.

The dictation software group collected a number of software dedicated to dictation and arranged them in a

spreadsheet. The listed software/apps were afterwards evaluated and comments were created to denote

their usefulness. This was just a basic sifting, once the specific requirements have been defined for the

dictation software through testing or discussion, the table can be adjusted. The group will need to come

up with a way to compare the dictation software available. It is proposed that the group take the results of

speech to text latency tests and compare them to similar tests with individual dictation software. This and

an analysis on how each software can be manipulated to fit the groups needs will most likely produce the

criteria needed to narrow down the choices.

The microphone group talked with Dave in the electronic support lab and gave him the microphone

requirements and a few of the microphones we found. He is also tracking down a decibel meter, and

might have found a good one for $35 in Palm Bay. The group should get a response from him on Monday

with his findings.

The read time research team collected the data from different videos which include; movies, TV shows,

public speech and college lectures to determine the proper duration, time and amount of words for the text

shown on the display of the Google Glass. And the latency time of the text is suggested.

Next Week:

The focus for the coming week is to refine the requirements for the dictation and speech enhancement

software, begin gathering software that meet those requirements, and continue researching

programming. If we receive the microphones this week some experiments will be conducted. The group

will discuss the source extraction method for enhancing the speech signal. Based on this discussion, the

group will decide which method should be used in our project.

Week 8

This Week’s Plan:

The filtering research group was tasked with continuing their research into different filtering techniques

and looking into different programs that use them. The microphone research group was tasked with

ordering the microphones. The dictation research group was tasked with finding more dictation software,

generating a testing procedure for the dictation software, and begin testing the dictation research.

Major Issues: None.

This Week’s Progress: The microphone group talked with Dave and ordered the cheaper of the microphones on the list of

microphones with the best specs, and discussed the reasons for the difference in price between the

different microphones. Dave said that the big differences would be the built-in shielding and insulation

from electronic noise. Moving forward, while waiting for the microphones, the group will research

methods for insulating them.

The dictation group continued researching dictation software. The group got the apps from Chrome Store

and Google Play. They tested the apps in accuracy, latency, usability and rated each apps from 1~5 points.

They were tested by using a microphone headset connected to a computer and speaking, then the group

member rated the experience. They also took other users’ reviews into account when they rated these

software. In the future, the dictation software testing procedure will be formalized and a standard microphone will be used in testing them.

Page 146: Google Glass Project

137

The filtering research group looked into the three main types of beamforming; Acoustic, near-field

holography, and spherical. beamforming is a method of sound locating using microphone arrays. It was

found, that out of the three, the acoustic beamforming would be best suited for our project and an open

source software, beamformit was found that could be used.

Next Week:

The dictation group will use the experimental procedure they generated to begin testing dictation software

that they found and select one (if possible). The filter group will continue researching filtering techniques

and software that can be used. The programming group will continue learning how to integrate the

systems together.

Week 9

This Week’s Plan: The plan for this week was to continue researching into the three different categories: dictation, filtering,

and programming. Unlike the previous week, Brandon switched to programming due to the impending

need to have a basic program done by the end of the semester, where implementing the filtering aspect is

not pending for this semester. Alex is still working on filtering. The dictation group was tasked in

generating a testing procedure for the dictation software and rule out software that could not be used for

the project. The programming group now consists of Darius, Brandon, and Luke. Luke is working on

programming for Android devices, Brandon is working on programming for the Raspberry Pi and other

alternatives to mobile phones, and Darius is assisting both with their research.

This Week’s Progress:

This week the test protocol for evaluating the dictation software was created and uploaded on the Google

drive. Pretesting was completed to determine which programs have the best chance of meeting the

group’s needs.

Filtering software group continued to research into different filtering software methods and software

classifications.

Brandon began research on the platform capabilities of the Raspberry Pi (RPi) and what software would

be compatible. He also looked into using PocketSphinx, open source voice recognition software that was

created by Carnegie Mellon University, and discovered that it has been used by others on the RPi for

similar purposes. The dictation group has also found another way to run dictation software using the RPi

is to use software called WineHQ. WineHQ enables the Linux system to run Windows Programs, such as

Dragon Dictation, which, based on initial findings was evaluated as the most accurate and user friendliest

dictation software for the lowest price.

Cheng and Yanglong worked on finding more dictation software that can be used on a mobile phone.

Software with file type input and output were added into the spreadsheet that was created last week. The

programming team has found it easier to manipulate or use the data produced from the dictation software

if it can be accessed from a file. For this reason, the dictation team expanded their search criteria and the

programming team is making efforts to create a system for accessing and passing the information from

these files once the software has been chosen.

Luke continued research in Android programming, focusing on sharing information between apps and

saving files to storage. Both are doable with Android, but implementing reading and sharing of files in

real-time for translation might be challenging.

Major Issues:

Page 147: Google Glass Project

138

During the research process of “Real-Time Subtitles” it was discovered that there was a project done over

the summer that focused on something similar to our project. This project consisted of a real-time

language translation system using augmented reality glasses. After discovering this, another

comprehensive search for related systems was conducted by Brandon which led to the discovery of

several patents that claimed to do the same thing. This would significantly reduce the team’s ability to

patent their progress. The similar project, as well as the patents found, will be discussed at the next

advisor meeting.

The microphones on order through the electronic support lab have still not arrived. It has been decided

that if this is still the case by the end of this week, cheaper or more readily available microphones will be

purchased and prepped for testing while we continue to wait.

Next Week:

For the coming week the dictation research will be concluded and preliminary dictation software will be

selected. The programming and filter research groups will continue their work. If the dictation group

gets done early they will either aid the programming or filter research group with their work or (if

microphones have been received) they will begin conducting the microphone tests. By next week if the

microphones on order do not arrive, cheap test versions will be purchased so that testing can at least get

underway and any testing issues can be identified and addressed as well as allowing testing procedure

refinement. As for the Dictation software for the RPi, we need to do some testing to see if the RPi can

handle the WineHQ software to run Dragon Dictation.

Week 10

This Week’s Plan: For the week of the November 4th the plan was to get some basic microphones purchased and obtained

from a local vendor so testing of the dictation software. The purchase of the microphones would also

allow the microphone test to be conducted. Kyle and Constantin were put on the task of testing the

dictation research and initial microphone testing with Han, Cheng Lu and Yanglong Lu in charge of

conducting the audio-to-text and L/R comparison test. Luke was tasked with continuing the android

programming and looking into a new voice recognition SDK. Luke and Yanglong Lu were also tasked

with searching for alternatives to the Google Glass. Brandon was tasked with obtaining a Raspberry Pi,

learn how to program it, and to contact Will Powell.

This Week’s Progress: Brandon emailed Will Powell, the creator of the comparable product, on November 5th and is awaiting a

response. A Raspberry Pi was purchased from the electronics support lab on Nov. 5. An RPi log book

was created to document work done on the RPi so that, if another member of the team joined the

programming effort on the RPi, they would be aware of all that has already been done and how. The

major achievements on the RPi were; installing Raspbian (the Raspberry Pi operating system based on

Debian), getting the RPi to recognize and use a USB microphone, and installing PocketSphinx on said

RPi. Although PocketSphinx installed successfully, it is currently not recognizing the USB microphone as

the audio input. Luke looked into a new software called Ispeech that is designed for integration with application

programming. So far it looks promising, as they have a lot of good documentation on their web site for

programming. Luke also found several potential Google Glass replacements. The most promising are

from the company GlassUp. From which the group can obtain one in January. It costs $300-$400 with or

without camera, and can be connected to any mobile device and controlled by apps. Alex, Cheng and Yanglong conducted the audio-to-text and L/R comparison experiments. After testing

different dictation software, they found that the audio-to-text software still have some shortages and the voice signal can only be accepted within 3 feet. On the other experiment, one consisting of a block with

two microphones attached with a person reading a paragraph at different distances in front of the

Page 148: Google Glass Project

139

microphones. The microphone angles were changed from 0-180 with intervals of 10 degrees. The data

was collected for further filtering research.

Constantin and Kyle worked on setting up the Pocketsphinx software on a mobile platform and prepped

for initial microphone/dictation software testing scheduled for Monday Nov 11. Stand in microphones

were purchased for testing and the ordered microphones are scheduled to come in on Tuesday Nov 12.

Major Issues:

Behind on Testing

Changing directions on the output platform.

Next Week: The plan for the coming week is to continue the Android and RPi progress. For the RPi the focus will be

on getting PocketSphinx to use the USB microphone and to improve the audio being captured by the

microphone. The Microphone testing group will analyze the data they obtained and perform any

necessary follow up experiments. The dictation group will be reallocated into aiding the programming

effort.

Week 11

This Week’s Plan:

For the week of the November 11th the plan was for the microphone testing group (Cheng, Lu, and Han)

to finish all of the microphone experiments, analyze the data, and recommend changes to the

requirements based on what was found. Kyle was tasked with reading and summarizing the patents

related to the project and aiding other group members with their tasks. Constantin was to work on

figuring out how to mount the new microphones on the different devices and researching how to use them

with the RPi. Luke was assigned with continuing the android programming and Brandon was tasked with

trying to get PocketSphinx to work on the RPi.

This Week’s Progress:

Brandon got PocketSphinx to work on the RPi earlier than expected, so he worked on getting the output

to display on an external display and began testing PocketSphinx. He was successful in displaying the text

from the RPi via secure shell (SSH) to an iPad. It was found that PocketSphinx on RPi has a long latency

time (around 10-20s), the accuracy was poor and the output was cluttered with runtime diagnostics. It is

thought that the accuracy can be improved by increasing the word library of PocketSphinx. The runtime

diagnostics could probably be cleaned up by manipulating the source file and su5essing their output.

Further investigation will be conducted to figure out why the latency time is so long.

Microphones arrived and were given to Constantin to figure out a system for making the microphones

work with the RPI or any other processing device. It was established that the group will have to develop a

small circuit board and mount it to the microphones in order to get them to interface with other devices. A

basic electrical circuit layout has been acquired.

Luke is working with the iSpeech dictation software on the android. Currently, he has it installed as an

editable functioning sample that can record audio with the embedded microphone on the cell phone and

print it as text on the screen. The main focus now will be to integrate it into an app he has created and

interface with an external microphone.

Patent research was completed and the summaries will be included the week’s presentation.

Alex, Cheng and Yanglong conducted the microphone experiments. The experiments included the L/R

Comparison and the Audio-to displayed text. By using Dictanote to collect the audio source, and then

comparing with sample text, we found the accuracy of the speech to text primarily dependent on the

Page 149: Google Glass Project

140

distance between the microphone and the acoustic source. For the L/R comparison research; the two

microphones were used in a simulation incorporating real world acoustic situations. These situations

included sources coming from varying angles. By analyzing the decibel data selection from the right and

left it is possible to determine source direction. Furthermore, the difference between the vibration curves

of the right and left can be displayed on pratt, possibly aiding further source separation objectives.

Because the two microphones are not absolutely identical we added a correction factor to the each result

which made the difference in 90 degrees turn 0. The results graph created confirms our assumption that

we can use two microphones to determine the location of the sound source.

Major Issues:

Sphinx Latency

Microphone circuitry

Next Week:

For the coming week the team’s primary focus will be compiling all of our documents and creating the

interim report and the PDR. Other tasks that will be worked on are; continuing the android programming,

figuring out how to improve the output from PocketSphinx on the RPi. And talk to the electronic support

lab to work on the circuit for the microphone as well as considering creating a housing for the microphone

using the 3D-printer.

Week 12

This Week’s Plan: For the week of the November 18th the plan was to take a portion of the time for the group to begin

compiling the documentation and the rest of the time on tasks. The testing group was tasked with

rerunning some of the past experiments after an issue was found in the procedure and running a new

experiment. Constantin was tasked with building the microphones with the help of the electronics support

lab. Luke was tasked with continuing his work with the android programming. Brandon was tasked with

creating a demonstration of the RPi prototype, and determining methods for decreasing the latency and

increasing the accuracy. Kyle was tasked with aiding the others with their tasks.

Major Issues:

None.

This Week’s Progress:

Since the group has a working prototype, more focus this week was devoted to refining the project

documentation and combining it into a preliminary project report.

Brandon created the demonstration and began researching why the PocketSphinx was slow and

inaccurate. After some searching methods were found to decrease the latency time through various

settings in PocketSphinx. The inaccuracy appears to be partly due to the setting and partly due to the

limitations of the current speech dictionary (SphinxBase). To remedy this it was suggested that new

speech models (supplied by CMU) be used or SphinxTrain be used to create a custom speech library.

Shaopeng, Cheng, Yanglong conducted the tests for the microphone. Since last week, they analyzed the

data and found that there was not sufficient evidence gathered for a conclusion. For audio to test research,

they conducted another two runs, which represent 3 and 3.5 feet in order to get a better data analysis. For

the L/R comparison, they also conducted the experiment at 2 and 3 feet. Shaopeng also created a new

experiment to test the feasibility of the microphone in different environments. This week, they started to

do the final report, Yanglong and Cheng Lu wrote about the procedure, results, recommendation and

conclusion. Shaopeng wrote about filtering research and Google glasses background research.

Page 150: Google Glass Project

141

Next Week:

For the coming week the group will continue to dedicate half of its time in compiling documentation and

generating the interim report/presentation. Brandon will test the various latency reduction techniques and

speech models that were found this week. Luke will continue building the app for the android. The rest of

the group will aid them, handle any problems that may arise during the week, and continue to work on the

report.

Week 13 This Week’s Plan: This week the plan was to get reorganized and divide the team into three sub-groups: filtering,

programming, and experimenting. Once the groups are established, overarching tasks will be generated

for the groups. Some other tasks to be addressed were to establish meeting times for the semester, do the

documentation for items done over the break, order the microphone, and other administrative items.

Major Issues: None.

This Week’s Progress: The group was divided into the three subgroups and their primary directive and team compositions are as

follows:

The filter group consists of Brandon and Han and is tasked with finding filtering programs capable of

being ported to the android platform and incorporated into the overall android application. This group will

be given a week to generate a comprehensive list of programs that could be used and if no existing

programs are found, they are to come up with alternative options. After their task is completed this group

will be repurposed to aid the other two groups with their tasks.

The experimental group consists of Constantin and Yanglong Lu; who has been tasked with the creation

and execution of experiments and tests to examine the attributes of the project and validate that the

requirements were met.

The programming group consists of Luke (Lead), Kyle, Darius, and Cheng Lu; who is to work on the

creation of the android application that will integrate all of the software components of the project and

interface with the Google Glass.

Also, preliminary experimentation/research was performed with the Google Glass to get a feel for what

capabilities it has, what applications are currently available for it, how long the battery lasts, as well as

other minor details. So far, there is not a whole lot of applications ready for use. The battery on the Glass

device lasts about 4-5 hours with intermittent use.

The group established Mondays from 3:00 pm until the advisory meeting and Wednesdays from 3:00 pm

until senior design as the primary meeting times with Mondays and Wednesdays after 6:15 pm as

auxiliary time slots. The microphone was ordered through Mrs. Jawnee and is expected to be shipped on

Monday.

Next Week: The filtering group will do as outlined previously. The experiment group will generate a series of

experiments to assess the capabilities and limitations of the Google Glass. They will have a week to create

and finalize the experiments and an additional week to run the experiments and compile their findings.

The programming group will divide and conquer the different aspects required of the application. So far,

the application needs to interact with an external microphone, have a simple to use interface to control the

entire app, implement speech to text software, and communicate all of this to the Google Glass.

Page 151: Google Glass Project

142

Week 14 This Week’s Plan: The plan was for each of the groups to work on their respective tasks for last week. The filtering group

was tasked with finding filtering programs for the project. The experimental group was tasked with

creating experiments for the Google Glass and running them when they were ready. The programming

group was tasked with looking into how to do the different aspects of the programming required for the

project and programming them.

Major Issues: None.

This Week’s Progress: The filter group created the requirements of being able to be installed onto the android and integrated into

an app, have low latency time and both noise reduction and speech isolation. The group generated a comprehensive list of SDKs and programs on the market (of which four meet the requirements).

Companies that did not have a trial version or price were contacted on 16th of January. Since the group

finished its primary directive early the group switched to aiding the programming group with their tasks.

Once the team gets a functioning version of their speech recognition app started the filtering group will

work on integrating the speech enhancement software (if necessary). The group discovered there is a

application available for the Google Glass that allows the user to send text to the Glass via a smart phone

where it is used like a sticky note. Through more in depth research it was discovered that the creator of

the app has expressed that he is not willing to release his source code, but the app does show that it is

possible to send text to the Google Glass via Bluetooth. If available software are not feasible, the filter

group will implement plan B. Plan B is, Programming our own app for filtering which may include those

basic functions, speech enhancement and sound subtraction. So far, the filtering group has found some

samples of filtering code which is similar to the Apple “adaptive high pass filter” example. This code may

be considered by our group and possibly modified to meet our goals.

As far as programming is concerned, the different parts of the group were to begin the learning process of

coding for their section and come up with a preliminary structure for the code. On the Glass end, it was

found that there are currently two programming API’s for making apps for the Google Glass: the Mirror

API and the GDK. The Mirror API is currently very limited in its functions. It is meant to keep all of the

code running on the mobile device and only print the results to the Google Glass screen. Although this

function would be preferable for our purposes, it can only interact with timeline cards and notifications so

far. The GDK is the more standard app development environment that enacts most of its code on the

Glass itself.

In terms of app structure for the Google Glass, there are three different formats currently: static cards, live

cards, and immersions. Static cards just insert a card object into the past section of the timeline, live cards

are updatable and inserted in the front of the timeline, and immersions are used to make complex user

interfaces. This application does not need a complex user interface and it needs to be able to update the

card. This led to the choice of a live card, which also has several options. It can be a high frequency card

or a low frequency card. A high frequency card can update many times a second and is mainly used for

graphics and video; however, a low frequency card can only be updated once every couple of seconds.

This is a tradeoff that will have to be researched further.

The test group conducted tests regarding the sensitivity of the Google Glass with and without the

Headphones. As it turns out, there is no difference in audio quality or strength of the audio whether the

headphones were plugged in or not. Also, it was observed that in both scenarios, the microphone pattern

appears to be omnidirectional. The battery life of the Google Glass was tested as well, and after 12 hours

of wearing the Google Glass with minimum usage, the glass lasted the entire day with 26% battery life remaining. All tests have been recorded using the Google Glass and videos saved for future presentations.

Page 152: Google Glass Project

143

Next Week: For the coming week the filtering group (which is now the floating group) will continue to aid in the

programming.

The programming group moving forward will focus on creating a sample application that can display to

the Google Glass using a live card of both high frequency and low frequency. After reaching this goal,

integration with the microphone will be attempted followed by the speech to text software.

Week 15 This Week’s Plan: The plan this week was for the programming group to get a better understanding of programming for the

Glass and android via tutorials and research. After which they were tasked with applying the acquired

knowledge towards the projects goals. The test group was assigned to documenting all of the tests that

they have conducted and test the new microphone. Yanglong Lu was temporarily moved to the

programming group for this week.

Major Issues:

None. This Week’s Progress: The filtering group has not heard back from the companies that they contacted. If they still have not

gotten a response by Wednesday of the coming week they will attempt to contact the companies again.

The first part of the this week was used to get the Android ADT bundle working on all of the

programming groups computers and to get everyone acquainted with where things were. From there the

group gathered guides for programming on both Android and on the Glass in Java. During this gathering

the group found the methods and intents used to implement voice to text within an android or Glass app.

The rest of the week was used working on implementing this code. At this point, the group expects to

have code ready for initial testing on the Google Glass by Wednesday with a functioning prototype of the

speech to text aspect of the app completed within the next two weeks.

Constantin worked on documenting the results of the previous week’s experiments and Yanglong Lu

worked on acquainting himself with programming. The microphone that recently arrived was tested to

verify its ability to work with an Android phone using a Samsung Galaxy SIII running “Voice Recorder”

app, this test was successful. Later in the week the test group was given the microphone and tried testing

it using a Samsung Galaxy SIV under the same conditions and it did not work. For this reason and the

looming deadline for purchasing the group has decided to purchase plan C, a microphone designed for

android phones that is being sold for $50 from Wal-Mart in Melbourne. The group will be

troubleshooting why the microphone is not working on the SIV and then testing both microphones in the

coming week.

Next Week: The programming group is being allotted two weeks to get a functioning prototype of the speech to text

on the Glass. After the prototype is complete, the group will take approximately three weeks to work on

debugging and optimizing as well as incorporating a user interface. From there the test group will take a

week to determine if speech enhancement software is needed. If additional speech enhancement is needed

the filtering group will work on implementing it by spring break.

Page 153: Google Glass Project

144

Week 16 This Week’s Plan: The plan for the programming group was to divide into 3 sub teams. Luke and Han were to work on

creating an apps independently that could be placed on the Google Glass. Zhu and Cheng Lu were tasked

with working on an app that uses the “Voice Typing” function to display texts as it is spoken. Brandon

and Kyle were tasked with working on the creation of an app that uses Google’s basic speech recognition

function. Brandon was also tasked with contacting the creator of “ToGlass”, an app designed to display

text for the phone on the Google Glass. The test group was assigned to test the new microphones and to

process the audio from the Google Glass experiments.

Major Issues:

None. This Week’s Progress: Brandon contacted Tejas Lagvankar, the creator of ToGlass and is currently awaiting a response.

Brandon and Kyle created a basic program that displays the text being spoken using Google’s basic

speech recognition function. They later improved the app using a separate source code to enable the app

to continuously listen and display text. The app will currently “chime” indicating it is ready to listen,

listen to spoken words, and then display the text to the screen. It will continue this process automatically,

while keeping the previously spoken text on the display until new text is ready. After some basic testing,

a bug was found where the app will freeze after a random amount of iterations. The team is currently

looking at causes of the problem.

Luke continued working on a rudimentary app that recognizes speech and prints to the screen of the

Google Glass. So far, he has the basic structure of the app including speech recognition completed and is

working on printing to the screen.

Constantin and Yanglong conducted the test for the external microphone, Plan C, which works well on

mobile phones. From the test result, the performance of the external microphone is better than the built-in

microphone on cell phones, and will be compared to that of the glass itself. The test group also tested the

dictation software Brandon wrote. The accuracy and the latency are excellent, but the software couldn’t

be triggered from 4 feet with the external microphone. Another issue is that it cannot work continuously

to show the text on screen.

Brandon discovered a new type of speech recognition on Android devices called “Voice Typing”, which

can print the text as it is spoken. Cheng Lu is trying to implement it into an app by using a Google Glass

emulator.

Next Week: The programming group has been making progress in their respective apps and is confident in reaching its

goal of a function prototype on the Glass and Android in two weeks. For the next week they will continue

on working on their respective apps. After the prototype is complete, the group will take approximately

three weeks to work on debugging and optimizing as well as incorporating a user interface. During this

time the test group will take a week to determine if speech enhancement software is needed. If additional

speech enhancement is required the filtering group will work on implementing it by spring break.

The test group will also continue testing any apps as they are developed, such as the software Shaopeng

and Luke built respectively as soon as they complete the coding. This group will take the performance of

the software into consideration when evaluating the necessity of filtering and speech enhancement

technology in the system. They will also be continually evaluating the prototypes, focusing on specific

system requirements and provide advice for optimization.

Page 154: Google Glass Project

145

Week 17 This Week’s Plan: The plan for this week included the programming group being tasked with completing the Google Glass

apps and adding in the continuous speech recognition. The test group was assigned to testing the current

version of the Google Glass app to find the range of the built in microphones.

Major Issues:

None.

This Week’s Progress: Since QSound Labs and Malaspina Labs, creators of QVoice and VoiceBoost, did not respond they have

been ruled out as filtering options. This leaves GritTec’s Noise Cancellation and bdSounds Speech

Enhancement Suite (bdSES) . If the testing group decides it is necessary to include the speech

enhancement, the bdSES will be implemented first because it it more user friendly on the programming

end of things and has a more comprehensive list of features to employ.

Luke continued work on the Live Card app which is nearly complete. The live card structure is in place as

well as the speech recognition function. The current obstacle is the integration of the two. Further testing

and debugging is required before the app is completely finished.

Shaopeng and Cheng continued upgrading the software ‘speechtotext’, which concentrated more on

figuring out the problem with the app closing upon navigating to it. By comparing this issue with a couple

of problems that others have had, we believe that the problem is mainly caused by the onActivityResult()

never getting called. So the next step is to rerun the software with new code and keep testing it after

inputting the new code. Currently it can achieve the basic function of the speech to text, and producing

that text on the screen.

The testing group focused on testing Alex’s application for the Google Glass. As it turns out, the

application is able to translate audio to text at a distance to up to 5 ft. with some limitations. Longer

sentences still seem to cause issues. Besides the testing, the test group is working on converting the audio

files of the video into “.wave” files, to be able to run it in the Praat software.

Next Week: The programming group has been making progress in their respective apps and is confident in reaching its

goal of a function prototype on the Glass and Android in two weeks. On Monday the two Glass apps will

be presented to the group and the app with the most promise will be chosen as the main focus. The team

will be adding any features that are not present from the other app to the main app. The group will take

approximately three weeks to work on debugging and optimizing as well as incorporating a user interface.

During this time the test group will take this week to determine if a speech enhancement software is

needed. If additional speech enhancement is required the filtering group will work on implementing it by spring break.

Week 18 This Week’s Plan: The plan for this week included the programming group being tasked with completing the Google Glass

apps, adding in the continuous speech recognition and implementing some new features. The test group

was assigned to testing the ambient noise threshold of the Google Glass app.

Page 155: Google Glass Project

146

Major Issues: None.

This Week’s Progress: Brandon was tasked to research and code a method of saving the text data to a file that could be retrieved

in the future. He completed a code that currently works with the android app and should work with the

Glass app. During the coming week it will be added to the existing Glass code and tested. During his

research he discovered that the Wake Lock feature (a feature that keeps a device from going to standby) is

only available on the Google Glass via immersion cards. This led the group to changing the live card style

into an immersion card style. This conversion should take approx. 1 week to complete.

The programming group is working on finishing off the last challenges of making a complete functioning

app. It will then transition into making an immersion card app from the live card app.

It was recently discovered that GritTec’s Noise Cancelling software does not work on Android platforms,

although it states on their website that the software works on any ARM based platform. The bdSound

demo is currently limited to an echo cancellation software which is unable to run on the Glass. The

company was contacted requesting access to their complete SDK trial and the group is currently awaiting

a response. Due to these dilemmas, other companies that create speech enhancement were sought and

contacted and the group is currently awaiting responses from them as well. The group is currently looking

into programming their own speech enhancement software. Preliminary research into programming it for

Android has shown there is a significant learning curve that may be difficult to overcome without external

help. The group would like to consult the CS department (namely Dr. Shoaff and Dr. Crawford) for

recommendations on how to go about doing it.

The testing group worked on an experiment to test the app’s performance in a noisy environment as well

as a test to specifically test the application to measure progress throughout the development phase.

Next Week: The programming group has created an Glass app capable of real time speech to text, however it currently

does not run continuously nor does it have some of its extra features incorporated into it yet. The group

will take the next two weeks to remedy these problems and do any debugging necessary. During this time

the test group will take this week to determine if a speech enhancement software needs and make

recommendations to the programming group. If additional speech enhancement is required the a portion

of the programming group will begin working on it. If one of the speech enhancement companies respond

back with a trial of their software, the group will implement it by spring break barring any major issues. If

none of the companies respond back an initial estimate of five weeks will be required to get code working

for it, but due to the nature of the task the ultimate time is currently unknown. If the group fails to

implement the speech enhancement the consequence would be limitations of the Glass app’s range

(currently 3-5 ft.) and it would only be able to be used in a noise free environment.

Week 19 This Week’s Plan: The plan for this week included the programming group being tasked with completing the Google Glass

app. The test group was assigned to refine the testing the ambient noise threshold of the Google Glass

app. The filter group was to continue researching filtering techniques on android.

Major Issues:

None.

Page 156: Google Glass Project

147

This Week’s Progress: The Glass app is completed and functioning; however, the speech recognizer is in a separate activity and

is not interacting correctly with the live card service. As a result of this, it can only perform a single

iteration before it needs to be restarted and does not continuously display the text on the screen.

Preliminary results of the ambient noise threshold for the Glass app and research into filtering on android

indicates that the android platform already employs a broadband noise filter/suppressor. Broadband noises

are sounds from traffic, music and repetitive sounds like tapping. It is believed that it does this by

restricting the frequency range to that of human speech.

The testing group performed multiple tests on the app in a room permeated by broadband noises of

different types and levels to determine the effectiveness of the app and innate filtering software. Initial

results indicated that there is a drastic reduction in the Glass’s ability to interpret speech with too much

broadband interference.

Further research into both filtering and voice recognition revealed that the speech filtering must be done

during the “capture path” (when the audio is being captured by the microphone) since there is no known

mobile speech to text software on the market that can translate speech to text from an audio file. This

means the audio can not be recorded, filtered, and then ran through a speech to text software. Also

discovered in the research, were some functions in the Android SDK that enable the coder to use the

phones speech enhancement functions that the manufactures embedded in the phones hardware and

software. This, of course, relies heavily on the device being used and whether or not the manufacturer

incorporated these features. Research into which devices offer this yielded no concrete results, just a lot of

speculation by programmers and a couple of technical papers discussing how some mobile device

manufacturers employ speech enhancement techniques.

Next Week: The group will take the next week to remedy problems with the Glass app and do any debugging

necessary. Similar to last week if one of the speech enhancement companies responded back with a trial

of their software, the group will implement it by spring break barring any major issues. Since none of the

companies responded back as of yet some members of the group have looked into creating our own

speech enhancement an initial estimate of five weeks will be required to get code working for it, but due

to the nature of the task the ultimate time is currently unknown. If the group fails to implement the speech

enhancement the consequence would be limitations of the Glass app’s range (currently 3-5 ft) and it

would only be able to be used in a noise free environment.

To combat the single iteration code problems, research into doing the speech recognition in the live card

service itself is being conducted. Also, since the voice typing cannot be implemented in a live card, the

programming subgroup is constructing a more complicated immersion card that can implement the Glass

app code and utilize voice typing.

Week 20 This Week’s Plan: The plan for this week included the programming group being tasked with completing the Google Glass

app, which will remain the priority for this group until it is completed. Within the programming group

the subtasks were for Luke to debug his app and for Shaopeng, Cheng, Zhu and Tong to create an

immersion card skeleton for Luke and Shaopeng’s app to be migrated to. If either sub group completed

their portion early they would begin working on multithreading it. Multithreading is what Dr. Crawford

had mentioned would allow the app to filter the audio and pass it to the voice recognition service. The test

group was assigned to get context for their ambient noise threshold experiment and to determine what

environments would be plausible for the Glass to be deployed in. The filter group was to continue

Page 157: Google Glass Project

148

researching filtering techniques on android, determine some simple and effective techniques to use and

deploy them.

Major Issues:

None.

This Week’s Progress: The filter group determined a couple of promising filtering techniques. These included a simple filter that

would only allow frequency between set values to be passed through. A second was called a “Bell Filter”

which applies a gain to frequency within a set bandwidth of a set average desired frequency. Both of these

examples gave a basic algorithm to try which requires some programming, converting the pseudo code

into android java. For testing purposes it was determined that a simple app should be developed that

would record audio, apply a given filter/s and save both the original and filtered audio for playback and

analysis in Praat. This app is near/ at completion (pending some testing and a bit more research into

converting the audio files into bytes and back again). The programming group continued work on the immersion card app skeleton and fixing the issues with

the current app. So far, some headway has been made, but the app still needs some restructuring before it

is completed.

Next Week: The programming group will continue to work on remedying problems with the Glass app and do any

debugging necessary. They will also work on converting the current live card version of the app to the

immersion card and work on the multithreading. The programming group is aiming to get the Glass app

function within 2-3 weeks with the multithreading and filter integration taking an additional 2 weeks. The

filter group will complete the filter testing app and will begin implementing and testing the different

filters. The filter group is confident that it can get some rudimentary filters working within two weeks.

This does not include tying it into the Glass app using the multithreading. Provided that no new concepts

arise that need testing, the test group will aid the filtering group with their work. If all goes as expected

this will put the completion date for the app in the beginning of April, allowing for a week of testing and a

week buffer.

The programming group gave Dr. Crawford a copy of the current app code. She will look over it during

the next week while the programming group continues work on the app and give input on areas of

improvement.

Week 21 This Week’s Plan: Due to it being Spring Break, the group took it a little easier this week and work was considered optional.

The overall plan was for each of the sub times to continue working on the respective goals. For

programming it was to complete the app. For filtering was to complete the test bed for the software and

figure out how to program the filters. For the test group it was complete any testing and any related paper

work.

Major Issues:

None. This Week’s Progress: Plan for this week was pretty much the same as the previous week. The programming group's primary

task was working on completing the Glass. This included debugging the app, debugging the immersion

Page 158: Google Glass Project

149

card, and researching how to implement multithreading for integrating the filter in the future. The filter

group was tasked with debugging the filter test app, finding time domain filters to test, and implementing

filters. The test group was tasked with getting ambient noise levels for conditions that that Glass would be

used in and testing the current Glass app in those conditions.

Next Week: The group’s future plans remain the same as last week with some minor changes underlined and is

provided below.

The programming group will continue to work on remedying problems with the Glass app and do any

debugging necessary. They will also work on converting the current live card version of the app to the

immersion card and work on the multithreading. The programming group is aiming to get the Glass app

function within 2-3 weeks with the multithreading and filter integration taking an additional 2 weeks. The

filter group will begin implementing and testing the different filters. The filter group is planning on

getting some rudimentary filters working in approx. two weeks. This does not include tying it into the

Glass app using the multithreading. Provided that no new concepts arise that need testing, the test group

will aid the filtering group with their work. If all goes as expected this will put the completion date for the

app in the beginning of April, allowing for a week of testing.

Week 22 This Week’s Plan: Plan for this week was pretty much the same as the previous week. The programming group's primary

task was working on completing the Glass. This included debugging the app, debugging the immersion

card, and researching how to implement multithreading for integrating the filter in the future. The filter

group was tasked with debugging the filter test app, finding time domain filters to test, and implementing

filters. The test group was tasked with getting ambient noise levels for conditions that that Glass would be

used in and testing the current Glass app in those conditions.

Major Issues: None.

This Week’s Progress: The filter group completed debugging the app (no known problems now) and has been working on

implementing the bell filter. The code was created and added to the filter test app, but needs some

debugging. The current problem with the filter is that it is not either reading in the audio data correctly or

is not writing it out properly. This is will be remedied in the coming week. The filter test app records

audio and will apply the user specified filters. Each filter's results, as well as the original recording, are

available for playback through the app for immediate qualitative comparison. The files are also saved to a

folder on the phone and can be transferred to a computer to be processed in Praat to view the spectrogram

and other information. A couple of new pseudo codes were found, which might be able to be used in the

code, though have to be tested first.

The coding group is still focused on the debugging between the speech recognition and the immersion

card. So far, the immersion card is being built. The group is also writing a cyclic program target in the

non-continuous function, and the speech to text can accomplish the major function. Tomorrow, the group

will debug the glass then import the immersion code. If it works, the group will combine those two codes

and import a cyclic program. The live card app is also coming along, and should be completed by the end

of next week.

Page 159: Google Glass Project

150

The test group collected data on baseline dB readings for different locations on campus, such as; the

dining hall, next to Babcock, outside OEC, and the OEC lounge. The group ran into some issues while

attempting to test the Glass app directly in these locations. Without a connection to a wireless network or

a strong signal from a cell phone specifically paired to the Google Plus account of the Glass, the app

cannot access the speech to text library. This was a previously identified issue; however, this is one of the

first major encounters with the issue while testing. It is assumed that this connectivity issue will continue

to play a role in other similar locations like in a car and public transportation, where signal strength can be

erratic.

Next Week: The programming group will continue to work on remedying problems with the Glass app and do any

debugging necessary. They will also work on converting the current live card version of the app to the

immersion card and work on the multithreading. The programming group is aiming to get the Glass app

functional within 1 weeks with the multithreading and filter integration taking an additional 2 weeks. The

filter group will begin implementing and testing the different filters. The filter group is planning on

getting some rudimentary filters working in approx. a week. This does not include tying it into the Glass

app using the multithreading. Provided that no new concepts arise that need testing, the test group will aid

the filtering group with their work. Once the filters have been integrated into the test app the filter and test

group will spend one week determining which would be the best universal filter to use in the final Glass

app. If all goes as expected this will put the completion date for the app in the beginning of April,

allowing for approx. a week of testing.

Week 23 This Week’s Plan: With the core speech to text program complete, the program group was subdivided into new tasks. Luke

and Alex were tasked with enhancing the user interface, and Cheng, Zhu and Constantin were tasked with

looking into microphones that can interface with the google glass. Brandon was to get the Filter Tester

app working and implement some filters. Part of enhancing the user interface was continuously testing the

user friendliness as adjustments were made, Yanglong and Kyle assisted Luke and Alex for this.

Major Issues: None.

This Week’s Progress:

Brandon was able to get the Filter Tester app working with the aid of Tong. He was also able to

implement a couple of filters. The filters that were implemented were the Bell Filter, Low-Pass Filter

(LPF) and Butterworth LPF. All of the filters were implemented in the time domain using a difference

function and coefficients. The filters functioned but did not work well. This is due to needing to figure out

how to derive what the Bandwidth is. Another issue that was found was that the filter took approximately

10-15 seconds to process 3 seconds worth of audio. Tong was able to fix this delay by changing some of

the read/write coding methods in the code reducing the filtering time to approximately a second.

The programming group has been focused on sprucing up the code. The timing has been changed to 2

seconds on the speech recognizer delay. All in all, after 2 seconds, the interface will move to the speech to

text. 2 seconds after not receiving speech it will have a 2 second brief storage to the user then turn back to

the major function again.

Furthermore, the programming group start to working on the filtering with Multithreading.

Next Week:

Page 160: Google Glass Project

151

For the coming week, Brandon is going to continue trying to enhance the filters. Luke and Alex are going

to continue to work with Yanglong and Kyle on tweaking the Glass App. The microphone subgroup is

going to generate a short list of microphones that may work with the Glass. For the final two weeks the

bulk of the group is going to focus on enhancing the current app, run experiments and have others test the

app to see what needs improvement. The rest of the group will focus on how it could be improved on in

the following semesters.

Week 24 This Week’s Plan: The main tasks this week were testing the current capabilities of the Glass app, enhancing the user

interface of the Glass app, developing the filters, and looking into microphones for the Glass and how to

implement them.

Major Issues: None.

This Week’s Progress:

For the filtering, several new filtering algorithms were used with the goal of determining whether the

problem with the filters was in the filters themselves or how they were coded into the Filter Tester app.

Going off of Tong’s suggestion that the issue could be with the number of points being used in the

calculation, a 4-pole Butterworth LPF filter was attempted using an algorithm generated by an online

guide provided by the University of York http://www-users.cs.york.ac.uk/~fisher/mkfilter/. This had no

better results. Next more research was done into different types of filters and their uses, this lead to the

use of a simple single-pole LPF and a four-stage single-pole filter, both were unsuccessful. The

conclusion has been made that the issue with the filters is related to how they are implemented, and how

the program handles the rounding.

For the programming, the team is still working on microphone searching and the Multithreading process

of the filter. Currently we have found multiple microphones, but most of them do not work due to the

USB on the glass is abnormal. Furthermore, the debugging of the microphone has become an issue. Some

of the microphones need to change some coding parameters in order to work, forcing changes to the code.

These are the major problems the team has been facing right now. The team will continue working on this

integration process.

The testing group conducted experiments for the immersive speech program that mirrored the tests

conducted for “speech to text”. Overall, the immersive speech program was able to operate over greater

distances, for example up to 6ft 7in in a quiet environment (40-52 db). The battery life and car ride tests

are still on-going and the group expects results before the weekly presentation. It has been noted that the

app needs constant input every 2 seconds or the app will shut down. This may cause a problem if

someone wants to use the program in an everyday environment where people are not talking 24/7.

Next Week: For the coming week, Brandon is going to continue trying to enhance the filters. Luke and Alex are going

to continue to work with Yanglong and Kyle on tweaking the Glass App. The microphone subgroup is

going to generate a short list of microphones that may work with the Glass. For the final two weeks the

bulk of the group is going to focus on enhancing the current app, run experiments and have others test the

app to see what needs improvement. The rest of the group will focus on how it could be improved on in

the following semesters.

Page 161: Google Glass Project

152

Week 25 This Week’s Plan: The main tasks this week were to finish testing the current capabilities of the Glass app, enhancing the

user interface of the Glass app, attempt to debug the coding for the filters, and to work on the poster,

presentation and final report.

Major Issues: The filter will not be implemented this semester.

This Week’s Progress: For the filtering, the code was tinkered with in an attempt to find what was causing the static introduction,

however this did not yield any results. Following Luke’s suggestion for a simple gain on the audio feed, a

scalar was multiplied by the signal. This did increase the signal strength, which could increase the Glass

app’s range but it also caused some clipping and increases the signal uniformly across all frequencies.

Tong enhanced this technique by writing an auto-gain controller in MATLAB that has better results and

does not introduce clipping. This would be able to increase the range of the Glass app in clean

environments such as in a lecture.

For the programming, we are still working on the implementation of an external microphone and

improvement of the current app. So far, the menu is not functioning yet due to compatibility issues with

the immersion card, and the text history is not yet functioning due to internal storage permissions. Also,

multithreading with the filter is still being implemented to the app.

For the testing, the system validation experiments are being run. This will go through and state

whether each requirement was met. The test group will set out polls to support their results of user

friendliness.

Next Week and Beyond: For the coming week, Luke is going to continue tweaking the app to enhance its user friendliness and try

to eliminate known bugs. The test group will conduct the final system validation experiments by testing

the app/device on a poll group. The rest of the group will focus on the creation of the promotional video

for the senior expo. All members of the group are also going to work on compiling all of the reports and

write-ups into the final report.

Page 162: Google Glass Project

153

Appendix D Raspberry Pi Programming Log

Sentences in blue are command lines, output, or configuration code in the LX terminal.

11/8

Raspberry Pi hardware was set up and powered and nothing came up on display. It was found that the

supplied micro SD from the ESL had a crack in it. SD cards compatible with the RPi were researched

using wiki and Getting Started with Raspberry Pi. A PNY Class 4 16GB SD card was purchased from

walmart for a5oximately $15 and Raspbian image was written to it.

11/9

The RPi was again set up and powered, this time though the setup menu loaded. The the image was set to

expand (first option on the list) and the default presentation method (second option) was set to

“Desktop”. The system was rebooted and loaded directly to the desktop. Internet connectivity was

verified and the Raspbian was explored. Next it was found that the only microphone option for the RPi is

either use a USB microphone or a 3.5 mm microphone through a USB sound card. A USB microphone

was found and a method for enabling it was sought. It was found that the RPi uses the ALSA Utilities by

default. In order to change the input/output mode the following command line is entered into the LX

Terminal.

sudo nano /etc/modprobe.d/alsa-base.conf

This brings up the configuration code for the sound settings. In the command line

#Keep snd-usb-audio from being loaded as first soundcard

options snd-usb-audio index = -2

the = -2 was switch to = 0

The RPi was then rebooted. The microphone was found to work by using the command

arecord -d 5 -r 48000 test.wav

This command tells the system to record for a duration of 5 seconds at 48,000 Hz sample

rate. The command ran with no errors, however when play back was attempted using the command

aplay test.wav

The following error came up,

alsa lib pcm_dmix.c: 1018: (snd_pcm_open) unable to open slave aplay: main:682: audio open

error: no such file or directory

After researching online, no solutions were found. After trial and error it was found that the command

that was used to set the input as the USB microphone also switch the audio output to the USB

microphone. This was interpreted to mean that the error meant that the audio output method selected was

unable to play audio.

After more research online a solution was found at http://atgn.tumblr.com/post/54588497569/how-to-set-

default-audio-input-output-devices-on. The process was used and tested and found to be a success.

The next portion was done following the instructions from

https://sites.google.com/site/observing/Home/speech-recognition-with-the-raspberry-pi which first begins

Page 163: Google Glass Project

154

by updating the software using the following command in the “Root Terminal” which grants

administrator privileges.

apt-get update && apt-get upgrade

This finds, downloads and installs any updates to the Raspbian software. The updates installed

successfully along with bison and the ALSA development headers. Next the latest pocketsphinx and

sphinxbase were installed and compiled as per the instructions, save for using ./configuration instead

./configuration --enable fixed for sphinxbase. When testing the software it was found that the software

was not using the USB microphone and therefore is not working.

11/16

After some trouble shooting and testing. It was found that the cause of the software not using the USB

microphone was due to splitting the audio in and audio out file. This was discovered by checking the

cat /proc/asound/modules

This command showed that the default RPi microphone was being used instead of the USB

microphone. This discovery lead to using the initial method of setting the USB microphone to being used

instead of splitting the audio in/out. The down side of this is that a sound card or head set must be used

inorder to have both audio in and out. This is not precieved as a problem since for the use of this project

only the audio in will be used.

The pocketsphinx program is run through the LXTerminal by using the following steps.

Change the directory to pocketsphinx-0.8 by using the following command

cd pocketsphinx-0.8

next run the program by using the following command

src/programs/pocketsphinx_continuous -samplerate 48000

This command tells pocketsphinx to run in "continuous" mode at a sample rate of 48,000 Hz. The

"continuous" mode means that the program will continue listening and translating speech that it

hears. The next thing sought was to get the text to appear on an external display. This was accomplished

by accessing the RPi through SSH. This was first done on computer by using “Terminal” on a mid-2009

Macbook. To do this Terminal was launched and pi@IP address was entered. the IP address was found

using the command ifconfig in LXTerminal on the RPi and finding the value following the inet. Next the

password for the RPi was entered which was still the default of “Raspberry”. Note that the text does not

show when it is entered as a security measure. From there, Terminal asked if it should “trust” the new

device which “Yes” must be entered to proceed. From there PocketSphinx was launched and data

showed on the screen. A similar method was used to connect the RPi to an Apple iPad 2.

The next phase of work will dedicated to cleaning up the audio and testing the software.

To improve audio the USB bus speed was limited to 12 Mbps. This technique a was suggested in the

book, Raspberry Pi for Secret Agents by Stefan Sjogelid. To do this the following process was used.

1. Open the boot configuration cmdline by typing nano /boot/cmdline.txt

2. Next the string dwc_otg.speed=1 was added in front of dwc_otg.lpm_enable=0 with a space between the two commands

3. reboot the RPi

Page 164: Google Glass Project

155

This caused the mouse, keyboard and microphone to stop working, effectivily bricking the board. To fix

this issue the SD card was plugged into a computer and luckily the cmdline.txt file was accessible from

the computer and was changed back.

11/23

Methods for decreasing the lag time between the speech registered and text output was researched. The

best option appears to be http://cmusphinx.sourceforge.net/wiki/pocketsphinxhandhelds. This explains

the different settings in Sphinx that affects lag time and explains how to tweak them.

A solution for the inaccuracy problem was also sought and it appears that main culprit is the limited

speech dictionary of SphinxBase. To fix this it was recommended that a different base be used or

created. CMU Sphinx’s website offers a large selection of bases at

http://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/. In order to

create a model CMU provides a program called SphinxTrain which always the user to build their speech

dictionary.

All of these options will be tested in the beginning of the spring semester.

Page 165: Google Glass Project

156

Appendix E Dictation Research Initial Research Log

Coleman

9/14

Conducted Google search for research into speech to text software, read documentation on current state of

the art (http://www.eetimes.com/document.asp?doc_id=1266606).

Returned to google search and conducted inquiry into Dragon dictation. Found Dragon dictation main cite

for Nuance software (http://www.nuance.com/dragon/index.htm). Continued pursuing Dragon Nuance

software and came across mobile app for using an android device to send audio recordings to computer

based Nuance platform (https://play.google.com/store/apps/details?id=nuance.com&hl=en).

Observed video of app/Nuance in action.

Conducted search of android app store to find above stated program

(https://play.google.com/store/apps/details?id=com.nuance.balerion&hl=en).

Reported findings to group.

9/15

Went back to Dragon Dictation website in an attempt to ascertain how the Nuance software and app could

be manipulated to meet our goals, as well as find a price range for software. Unable to obtain price range

through online means, however managed to find a product list

(http://www.nuance.com/products/index.htm). Upon exploring product list, found a developer’s kit for

android app programming and tethering to the Dragon Nuance Software (http://www.nuance.com/for-

developers/dragon/client-sdk/index.htm). Read attached PDF file on specific integration and source code

manipulation, unable to find price of developers kit. recommended direct phone contact if the group

wished to continue through this route. Uploaded PDF to group Google Drive File for future reference.

Page 166: Google Glass Project

157

Appendix F Microphone Search Log

Coleman, Jessen, Cheng

Cites visited and microphones found:

Newegg.com - About: “Newegg Inc. is a leading e-retailer committed to becoming the most

loved and trusted marketplace on the web by offering superior shopping experience, rapid

delivery, and stellar customer service.With more than 3 million products and an award-winning

web site, Newegg earns the loyalty of tech-enthusiasts and novice e-shoppers alike. We equip our

customers with state of the art decision-making resources such as detailed specs, how-to’s, 2

million customer reviews, and high resolution photo galleries [61].”

a. ATR-4650

Soundprofessionals.com - About: “The core product lines of The Sound Professionals include

innovative binaural microphone products, USB microphones, microphone preamps, cables and

others - currently, there are over 400 different Sound Professionals brand products that are

designed and manufactured in the USA. Along with the Sound Professionals brand products, they

also are a reseller of related audio recording products. This combination of manufacturing and

reselling in one business gives them a unique ability to offer an exceptional value to their

customers [62].”

a. MS-MMM-1

b. SP-MMM-1

Fullcompass.com - About: “Full Compass is a national leader in Professional Audio,

Professional Video, A/V, Lighting and Musical Instrument sales. Since 1977, our goal has been to

provide the best possible value with exceptional customer service. We offer over 700 top

equipment brands including; Shure, Yamaha, Panasonic, Sony, JBL, Sennheiser, Canon, QSC,

Audio-Technica, Crown and much more. We also provide services such as computer systems

integration, lighting design, and rentals (rentals available in Wisconsin only). Full Compass is an

authorized service center and parts distributor for most of the major manufacturers we carry

[63].”

a. i456

b. i266

c. i436

d. BP896C-TH

Audio-technica.com - About: “Established in 1962, Audio-Technica is a worldwide group of

companies devoted to the design, manufacture, marketing and distribution of problem-solving

audio equipment. Initially known for state-of-the-art phonograph cartridges, A-T now creates

high-performance microphones, headphones, wireless systems, mixers and electronic products for

home and professional use. Audio-Technica products have provided seamless audio coverage and

technical support in U.S. presidential debates since 1988. A-T mics also deliver versatile

solutions at high-profile sports broadcasts, including World Cup Soccer, the Super Bowl, and the

Commonwealth Games, as well as the Summer Games in Beijing (2008), Athens (2004), Sydney

(2000), and Atlanta (1996), and the Winter Games in Vancouver (2010), Torino (2006) and Salt

Lake City (2002) [64].”

a. PRO 42

b. BP896 MicroPoint™

c. BP896c MicroPoint™

Page 167: Google Glass Project

158

d. AT898

e. AT831b

f. MT830R

g. AT899c

Microphones.com - About: “New World Creations produces and manages

MICROPHONES.com which is a premiere reseller of microphones, digital recorders and voice

recognition products since 1998. We saw that there was an overwhelming demand from

individuals and businesses that had unique problems and in many cases required a variety of

microphones to fit their specific needs, they also needed experts to help them resolve these issues.

We made it our goal to solve their issues by provide them with the most advanced technological

hardware and software solutions. Whether it be for speech recognition accuracy and comfort, call

center computer integration, internet telephones or accessibility, we can provide you with the

right solution. [5].”

a. PRAM- 1

b. MM Series BSM-1

Cui.com - About: “CUI is a technology company focused on the development and distribution of

electronic components. At the leading edge of power supply design, the organization supports

customers as they strive to improve the energy efficiency and environmental credentials of their

application. The company's power group is complemented by a portfolio of world-class board

level components, consisting of interconnect, sound, motion control and thermal products. An

unwavering commitment to create collaborative partnerships with customers and a drive to see

that their design project is a success has been a hallmark of CUI's sustained growth since its

founding in 1989. As a leader in the industry, CUI will continue to invest in the future through

new technologies, talented employees, expanded manufacturing capabilities, and a growing

global reach. [65].”

a. 102-1725-ND(Digi-Key Part Number)

Digikey.com - About: “Digi-Key Corporation is one of the fastest growing distributors of

electronic components in the world.Since its founding in 1972, Digi-Key has been committed to

offering the broadest selection of in-stock electronic components, as well as providing the best

service possible to its customers, aiding engineers through the entire design process, from

Prototype to Production®. This has led the company to be highly ranked year after year in

industry surveys, in North America as well as Europe and Asia, in categories covering such facets

of business as availability of product, speed of service, responsiveness to problems, and more

[66].”

a. CMA-6542PF

b. CMA-6542TF-K

c. CMC-2242PBL-A

d. CMC-2742PBJ-A

e. CMC-2742WBL-25L

f. CMC-5042PF-AC

g. EM-6022

h. EM-6022P

i. EM-6027

j. EM-6027P

k. EM-6050

l. EM-6050P m. POM-3535L-3-R

n. ME-1538-100LB

Page 168: Google Glass Project

159

Appendix G Experiments

Read Time Research

Purpose

The purpose of this research is to understand how much time the text that was translated should be

displayed on the Google Glass to allow the user ample time to read and comprehend it. Although the

program should ultimately allow the user to select the duration the text will be displayed, this

information will be used both to create an “automatic” or “default” time duration setting and the time

may factor into the allotted time for the data processing of the program.

Theory

The theory behind this research is that movie/show producers and studios have conducted

experiments and tests on how long it takes the hearing impaired and those who have had the pleasure

of spending time in a waiting room (i.e. DMV) to read the subtitles and keep up with the program.

This research will simply be “piggy-backing” on the their research.

Procedure

For this research, members of the group will watch movies/shows that are commonly available (seen

on national television) and fill out the dat. It is recommended that the movies/shows selected should

be “On Demand” or able to be paused and rewound to allow the researcher the ability to both count

and time the subtitles.

Audio-to-Displayed Text Test

Purpose

The purpose of this experiment is to test the feasibility of ranged audio to text translation. This is the

first in a series of tests aimed at reaching the final goal of a portable audio to text translator that is

capable of capturing and translating audio in various ambient conditions and displaying the text to the

google glass. If this experiment is deemed a success, it will act as the base for future experiments.

Page 169: Google Glass Project

160

Equipment Needed

To perform this experiment the following equipment will be used,

Microphone

Dictation Software

A computing platform, (cell phone or laptop)

Measuring Tape

Procedure setup

In a quiet room, connect the microphone to a mobile platform that has a dictation software on it. The

dictation software will be turned on and a person shall recite a predefined text within 2 feet of the

microphone. The text shall be spoken by an American English speaker. This is because the software

was designed for dictating American speech and in an ideal situation it should achieve maximum

accuracy, leaving the only variable to be the distance the speaker is at. The audio should be

converted to text and displayed in the dictation software's interface. The accuracy of its translation

shall be recorded as the baseline. Next the speaker will move an additional foot from the microphone

and repeat the text. Again the accuracy will be recorded. This procedure will be repeated until the

accuracy begins decline.

Criteria

For the experiment to be determined a success, the dictation software shall have an accuracy of 90%

at a range of 15 ft. Accuracy is defined as number of correct word displayed on the screen. If the

word does not the match the word spoken by the person, it is measured on how close the word

displayed is to the word spoken by number of matching letters.

L/R Microphone Comparison Experiment

Purpose

The purpose for this experiment is to test the feasibility of using two microphones to determine a

sound source's location. The data collected from this experiment will be used in the sound filtering

process by allowing the user to indicate which sound source is desired simply.

Theory

Page 170: Google Glass Project

161

The theory behind this experiment is that microphones can be placed on the left and right side of a

person and when comparing the data they generate for a given sound source, can determine the

a5oximate direction of that source in relation to the user. It is believed that if the average intensity

from the two sources is compared that they will be identical or close to identical if the source is

directly in front of it.

Equipment Needed

Two Identical Microphones

A sound proof block a5oximately 6" wide and 10" tall (roughly the size of a human head)

Sound Analysis Software (Praat)

Protractor and string or some other device for measuring angles.

Masking Tape

Two People

Sound Room

Tape Measure

Procedure

1. In an open room (free of obstructions) place the sound proof block in a set location in the

middle of the room.

2. Place the microphones to the left and the right of it, with both being connected to a computer.

3. Use the protractor and string to mark the locations from 0-180 degrees every 10 degrees at a

set distances from the block (2 feet and 3 feet according to the Audio-to-Displayed Text Test)

and place a piece of the masking tape in line with degree (Figure 37).

4. With the software setup, have a person read a set script at a5oximately the same way or play a

recording at each of the positions at the first distance from the block.

5. Record the audio for both microphones at all of the location.

6. Repeat steps 4-5 until all the locations have been recorded.

Page 171: Google Glass Project

162

Figure 37: Experiment Setup

Analysis

Each of the recordings will be read into the sound analysis software which will be used to calculate

the average intensity for that microphone. The average intensity of the left microphone will be

compared to that of the right and the difference will be calculated. The data can then be presented on

a difference vs. angle graph for each of the distances. The sound data should be stored in case the

average intensity is found to not be the ideal comparison method.

Dictation Software Comparison

Purpose

The purpose for this experiment is to test and collect empirical data about the different dictation

software. The results are put into a decision matrix to determine the dictation software with the

highest usefulness for our project.

Theory

The theory behind this experiment is mostly based on personal evaluation. Criteria just as interface

and user-interface cannot be calculated and have to be therefore evaluated using a scale system in

which the tester rates each of the criteria from a number 1-3, where 1 is below average, 2 is average, and 3 being above average. Afterwards those numbers are put into a decision matrix to determine the

dictation software with the highest score.

Page 172: Google Glass Project

163

For the accuracy of the dictation software a sample text is provided to calculate the accuracy of the

dictation software. Unlike the other criteria, the accuracy is measured in percentage and is to be

presented separately to the decision matrix.

Equipment Needed

Test Room (Room should stay the same for all tests)

Dictation software

Platform on which the dictation software is operating on (Computer, Android Phone, etc.)

Sound recording device/Microphone (Should stay the same for all tests)

Sample Text

Procedure

1. Set up Microphone

2. Turn on the dictation software

3. Start the recording of the dictation software

4. Read sample text

5. Stop recording

6. Count the amount of spelling mistakes of the dictation software

7. Start recording

8. Read first sentence of sample text

9. Measure latency

10. Fill out decision matrix

Analysis

While most of the criterias in the decision matrix are evaluate by the tester using his/her experience

with the software, the accuracy can be determined by:

Accuracy % = (Amount of correct words / Amount of total words)*100

As for the latency, following chart can be used to determine the score:

Latency Time (seconds)

Score

Page 173: Google Glass Project

164

x < 0.5

3

0.5 < x < 1

2

1 < x

1

Different Environments Test

Purpose

The purpose of this experiment is to test how the different microphone properties are affected by

different environments. This will be tested by capturing the sound source, then analyzing it with the

Pratt software. The major goal of this experiment is to figure out how the reverberation and the sound

decay change based on the environment and to make recommendations on requirements based on

these findings.

Equipment needed

pratt software

computer

tape measure

microphone

digital instrument

Procedure setup

In this experiment, four to five different room sizes will be chosen. In order to minimize variables in

the experiment, the sound resource is recorded and replayed by an iPhone. Before the testing,

measure the size of the room and determine the geometric center and place the iPhone there. Then

play the audio with the microphone recording, use the Pratt software to determine the different

recording attributes. (Need a standard position of the microphone from the audio source)

Criteria

For the experiment to be determined a success, the sound source must be in geometry center. The

ambient noise also must be minimized as much as possible. The only variable should be the room's

dimensions. We may use ISSS to a reference to set the different situation to our test issue.

Page 174: Google Glass Project

165

Future experiment

This experiment can set in another two case in order to test the sensibility of microphone.

1. If we change the various ambient noise keeping the other variables same, how does the

intensity, pitch and other audio attributes change?

2. If we change the position of sound source then redo this experiment in the same room, how

will the attributes change?

Page 175: Google Glass Project

166

Appendix H Smartphone Audio-to-Text

Main Activity

//Google Glass Group Smartphone Audio-to-Text

//Written by Brandon Mills

//Last Updated: 2/24/2014

package com.androidddev101.ep8;

import java.io.BufferedWriter;

import java.io.File;

import java.io.FileWriter;

import java.util.ArrayList;

import android.annotation.SuppressLint;

import android.app.Activity;

import android.content.Context;

import android.content.Intent;

import android.media.AudioManager;

import android.os.Bundle;

import android.os.Environment;

import android.os.Handler;

import android.os.PowerManager;

import android.speech.RecognitionListener;

import android.speech.RecognizerIntent;

import android.speech.SpeechRecognizer;

import android.util.Log;

import android.view.Menu;

import android.widget.TextView;

import android.widget.Toast;

@SuppressLint("SimpleDateFormat")

public class Ep8Activity extends Activity {

private static final String TAG = Ep8Activity.class.getName();

//wakelock to keep screen on

protected PowerManager.WakeLock mWakeLock;

//speach recognizer for callbacks

private SpeechRecognizer mSpeechRecognizer;

//handler to post changes to progress bar

private Handler mHandler = new Handler();

Page 176: Google Glass Project

167

//ui textview

TextView responseText;

public boolean isAvailable = false;

//intent for speech recogniztion

Intent mSpeechIntent;

//this bool will record that it's time to kill P.A.L.

boolean killCommanded = false;

@Override

public void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

setContentView(R.layout.activity_ep8);

responseText = (TextView) findViewById(R.id.responseText);

//Speech Enhancement?

AudioManager localAudioManager = (AudioManager)

getSystemService("audio");

localAudioManager.setParameters("noise_suppression=auto");

}

@SuppressWarnings("deprecation")

@Override

protected void onStart() {

mSpeechRecognizer =

SpeechRecognizer.createSpeechRecognizer(Ep8Activity.this);

SpeechListener mRecognitionListener = new SpeechListener();

mSpeechRecognizer.setRecognitionListener(mRecognitionListener);

mSpeechIntent = new

Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);

mSpeechIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE,"com.androi

ddev101.ep8");

// Given an hint to the recognizer about what the user is going to say

mSpeechIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,

RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);

// Specify how many results you want to receive. The results will be sorted

// where the first result is the one with higher confidence.

mSpeechIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 20);

mSpeechIntent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true);

Page 177: Google Glass Project

168

//aqcuire the wakelock to keep the screen on until user exits/closes app

final PowerManager pm = (PowerManager)

getSystemService(Context.POWER_SERVICE);

this.mWakeLock =

pm.newWakeLock(PowerManager.SCREEN_DIM_WAKE_LOCK, TAG);

this.mWakeLock.acquire();

mSpeechRecognizer.startListening(mSpeechIntent);

super.onStart();

}

@Override

public boolean onCreateOptionsMenu(Menu menu) {

return false;

}

@Override

protected void onPause() {

//kill the voice recognizer

if(mSpeechRecognizer != null){

mSpeechRecognizer.destroy();

mSpeechRecognizer = null;

}

this.mWakeLock.release();

super.onPause();

}

class SpeechListener implements RecognitionListener {

public void onBufferReceived(byte[] buffer) {

Log.d(TAG, "buffer recieved ");

}

public void onError(int error) {

//if critical error then exit

if(error == SpeechRecognizer.ERROR_CLIENT || error ==

SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS){

Log.d(TAG, "client error");

}

//else ask to repeats

else{

Log.d(TAG, "other error");

mSpeechRecognizer.startListening(mSpeechIntent);

}

}

public void onEvent(int eventType, Bundle params) {

Log.d(TAG, "onEvent");

}

Page 178: Google Glass Project

169

public void onPartialResults(Bundle partialResults) {

Log.d(TAG, "partial results");

}

public void onReadyForSpeech(Bundle params) {

Log.d(TAG, "on ready for speech");

}

public void onResults(Bundle results) {

Log.d(TAG, "on results");

ArrayList<String> matches = null;

if(results != null){

matches =

results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);

if(matches != null){

Log.d(TAG, "results are " + matches.toString());

final ArrayList<String> matchesStrings = matches;

//suppress from here

//Select Top Result

String response = matchesStrings.get(0);

//Send top result to the display

final String finalResponse = response;

mHandler.post(new Runnable() {

public void run() {

responseText.setText(finalResponse);

//Establishes File Info

File directory =

Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);

String name = "History.txt";

File myFile = new File(directory, name);

//Tries to create/append the file

try {

//Creates a new file with the given

info if one does not exist, appends the file if it does exist

FileWriter fileWriter = new

FileWriter(myFile,true);

BufferedWriter bufferWriter = new

BufferedWriter(fileWriter);

bufferWriter.write(finalResponse);

bufferWriter.write("\r\n");

bufferWriter.close();

//The "Toast" displays on screen

verification (Probably won't work on the glass)

Page 179: Google Glass Project

170

Toast.makeText(getBaseContext(),"file

saved",

Toast.LENGTH_SHORT).show();

//Catches it if it doesn't work

} catch (Exception e) {

// TODO Auto-generated catch block

e.printStackTrace();

Log.d(TAG, "Error Writing to File");

}

}

});

//to here to and unsuppress following command to return to

original function

//processCommand(matchesStrings);

if(!killCommanded)

mSpeechRecognizer.startListening(mSpeechIntent);

else

finish();

}

}

}

public void onRmsChanged(float rmsdB) {

//Toast.makeText(getBaseContext(),"Speech Level Changed",

// Toast.LENGTH_SHORT).show();

}

public void onBeginningOfSpeech() {

//Toast.makeText(getBaseContext(),"Voice Detected",

//Toast.LENGTH_SHORT).show();

}

public void onEndOfSpeech() {

//Toast.makeText(getBaseContext(),"Speech Ended",

//Toast.LENGTH_SHORT).show();

}

};

}

Layout

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"

xmlns:tools="http://schemas.android.com/tools"

android:layout_width="match_parent"

android:layout_height="match_parent" >

Page 180: Google Glass Project

171

<TextView

android:id="@+id/responseText"

android:layout_width="fill_parent"

android:layout_height="wrap_content"

android:layout_alignParentLeft="true"

android:layout_alignParentTop="true"

android:text="@string/hello_world"

android:textSize="35dp" />

</RelativeLayout>

Resources

<resources>

<string name="app_name">G3 Android ASR Test</string>

<string name="hello_world">Hello</string>

<string name="menu_settings">Settings</string>

<string name="title_activity_ep8">G3 Android ASR Test</string>

</resources>

Android Manifest

<manifest xmlns:android="http://schemas.android.com/apk/res/android"

package="com.androidddev101.ep8"

android:versionCode="1"

android:versionName="1.0" >

<uses-sdk

android:minSdkVersion="8"

android:targetSdkVersion="15" />

<uses-permission android:name="android.permission.WAKE_LOCK" />

<uses-permission android:name="android.permission.RECORD_AUDIO" />

<uses-permission android:name="android.permission.INTERNET" />

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

<application

android:icon="@drawable/ic_launcher"

android:label="@string/app_name"

android:theme="@style/AppTheme" >

<activity

Page 181: Google Glass Project

172

android:name=".Ep8Activity"

android:label="@string/title_activity_ep8" android:screenOrientation="portrait">

<intent-filter>

<action android:name="android.intent.action.MAIN" />

<category android:name="android.intent.category.LAUNCHER" />

</intent-filter>

</activity>

</application>

</manifest>

Page 182: Google Glass Project

173

Appendix I Filter Tester App

Main Activity

//Google Glass Group Filter Tester

//Written by Brandon Mills and Fengyuan Tong

//Last Updated: 4/4/2014

package com.example.filtertester;

import java.io.BufferedOutputStream;

import java.io.BufferedInputStream;

import java.io.File;

import java.io.FileInputStream;

import java.io.FileNotFoundException;

import java.io.FileOutputStream;

import java.io.IOException;

import android.app.Activity;

import android.media.AudioFormat;

import android.media.AudioRecord;

import android.media.MediaPlayer;

import android.media.MediaRecorder;

import android.os.Bundle;

import android.os.Environment;

import android.util.Log;

import android.view.Menu;

import android.view.View;

import android.widget.Button;

import android.widget.Toast;

public class MainActivity extends Activity {

private static final String TAG = MainActivity.class.getName();

private static final String TIMETAG_STRING = "Timestamp ";

private static final int RECORDER_BPP = 16;

private static final String AUDIO_RECORDER_FILE_EXT_WAV = ".wav";

private static final String AUDIO_RECORDER_FOLDER = "AudioRecorder";

private static final String AUDIO_RECORDER_TEMP_FILE = "record_temp.raw";

private static final String AUDIO_RECORDER_TEMP_FILE2 = "record_temp2.raw";

private static final int RECORDER_SAMPLERATE = 44100;

private static final int RECORDER_CHANNELS = AudioFormat.CHANNEL_IN_MONO;

private static final int RECORDER_AUDIO_ENCODING =

AudioFormat.ENCODING_PCM_16BIT;

private MediaPlayer mPlayer = null;

private AudioRecord recorder = null;

Page 183: Google Glass Project

174

private int bufferSize = 0;

private Thread recordingThread = null;

private boolean isRecording = false;

private int q = 0;

private byte[] Data = null;

private int fileLength = 0;

@Override

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

setContentView(R.layout.activity_main);

// bufferSize = AudioRecord.getMinBufferSize(RECORDER_SAMPLERATE,

RECORDER_CHANNELS, RECORDER_AUDIO_ENCODING);

bufferSize = 4096;

Log.d(TAG, "bufferSize = "+bufferSize);

Data = new byte [bufferSize];

final Button button = (Button) findViewById(R.id.RecordButton);

button.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

startRecording();

}

});

final Button button1 = (Button) findViewById(R.id.StopButton);

button1.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

stopRecording();

}

});

final Button button2 = (Button) findViewById(R.id.PlayOriginalButton);

button2.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

startPlayback();

}

});

final Button button3 = (Button) findViewById(R.id.FilterButton);

button3.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

q=1;

Log.d(TAG, "q"+1);

Page 184: Google Glass Project

175

Toast.makeText(getBaseContext(),"Filter

On",Toast.LENGTH_SHORT).show();

}

});

final Button button4 = (Button) findViewById(R.id.FilterOffButton);

button4.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

q=0;

Toast.makeText(getBaseContext(),"Filter

Off",Toast.LENGTH_SHORT).show();

}

});

final Button button5 = (Button) findViewById(R.id.PlayFilteredButton);

button5.setOnClickListener(new View.OnClickListener() {

public void onClick(View v) {

startFilteredPlayback();

}

});

}

private String getFilename(){

String filepath = Environment.getExternalStorageDirectory().getPath();

File file = new File(filepath,AUDIO_RECORDER_FOLDER);

if(!file.exists()){

file.mkdirs();

}

return (file.getAbsolutePath() + "/Originial Audio" +

AUDIO_RECORDER_FILE_EXT_WAV);

}

private String getFilename2(){

String filepath = Environment.getExternalStorageDirectory().getPath();

File file = new File(filepath,AUDIO_RECORDER_FOLDER);

if(!file.exists()){

file.mkdirs();

}

return (file.getAbsolutePath() + "/Filtered_Audio" +

AUDIO_RECORDER_FILE_EXT_WAV);

}

private String getTempFilename(){

Page 185: Google Glass Project

176

String filepath = Environment.getExternalStorageDirectory().getPath();

File file = new File(filepath,AUDIO_RECORDER_FOLDER);

if(!file.exists()){

file.mkdirs();

}

File tempFile = new File(filepath,AUDIO_RECORDER_TEMP_FILE);

if(tempFile.exists())

tempFile.delete();

return (file.getAbsolutePath() + "/" + AUDIO_RECORDER_TEMP_FILE);

}

private String getTempFilename2(){

String filepath = Environment.getExternalStorageDirectory().getPath();

File file = new File(filepath,AUDIO_RECORDER_FOLDER);

if(!file.exists()){

file.mkdirs();

}

File tempFile = new File(filepath,AUDIO_RECORDER_TEMP_FILE2);

if(tempFile.exists())

tempFile.delete();

return (file.getAbsolutePath() + "/" + AUDIO_RECORDER_TEMP_FILE2);

}

private void startRecording(){

recorder = new

AudioRecord(MediaRecorder.AudioSource.MIC,RECORDER_SAMPLERATE,

RECORDER_CHANNELS,RECORDER_AUDIO_ENCODING, bufferSize);

Toast.makeText(getBaseContext(),"Record Starting",Toast.LENGTH_SHORT).show();

recorder.startRecording();

isRecording = true;

recordingThread = new Thread(new Runnable() {

@Override

public void run() {

writeAudioDataToFile();

}

},"AudioRecorder Thread");

recordingThread.start();

}

Page 186: Google Glass Project

177

private void writeAudioDataToFile(){

String filename = getTempFilename();

BufferedOutputStream os = null;

try {

os = new BufferedOutputStream(new FileOutputStream(filename));

} catch (FileNotFoundException e) {

e.printStackTrace();

}

int read = 0;

if(null != os){

while(isRecording){

read = recorder.read(Data, 0, bufferSize);

if(AudioRecord.ERROR_INVALID_OPERATION != read){

try {

os.write(Data);

} catch (IOException e) {

e.printStackTrace();

}

}

}

try {

os.close();

} catch (IOException e) {

e.printStackTrace();

}

}

}

private void stopRecording(){

if(recorder != null){

isRecording = false;

Toast.makeText(getBaseContext(),"Recording

Stopped",Toast.LENGTH_SHORT).show();

recorder.stop();

recorder.release();

recorder = null;

recordingThread = null;

}

copyWaveFile(getTempFilename(),getFilename(), bufferSize);

if (q == 1){

Log.d(TAG, "Going to Filter");

Page 187: Google Glass Project

178

Filter();

}

}

private void startPlayback() {

mPlayer = new MediaPlayer();

try {

mPlayer.setDataSource(getFilename());

mPlayer.prepare();

mPlayer.start();

Toast.makeText(getBaseContext(),"Playback Starting",Toast.LENGTH_SHORT).show();

} catch (IOException e) {

Log.e(TAG, "prepare() failed");

}

}

private void startFilteredPlayback() {

mPlayer = new MediaPlayer();

try {

mPlayer.setDataSource(getFilename2());

mPlayer.prepare();

mPlayer.start();

Toast.makeText(getBaseContext(),"Filtered Playback

Starting",Toast.LENGTH_SHORT).show();

} catch (IOException e) {

Log.e(TAG, "prepare() failed");

}

}

//needs work

//Applies the Filter (BELL or LPF based on Cookbook))

/*private void Filter(){

Log.d(TIMETAG_STRING, "Filtering Started");

BufferedOutputStream out = null;

BufferedInputStream in = null;

//Bell Filter Coefficients

float b0 = (float) 0.535817592185304;

float b1 = (float) -0.550431600591974;

float b2 = (float) 0.106471799676353;

float a1 = (float) -0.550431600591974;

float a2 = (float) -0.357710608138343;

//LPF Coefficients

/*float b0 = (float) 0.482357812999928;

float b1 = (float) 0.964715625999857;

Page 188: Google Glass Project

179

float b2 = (float) 0.482357812999928;

float a1 = (float) 0.556768925479519;

float a2 = (float) 0.553690866414616;

//Set variables used and initial values

float y,xmem1, xmem2, ymem1, ymem2;

xmem1 = xmem2 = ymem1 = ymem2 =0;

Log.i(TIMETAG_STRING,"Initialization stage 1 completed");

try {

out = new BufferedOutputStream(new FileOutputStream(getTempFilename2()));

in = new BufferedInputStream(new FileInputStream(getTempFilename()));

byte[] data = new byte [in.available()];

fileLength = (in.available());

Log.i(TIMETAG_STRING,"FileLength read");

while (in.read(data) != -1){

Log.d(TIMETAG_STRING,"file loaded");

for(int i=0; i<fileLength; i++){

y = b0*(float)data[i]+b1*xmem1+b2*xmem2-a1*ymem1-a2*ymem2;

xmem2=xmem1;

xmem1 = (float)data[i];

ymem2 = ymem1;

ymem1 = y;

if(i % 1000==0)

{

Log.i(TIMETAG_STRING,"The "+i+"th data is processed");

}

if ((y > -Float.MIN_NORMAL) && (y < Float.MIN_NORMAL)){

y=0;

Log.i(TAG, "Denormal");

}

out.write((byte)y);

if(i % 1000==0)

{

// Log.i(TIMETAG_STRING,"The "+i+"th data is written");

}

}

}

in.close();

out.close();

} catch (FileNotFoundException e) {

Page 189: Google Glass Project

180

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

Log.d(TIMETAG_STRING, "Filtering Calculation Complete");

//Sends temp .raw file to be turned into a .WAV file

Log.i(TIMETAG_STRING,"Start generate WAV file");

copyWaveFile(getTempFilename2(),getFilename2(), fileLength);

Log.i(TIMETAG_STRING,"WAV file generated");

//Deletes Temp File

deleteTempFile();

}*/

//Applies 4-Pole Butterworth LPF based off website algorithm

/*private void Filter(){

Log.d(TIMETAG_STRING, "Filtering Started");

BufferedOutputStream out = null;

BufferedInputStream in = null;

//Set variables used and initial values

Log.i(TIMETAG_STRING,"Initialization stage 1 completed");

try {

out = new BufferedOutputStream(new FileOutputStream(getTempFilename2()));

in = new BufferedInputStream(new FileInputStream(getTempFilename()));

byte[] data = new byte [in.available()];

fileLength = (in.available());

Log.i(TIMETAG_STRING,"FileLength read");

float GAIN = (float) 2.895062552e+02;

float[] xv = new float [5];

float[] yv = new float [5];

float nov = 0;

while (in.read(data) != -1){

Log.d(TIMETAG_STRING,"file loaded");

for(int i=0; i<fileLength; i++){

xv[0] = xv[1]; xv[1] = xv[2]; xv[2] = xv[3]; xv[3] = xv[4];

xv[4] = data[i]/ GAIN;

yv[0] = yv[1]; yv[1] = yv[2]; yv[2] = yv[3]; yv[3] = yv[4];

yv[4] = ((xv[0] + xv[4]) + 4 * (xv[1] + xv[3]) + 6 * xv[2]

+ ((float) -0.2201292677 * yv[0]) + ( (float) 1.2062353665 * yv[1])

Page 190: Google Glass Project

181

+ ( (float)-2.5608371103 * yv[2]) + ( (float) 2.5194645027 * yv[3]));

nov = yv[4];

if(i % 1000==0)

{

Log.i(TIMETAG_STRING,"The "+i+"th data is processed");

}

if ((nov > -Float.MIN_NORMAL) && (nov < Float.MIN_NORMAL)){

nov=0;

Log.i(TAG, "Denormal");

}

out.write((byte)nov);

if(i % 1000==0)

{

// Log.i(TIMETAG_STRING,"The "+i+"th data is written");

}

}

}

in.close();

out.close();

} catch (FileNotFoundException e) {

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

Log.d(TIMETAG_STRING, "Filtering Calculation Complete");

//Sends temp .raw file to be turned into a .WAV file

Log.i(TIMETAG_STRING,"Start generate WAV file");

copyWaveFile(getTempFilename2(),getFilename2(), fileLength);

Log.i(TIMETAG_STRING,"WAV file generated");

//Deletes Temp File

deleteTempFile();

}*/

//Applies Cascade LPF or Single Stage LPF (Based on online Textbook)

private void Filter(){

Log.d(TIMETAG_STRING, "Filtering Started");

BufferedOutputStream out = null;

BufferedInputStream in = null;

//Four Stage LPF

/*float a0 = (float) 0.739169;

Page 191: Google Glass Project

182

float b1 = (float) 0.291093;

float b2 = (float) -0.0317756;

float b3 = (float) 0.001542;

float b4 = (float) 0.000028;*/

//Single Stage LPF

float a0 = (float) 0.15;

float b1 = (float) 0.85;

//Set variables used and initial values

float y,ymem1, ymem2, ymem3, ymem4;

ymem1 = ymem2 = ymem3 = ymem4 =0;

Log.i(TIMETAG_STRING,"Initialization stage 1 completed");

try {

out = new BufferedOutputStream(new FileOutputStream(getTempFilename2()));

in = new BufferedInputStream(new FileInputStream(getTempFilename()));

byte[] data = new byte [in.available()];

fileLength = (in.available());

Log.i(TIMETAG_STRING,"FileLength read");

while (in.read(data) != -1){

Log.d(TIMETAG_STRING,"file loaded");

for(int i=0; i<fileLength; i++){

y=5*(float)data[i];

/*y = a0*(float)data[i]+b1*ymem1;

ymem1 = y;*/

if(i % 1000==0)

{

Log.i(TIMETAG_STRING,"The "+i+"th data is processed");

}

if ((y > -Float.MIN_NORMAL) && (y < Float.MIN_NORMAL)){

y=0;

Log.i(TAG, "Denormal");

}

out.write((byte)y);

if(i % 1000==0)

{

// Log.i(TIMETAG_STRING,"The "+i+"th data is written");

}

}

}

in.close();

Page 192: Google Glass Project

183

out.close();

} catch (FileNotFoundException e) {

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

Log.d(TIMETAG_STRING, "Filtering Calculation Complete");

//Sends temp .raw file to be turned into a .WAV file

Log.i(TIMETAG_STRING,"Start generate WAV file");

copyWaveFile(getTempFilename2(),getFilename2(), fileLength);

Log.i(TIMETAG_STRING,"WAV file generated");

//Deletes Temp File

deleteTempFile();

}

private void deleteTempFile() {

File file = new File(getTempFilename());

file.delete();

File file2 = new File(getTempFilename2());

file2.delete();

Log.d(TAG, "Temp Files Deleted");

}

private void copyWaveFile(String inFilename,String outFilename, int buff){

FileInputStream in = null;

FileOutputStream out = null;

long totalAudioLen = 0;

long totalDataLen = totalAudioLen + 36;

long longSampleRate = RECORDER_SAMPLERATE;

int channels = 1;

long byteRate = RECORDER_BPP * RECORDER_SAMPLERATE * channels/8;

byte[] data = new byte[buff];

try {

in = new FileInputStream(inFilename);

out = new FileOutputStream(outFilename);

totalAudioLen = in.getChannel().size();

totalDataLen = totalAudioLen + 36;

WriteWaveFileHeader(out, totalAudioLen, totalDataLen,longSampleRate, channels,

byteRate);

while(in.read(data) != -1){

out.write(data);

Page 193: Google Glass Project

184

}

in.close();

out.close();

} catch (FileNotFoundException e) {

e.printStackTrace();

} catch (IOException e) {

e.printStackTrace();

}

Log.d(TAG, "Made it to here2");

}

private void WriteWaveFileHeader(

FileOutputStream out, long totalAudioLen,

long totalDataLen, long longSampleRate, int channels,

long byteRate) throws IOException {

byte[] header = new byte[44];

header[0] = 'R'; // RIFF/WAVE header

header[1] = 'I';

header[2] = 'F';

header[3] = 'F';

header[4] = (byte) (totalDataLen & 0xff);

header[5] = (byte) ((totalDataLen >> 8) & 0xff);

header[6] = (byte) ((totalDataLen >> 16) & 0xff);

header[7] = (byte) ((totalDataLen >> 24) & 0xff);

header[8] = 'W';

header[9] = 'A';

header[10] = 'V';

header[11] = 'E';

header[12] = 'f'; // 'fmt ' chunk

header[13] = 'm';

header[14] = 't';

header[15] = ' ';

header[16] = 16; // 4 bytes: size of 'fmt ' chunk

header[17] = 0;

header[18] = 0;

header[19] = 0;

header[20] = 1; // format = 1

header[21] = 0;

header[22] = (byte) channels;

header[23] = 0;

header[24] = (byte) (longSampleRate & 0xff);

header[25] = (byte) ((longSampleRate >> 8) & 0xff);

header[26] = (byte) ((longSampleRate >> 16) & 0xff);

Page 194: Google Glass Project

185

header[27] = (byte) ((longSampleRate >> 24) & 0xff);

header[28] = (byte) (byteRate & 0xff);

header[29] = (byte) ((byteRate >> 8) & 0xff);

header[30] = (byte) ((byteRate >> 16) & 0xff);

header[31] = (byte) ((byteRate >> 24) & 0xff);

header[32] = (byte) (2 * 16 / 8); // block align

header[33] = 0;

header[34] = RECORDER_BPP; // bits per sample

header[35] = 0;

header[36] = 'd';

header[37] = 'a';

header[38] = 't';

header[39] = 'a';

header[40] = (byte) (totalAudioLen & 0xff);

header[41] = (byte) ((totalAudioLen >> 8) & 0xff);

header[42] = (byte) ((totalAudioLen >> 16) & 0xff);

header[43] = (byte) ((totalAudioLen >> 24) & 0xff);

out.write(header, 0, 44);

}

@Override

public boolean onCreateOptionsMenu(Menu menu) {

// Inflate the menu; this adds items to the action bar if it is present.

getMenuInflater().inflate(R.menu.main, menu);

return true;

}

}

Layout <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"

xmlns:tools="http://schemas.android.com/tools"

android:layout_width="match_parent"

android:layout_height="match_parent"

android:paddingBottom="@dimen/activity_vertical_margin"

android:paddingLeft="@dimen/activity_horizontal_margin"

android:paddingRight="@dimen/activity_horizontal_margin"

android:paddingTop="@dimen/activity_vertical_margin"

tools:context=".MainActivity" >

<Button

android:id="@+id/RecordButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_alignLeft="@+id/StopButton"

android:layout_alignParentTop="true"

Page 195: Google Glass Project

186

android:layout_alignRight="@+id/StopButton"

android:onClick="Record"

android:text="@string/Record_Button" />

<Button

android:id="@+id/StopButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_alignParentLeft="true"

android:layout_alignParentRight="true"

android:layout_below="@+id/RecordButton"

android:onClick="stop()"

android:text="@string/Stop_Record_Button" />

<Button

android:id="@+id/FilterButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_alignLeft="@+id/StopButton"

android:layout_alignRight="@+id/RecordButton"

android:layout_below="@+id/StopButton"

android:text="@string/Filter_Button" />

<Button

android:id="@+id/PlayOriginalButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_alignLeft="@+id/FilterButton"

android:layout_alignRight="@+id/FilterOffButton"

android:layout_centerVertical="true"

android:text="@string/Play_Original_Button" />

<Button

android:id="@+id/PlayFilteredButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_alignLeft="@+id/FilterButton"

android:layout_alignRight="@+id/PlayOriginalButton"

android:layout_below="@+id/PlayOriginalButton"

android:text="@string/Play_Filtered_Button" />

<Button

android:id="@+id/FilterOffButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_above="@+id/PlayOriginalButton"

Page 196: Google Glass Project

187

android:layout_alignParentLeft="true"

android:layout_alignRight="@+id/FilterButton"

android:text="@string/Filter_Off_Button" />

</RelativeLayout>

Resources

<?xml version="1.0" encoding="utf-8"?>

<resources>

<string name="app_name">Filter Tester</string>

<string name="action_settings">Settings</string>

<string name="Filter_Button">Apply Filter</string>

<string name="Record_Button">Record</string>

<string name="Play_Original_Button">Play Original</string>

<string name="Stop_Record_Button">Stop Recording</string>

<string name="Play_Filtered_Button">Play Filtered Audio</string>

<string name="Filter_Off_Button">Turn Off Filter</string>

</resources>

Android Manifest

<?xml version="1.0" encoding="utf-8"?>

<manifest xmlns:android="http://schemas.android.com/apk/res/android"

package="com.example.filtertester"

android:versionCode="1"

android:versionName="1.0" >

<uses-sdk

android:minSdkVersion="11"

android:targetSdkVersion="19" />

<uses-permission

android:name="android.permission.RECORD_AUDIO"></uses-permission>

<uses-permission

android:name="android.permission.WRITE_EXTERNAL_STORAGE"></uses-permission>

<application

android:allowBackup="true"

android:icon="@drawable/ic_launcher"

android:label="@string/app_name"

android:theme="@style/AppTheme" >

<activity

android:name="com.example.filtertester.MainActivity"

android:label="@string/app_name" >

<intent-filter>

<action android:name="android.intent.action.MAIN" />

Page 197: Google Glass Project

188

<category android:name="android.intent.category.LAUNCHER" />

</intent-filter>

</activity>

</application>

</manifest>

Page 198: Google Glass Project

189

Appendix J AGC MATLAB CODE

%Googl e Gl ass Gr oup AGC %Wri tt en by Fengyuan Tong %% Confi gur ati on % Defi ne a MACRO t o control audi o pl ayback. % If I F_PLAY_BACK i s set t o 1, bot h ori gi nal audi o and processed audi o will % be pl ayed, ot her wi se nei t her of t hose audi os will be pl ayed. I F_PLAY_BACK=1; % MACRO f or pl ot. % i f t hi s MACRO i s set t o 1, ampl it ude vs ti me will be pl ott ed, ot her wi se % not hi ng will be pl ott ed. I F_PLOT_I N_TI ME_DOMAI N=1; % Decl ar e fil e pat h and name. % The sampl e I got i s di vi ded i nt o 5 part s accor di ng t o t he cont ent. % Ther e' re t ot all y 5 sampl es, 3ft. wav/ 6ft. wav/ 9ft. wav/ 12ft.wav/ 15ft. wav % Change t he sampl e_fil e vari abl e bel ow t o check how t hi s wor ks on % i ndi vi dual sampl e. sampl e_fil e=' audi o_sampl e/ 3ft. wav' ; % I nt ended out put vol umn l evel (i n dB). % Thi s contr ol vari abl e ranges i n [-30, 30]. % You can use any val ue, al t hough t hi s range i s recommended. % Fr om my experi ence, [-10, 0] i s a good r ange t o st art wi t h. % If you set it t o be a posi ti ve val ue, qualit y of t he sound wi ll suff er out put _gai n_l evel _dB=- 10; %% Pr ocessi ng % s = audi o sampl e(fl oat number s) % Fs = sampl e r at e [ s, Fs] =wavr ead( sampl e_fil e); % f or checki ng t he t he number of channel s MONO = 1; STEREO = 2; % Extr act basi c i nf or mati on % Fi l eLent h = t he number of sampl es i n t he audi o cli p % channel = number of channel s i n t he audi o cli p [ Fil eLengt h, channel ] =si ze(s); % Ti meDur ati on = t he ti me l engt h of t he audi o cli p Ti meDur ati on=Fi l eLengt h./ Fs; % cal cul at e i nput power p_db=10*l og10( sum( s( 1: Fil eLengt h,:). ^2)/ Fil eLengt h);

Page 199: Google Glass Project

190

% cal cul at e t he nor mal val ue of t he out put power out put _power _Nor mal = 10. (̂ out put _gai n_l evel _dB/ 10); % cal cul at e t he ener gy of t he i nput ener gy = sum( s( 1: Fil eLengt h,:). ^2); % out put buff er = our out put if( channel == STEREO) % cal cul at e t he mul ti pli cati on f act ors K1 and K2 f or st ereo si gnal K1 = sqrt( (out put _power _Nor mal *Fil eLengt h) / ener gy(:, 1)); K2 = sqrt( (out put _power _Nor mal *Fil eLengt h) / ener gy(:, 2)); % mul ti pl y K wi t h i nput sampl es t o get requi red st er eo out put sampl es out put buff er =zer os( Fil eLengt h, 2); out put buff er(:, 1) = s(:, 1) * K1; out put buff er(:, 2) = s(:, 2) * K2; el se % cal cul at e t he mul ti pli cati on f act or K1 f or mono si gnal K1 = sqrt( (out put _power _Nor mal *Fil eLengt h) / ener gy); % mul ti pl y K1 wi t h i nput sampl es t o get requi r ed out put sampl es out put buff er =zer os( Fil eLengt h, 1); out put buff er(:, 1) = s(:, 1) * K1; end %% Pl ot if(I F_PLOT_I N_TI ME_DOMAI N==1) x_axi s=1: Fil eLengt h; x_axi s=x_axi s./ Fs; pl ot _handl e_1=subpl ot(1, 2, 1); pl ot _handl e_2=subpl ot(1, 2, 2); % pl ot t he pr ocessed si gnal i n ti me domai n pl ot(pl ot _handl e_2, x_axi s, out put buff er); xl abel (pl ot _handl e_2,' Ti me/ s' ,' font si ze' , 14); titl e(pl ot _handl e_2,' Pr ocessed si gnal i n ti me domai n' ,'font si ze' , 20); % pl ot t he ori gi nal si gnal i n ti me domai n pl ot(pl ot _handl e_1, x_axi s, s); titl e(pl ot _handl e_1,' Ori gi nal si gnal i n ti me domai n' ,' fontsi ze' , 20); xl abel (pl ot _handl e_1,' Ti me/ s' ,' font si ze' , 14); yl abel (pl ot _handl e_1,' Ampl it ude' ,' font si ze' , 14); % sync axi s li nkaxes([ pl ot _handl e_2 pl ot _handl e_1],' y' ); end; %% Pl ay audi o

Page 200: Google Glass Project

191

if(I F_PLAY_BACK==1) % pl ay t he ori gi nal audi o pl ayer _ori gni al =audi opl ayer(s, Fs); pl ay( pl ayer _ori gni al ); % Pause t he t hr ead t o avoi d 2 fil es pl ayi ng at t he same ti me pause( Ti meDur ati on); % Pl ay pr ocessed audi o pl ayer _post pr ocess=audi opl ayer( out put buff er, Fs); pl ay( pl ayer _post pr ocess); end;

Page 201: Google Glass Project

192

Appendix K Google Glass Application

Main Activity: MainActivity.java

package com.example.myimmersion;

import java.io.BufferedReader;

import java.io.File;

import java.io.FileReader;

import java.io.FileWriter;

import java.io.IOException;

import java.text.DateFormat;

import java.util.ArrayList;

import java.util.Date;

import java.util.List;

import java.lang.String;

import android.app.Activity;

import android.content.Context;

import android.content.Intent;

import android.os.Bundle;

import android.os.Environment;

import android.speech.RecognizerIntent;

import android.util.Log;

import android.view.KeyEvent;

import android.view.MotionEvent;

import com.google.android.glass.app.Card;

import com.google.android.glass.touchpad.Gesture;

import com.google.android.glass.touchpad.GestureDetector;

//the following two classes is needed for PowerManager and WakeLock class

//you will need to provide permission in AndroidManifest.xml to enable this feature

import android.os.PowerManager;

import android.os.PowerManager.WakeLock;

public class MainActivity extends Activity {

//Tag for WakeLock

private static final String TAG =

"com.example.myimmersion.ScreenOnWakeLockActivity.WAKE_LOCK_TAG";

//Tag for debugging

private static final String dTAG = "MainActivity_DEBUG_TAG";

//Declare a global variable named "testWakeLock" in the type of "WakeLock"

private WakeLock TestWakeLock;

//Declare testCard as global

private Card mainCard=new Card(this);

Page 202: Google Glass Project

193

private static final int SPEECH_REQUEST = 0;

public Boolean stopSpeechToText=false;

public Boolean runNewCycle=true;

public Boolean isHistoryMode = false;

private int historyTextEntryLineNumber = -1;

private ArrayList<String> lines = new ArrayList<String>();

private GestureDetector mGestureDetector;

//Declare FileWriter as global variable is not a good practice, but since we need to write

log to disk

//all the time, declare it here can save a bit time allocating memory

public FileWriter yboFileWriter;

//logFileDirectory must exist as global variable,

//or the FileWriter's behavior in each single thread will be undefined

public File logFileDirectory;

//onCreate() method will be invoked when the application starts

//it will only be invoked once

@Override

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

mGestureDetector = createGestureDetector(this);

setContentView(R.layout.activity_main);

Log.d(dTAG, "onCreate() is called");

//Initialize mainCard

displayContentOnMainCard_FitFootNote(" ",0);

Log.d(dTAG, "Card UI created");

//Deploy power management

PowerManager TestPowerManager = (PowerManager)

getSystemService(POWER_SERVICE);

TestWakeLock =

TestPowerManager.newWakeLock(PowerManager.SCREEN_DIM_WAKE_LOCK, TAG);

Log.d(dTAG, "PowerManager is created");

//acquire WakeLock

TestWakeLock.acquire();

Log.d(dTAG, "WakeLock Acquied");

//logFileDirectory is the path to TextHistory.txt, here it is set to be

sdcard/FITGoogleGlassTeam

logFileDirectory = new File(Environment.getExternalStorageDirectory(),

"FITGoogleGlassTeam");

//Check if destination directory exists, create it if not

if (!logFileDirectory.exists()) {

logFileDirectory.mkdirs();

Page 203: Google Glass Project

194

}

//invoke voice recognition built in google glass

enterInfiniteLoop();

}

private GestureDetector createGestureDetector(Context context) {

GestureDetector gestureDetector = new GestureDetector(context);

//Create a base listener for generic gestures

gestureDetector.setBaseListener( new GestureDetector.BaseListener() {

@Override

public boolean onGesture(Gesture gesture) {

//Tap to switch working mode

if (gesture == Gesture.TAP) {

if (!stopSpeechToText){ //Recognition is running-->It must be in

Recognition Mode

//Stop Recognition

stopSpeechToText = true;

displayContentOnMainCard_FitFootNote("Recognition

Stopped",1500);

}else return resumeToRecognitionMode();

//Swipe right display next entry

} else if (gesture == Gesture.SWIPE_RIGHT) {

if (isHistoryMode){

stopSpeechToText=true;

//if the historyTextEntryLineNumber is greater than size of

the file,

//reset it to 0

if (historyTextEntryLineNumber >= lines.size()){

historyTextEntryLineNumber=0;

}

//restore the previous session if sanity check passes

else{

historyTextEntryLineNumber++;

}

displayContentOnMainCard_HistoryMode(lines.get(historyTextEntryLineNumber),0);

return true;

}else {

initializeHistoryMode();

return true;

}

//Swipe left display previous entry

} else if (gesture == Gesture.SWIPE_LEFT) {

if (isHistoryMode){

stopSpeechToText=true;

Page 204: Google Glass Project

195

//if the historyTextEntryLineNumber is greater than size of

the file,

//reset it to 0

if (historyTextEntryLineNumber >= lines.size()){

historyTextEntryLineNumber=0;

}

//restore the previous session if sanity check passes

else{

historyTextEntryLineNumber--;

}

displayContentOnMainCard_HistoryMode(lines.get(historyTextEntryLineNumber),0);

return true;

}else {

initializeHistoryMode();

return true;

}

}

return false;

}

});

return gestureDetector;

}

/*

* Send generic motion events to the gesture detector

*/

@Override

public boolean onGenericMotionEvent(MotionEvent event) {

if (mGestureDetector != null) {

return mGestureDetector.onMotionEvent(event);

}

return false;

}

/*

@Override

public boolean onKeyDown(int keycode, KeyEvent event) {

//Tap to switch working mode

if (keycode == KeyEvent.KEYCODE_DPAD_CENTER) {

if (!stopSpeechToText){ //Recognition is running-->It must be in Recognition Mode

//Stop Recognition

stopSpeechToText = true;

displayContentOnMainCard_FitFootNote("Recognition Stopped",1500);

}else return resumeToRecognitionMode();

Page 205: Google Glass Project

196

}

//Swipe down display next entry

if (keycode == KeyEvent.KEYCODE_BACK){

if (isHistoryMode){

stopSpeechToText=true;

//if the historyTextEntryLineNumber is greater than size of the file,

//reset it to 0

if (historyTextEntryLineNumber >= lines.size()){

historyTextEntryLineNumber=0;

}

//restore the previous session if sanity check passes

else{

historyTextEntryLineNumber++;

}

displayContentOnMainCard_HistoryMode(lines.get(historyTextEntryLineNumber),0);

return true;

}else {

initializeHistoryMode();

return true;

}

}

return false;

}*/

//this is a dead loop which will never end, we can probably try to tune the

stopSpeechToText parameter to terminate this loop

//I put this in another thread as a multi-threading template(since you want to do multi-

threading but I have no idea what you want to achieve)

public void enterInfiniteLoop(){ //Infinite RecognizerIntent Loop

new Thread(new Runnable()

{

@Override

public void run()

{

while(stopSpeechToText==false)

{

if(runNewCycle==true)

startSpeechRecognizer();

try {

Thread.sleep(500);

} catch (InterruptedException e) {

Log.d(dTAG,"Speech To Text background

thread.sleep() error");

Page 206: Google Glass Project

197

e.printStackTrace();

}

}

Log.d(dTAG, "Speech to text background thread stopped");

}

}).start();

}

public void startSpeechRecognizer() { //start RecognizerIntent

Log.d(dTAG,"startSpeechRecognizer() is called");

//set runNewCycle to false to stop issuing new speech recognition request when

there is one running

runNewCycle=false;

Intent SpeechToTextIntent = new

Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);

startActivityForResult(SpeechToTextIntent, SPEECH_REQUEST);

}

public void displayContentOnMainCard_FitFootNote(String bodyText,int

stoppingInterval){

mainCard.setText(bodyText);

mainCard.setFootnote("Google Glass Group");

setContentView(mainCard.toView());

if(stoppingInterval>0){

try {

Thread.sleep(stoppingInterval);

} catch (InterruptedException e) {

e.printStackTrace();

}

}

}

public void displayContentOnMainCard_HistoryMode(String bodyText,int

stoppingInterval){

mainCard.setText(bodyText);

mainCard.setFootnote("History Mode");

setContentView(mainCard.toView());

if(stoppingInterval>0){

try {

Thread.sleep(stoppingInterval);

} catch (InterruptedException e) {

e.printStackTrace();

}

}

}

Page 207: Google Glass Project

198

@Override

protected void onActivityResult(int requestCode, int resultCode,Intent data) {

if (requestCode == SPEECH_REQUEST && resultCode == RESULT_OK) {

List<String> results =

data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);

String spokenText = results.get(0);

//do some job according to result

//if "history mode" is spoken, enter history mode

if (spokenText.trim().equalsIgnoreCase("history mode")){

initializeHistoryMode();

}

//if "detele file" is heard, history file will be deleted

else if (spokenText.trim().equalsIgnoreCase("delete file")){

deleteHistoryTXTFile();

}

//anything else will be treated as normal talkings

else{

//log it to file first

writeEntryToHistoryTXTFile(spokenText);

//display it for 2 seconds

displayContentOnMainCard_FitFootNote(spokenText,2000);

}

//set runNewCycle back to true to let speech recognition run again

runNewCycle=true;

}

super.onActivityResult(requestCode, resultCode, data);

}

public void initializeHistoryMode(){ //Prepare the app switching to history mode

isHistoryMode=true;

stopSpeechToText=true;

runNewCycle=false;

lines.clear();

File historyFile= new File(logFileDirectory, "TextHistory.txt");

if (historyFile.exists()){

BufferedReader yboBufferedReader;

//read all the record from file and store them in an ArrayList<String>

try {

// open the reader

yboBufferedReader = new BufferedReader(new

FileReader(historyFile));

String line = null;

// read all the lines till the end

while ((line = yboBufferedReader.readLine()) != null) {

lines.add(line);

Page 208: Google Glass Project

199

}

// close reader

yboBufferedReader.close();

}

catch (final IOException e) {

throw new RuntimeException(

"Error reading file : " + e.getMessage(), e);

}

//if this is the first time in history mode,

//show "Swipe down to view history, tap again to go back to recognition

mode"

if(lines.size()==0){

displayContentOnMainCard_HistoryMode("No History File or

File is empty",0);

}

else{

if (historyTextEntryLineNumber == -1){

displayContentOnMainCard_HistoryMode(lines.size()+"

records total in history file", 1500);

displayContentOnMainCard_HistoryMode("Swipe forward

or back to view history",0);

historyTextEntryLineNumber=0;

}

else

//if this is not the first time visiting history mode,

//restore the position last time was at

{

//if the historyTextEntryLineNumber, something must be

wrong

if (historyTextEntryLineNumber >= lines.size()){

displayContentOnMainCard_HistoryMode("Can't

restore previous session," +

" swipe down to view history from

start",0);

historyTextEntryLineNumber=0;

}

//restore the previous session if sanity check passes

else {

displayContentOnMainCard_HistoryMode(lines.size()+" records total in history file",

1500);

displayContentOnMainCard_HistoryMode(lines.get(historyTextEntryLineNumber),0);

}

}

}

Page 209: Google Glass Project

200

}else {

displayContentOnMainCard_HistoryMode("History file does not exists,

say something first", 1500);

if(resumeToRecognitionMode()){

runNewCycle=true;

}

}

}

public void deleteHistoryTXTFile(){ //Delete history file

File fileTobeDeleted = new File(new

File(Environment.getExternalStorageDirectory(),

"FITGoogleGlassTeam"), "TextHistory.txt");

if (fileTobeDeleted.delete()){

displayContentOnMainCard_HistoryMode("History File Deleted",2000);

historyTextEntryLineNumber=0;

}

else {

displayContentOnMainCard_HistoryMode("History File Not

Found",2000);

}

}

public void writeEntryToHistoryTXTFile(String entryToBeWritten){ //write result

to file

//Create a new FileWrite each time recognition result is given

try {

yboFileWriter=new FileWriter(new File(logFileDirectory,

"TextHistory.txt"),true);

} catch (IOException e1) {

e1.printStackTrace();

}

try {

//Log result to file while result is not empty

if(entryToBeWritten.length()>0){

//log entries are in the format of "Data Time Content"

//the exact format of the time can be controlled by modifying Data()

yboFileWriter.write(DateFormat.getDateTimeInstance().format(new

Date())+" "+entryToBeWritten);

//Append a new line for next entry

yboFileWriter.write("\r\n");

//Close FileWriter to avoid any potential error

yboFileWriter.close();

Log.d(dTAG,"TextHistory saved");

}

Page 210: Google Glass Project

201

} catch (IOException e1) {

Log.d(dTAG, "Logging to txt file failed");

e1.printStackTrace();

}

}

//this method will override original onResume() method and keep the screen up even it is

resumed from background

//if this method is deleted, the application will keep the screen up on its first startup, but if

you pause it and resume it from the background,

//the WakeLock will be lost

@Override

protected void onResume() {

super.onResume();

Log.d(dTAG, "onResume() is called");

TestWakeLock.acquire();

Log.d(dTAG, "WakeLock Acquied");

//restart the background thread

stopSpeechToText=false;

isHistoryMode= false;

enterInfiniteLoop();

}

// this method must exist, or this application will mess up the power management of the whole

system

@Override

protected void onPause() {

super.onPause();

Log.d(dTAG, "onPause() is called");

TestWakeLock.release();

Log.d(dTAG, "WakeLock released");

//stop the background thread from running when the glass is paused

stopSpeechToText=true;

}

public boolean resumeToRecognitionMode(){

displayContentOnMainCard_FitFootNote("Resuming Auto Recognition",1000);

displayContentOnMainCard_FitFootNote("", 0);

stopSpeechToText=false;

isHistoryMode=false;

runNewCycle=true;

enterInfiniteLoop();

return true;

Page 211: Google Glass Project

202

}

}

Strings: strings.xml <?xml version="1.0" encoding="utf-8"?> <resources> <string name="app_name">Live Subtitles</string> <string name="action_settings">Settings</string> <string name="stop">Stop</string> <string name="hello_world">Hello world!</string> <string name="show_hellow_world">Live Subs</string> <string name="title_activity_menu">MenuActivity</string> </resources>

Dimensions: dimens.xml <resources> <!-- Default screen margins, per the Android Design guidelines. --> <dimen name="activity_horizontal_margin">16dp</dimen> <dimen name="activity_vertical_margin">16dp</dimen> </resources>

Styles: styles.xml <resources> <!-- Base application theme, dependent on API level. This theme is replaced by AppBaseTheme from res/values-vXX/styles.xml on newer devices. --> <style name="AppBaseTheme" parent="android:Theme.Light"> <!-- Theme customizations available in newer API levels can go in res/values-vXX/styles.xml, while customizations related to backward-compatibility can go here. --> </style> <!-- Application theme. --> <style name="AppTheme" parent="AppBaseTheme"> <!-- All customizations that are NOT specific to a particular API-level can go here. --> </style> </resources>

Layout: activity_main.xml <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools"

Page 212: Google Glass Project

203

android:layout_width="match_parent" android:layout_height="match_parent" android:paddingBottom="@dimen/activity_vertical_margin" android:paddingLeft="@dimen/activity_horizontal_margin" android:paddingRight="@dimen/activity_horizontal_margin" android:paddingTop="@dimen/activity_vertical_margin" tools:context=".MainActivity" > <TextView android:layout_width="wrap_content" android:layout_height="wrap_content" android:text="@string/hello_world" /> </RelativeLayout>

Manifest: AndroidManifest.xml <?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.myimmersion" android:versionCode="1" android:versionName="1.0" > <uses-sdk android:minSdkVersion="15" android:targetSdkVersion="15" /> <uses-permission android:name="android.permission.WAKE_LOCK" /> <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" /> <application android:allowBackup="true" android:icon="@drawable/team_logo" android:label="@string/app_name" android:theme="@style/AppTheme" > <activity android:name="com.example.myimmersion.MainActivity" android:icon="@drawable/team_logo" android:immersive="true" android:label="@string/app_name" > <intent-filter> <action android:name="android.intent.action.MAIN" /> <action android:name="com.google.android.glass.action.VOICE_TRIGGER" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter> <meta-data android:name="com.google.android.glass.VoiceTrigger" android:resource="@xml/voice_trigger" /> </activity> <activity android:name="com.example.myimmersion.MenuActivity" android:label="@string/title_activity_menu" >

Page 213: Google Glass Project

204

</activity> </application> </manifest>

XML: voice_trigger.xml <?xml version="1.0" encoding="utf-8"?> <trigger keyword="@string/show_hellow_world" />

Page 214: Google Glass Project

205

12. REFERENCES

[1] Wikipedia, "File: Optimic 1140 Fiber Optical Microphone for Wiki.jpg," 2009.

[2] F. Audio, "Micro Optics Technologies Fibersound Audio," 2013. [Online]. Available:

www.fibersound.com/fiberopticmicrophone.html. [Accessed 10 November 2013].

[3] Moser, Zimmermann and Ellis, Engineering Acoustics: An Introduction to Noise Control,

New York: Srpinger, 2009.

[4] DV247, "Shure PG58-XLR Dynamic Vocal Microphone," 2013. [Online]. Available:

http://www.dv247.com/microphones/shure-pg58-xlr-dynamic-vocal-microphone--28689.

[Accessed 25 November 2013].

[5] J. B. Calvert, "Microphones," August 2003. [Online]. Available:

http://mysite.du.edu/~jcalvert/tech/microph.htm.

[6] The International Philco., "Ribbon Microphone," [Online]. Available:

http://www.coutant.org/philco/. [Accessed 23 September 2013].

[7] N. Y. X. E. C. Ltd, "Omni-directional electret condenser microphone," [Online]. Available:

http://www.machsources.com/suppliers/microphone/products/19196.html.

[8] J. Lewis, "Understanding Microphone Sensitivity," May 2012. [Online]. Available:

http://www.analog.com/library/analogdialogue/archives/46-

05/understanding_microphone_sensitivity.html. [Accessed August 2013].

[9] Audio-Technica, 2013. [Online]. Available: http://eu.audio-

technica.com/en/resources/tutorials/Polar%20Patterns.pdf. [Accessed 25 November 2013].

Page 215: Google Glass Project

206

[10] Raspberry Pi Foundation, "FAQs," Multilingual Word Press, November 2013. [Online].

Available: http://www.raspberrypi.org/faqs. [Accessed 25 November 2013].

[11] Zwetsloot, "Raspberrypi-diagram," 29 February 2012. [Online]. Available:

http://www.linuxuser.co.uk/wp-content/uploads/2012/02/raspberrypi-diagram.png.

[Accessed 25 November 2013].

[12] J. Benesty, S. Makino and C. Jingdong, "Introduction," in Speech Enhancement, New York,

Springer, 2005, pp. 1-9.

[13] LMS, "Acoustic Holography," 2007. [Online]. Available: http://www.lmsintl.com/acoustic-

holography. [Accessed 2013 November 2013].

[14] LMS, "Spherical Beamforming," 2009. [Online]. Available:

http://www.lmsintl.com/spherical-beamforming. [Accessed 25 November 2013].

[15] J. O. Smith, Introduction to Digital Filters with Audio Applications, W3K Publishing ,

2007.

[16] "Digital Filters," Springer.

[17] R. Bristow-Johnson, "Cookbook Formulae for Audio EQ Biquad Filter Coefficents,"

[Online]. Available: http://www.musicdsp.org/files/Audio-EQ-Cookbook.txt. [Accessed 14

February 2014].

[18] E. W. Weisstein, "Fast Fourier Transform," Mathworld--A Wolfram Web Resource,

[Online]. Available: http://mathworld.wolfram.com/FastFourierTransform.html. [Accessed

26 March 2006].

[19] R. Gruhn, W. Minker and S. Nakamura, "Automatic Speech Recognition," in Statistical

Pronunciation Modelling for Non-Native Speech Processing, Berlag Berlin Heidelberg,

Page 216: Google Glass Project

207

Springer, 2011, p. 11.

[20] T. Fisher, "Interactive Digital Filter Design," University of York , 1 September 1999.

[Online]. Available: http://www-users.cs.york.ac.uk/~fisher/mkfilter/. [Accessed 20 March

2014].

[21] Sweetwater: Music Instruments and Pro Audio, "Bell Filter," 13 March 2005. [Online].

Available: http://www.sweetwater.com/insync/bell-filter/. [Accessed 26 March 2014].

[22] E. Maningo, "What is a Low Shelf and High Shelf Filter in Parametric Equalization?," 11

May 2011. [Online]. Available: http://www.audiorecording.me/what-is-a-low-shelf-and-

high-shelf-filter-in-parametric-equalization.html. [Accessed 19 March 2014].

[23] J. P. A. Perez, B. C. Lopez and S. C. Pueyo, "Introduction & AGC Fundamentals," in

Automatic Gain Control Techniques and Architectures for RF Recievers, New York, NY:

Springer, 2011.

[24] Vuzix, "M100 Smart Glasses," 2013. [Online]. Available:

http://www.vuzix.com/augmented-reality/products_m100ag.html#overview.

[25] Meta, "The World's Most Advanced Interface," 2013. [Online]. Available:

https://www.spaceglasses.com/.

[26] GlassUp, "Glassup," 2013. [Online]. Available: http://www.glassup.net/.

[27] Pebble Technology Inc., "Pebble Smartwatch -- Top Tech Gift 2013," 2013. [Online].

Available: https://getpebble.com/.

[28] S. E. Inc., "SmartWatch," 2013. [Online]. Available:

http://www.sonymobile.com/us/products/accessories/smartwatch/specifications/#tabs.

[29] Samsung, Inc., "Samsung Galaxy Gear Is Here," 2013. [Online]. Available:

Page 217: Google Glass Project

208

http://www.samsung.com/us/guide-to-galaxy-smart-devices/galaxy-gear.html.

[30] N. C. Institute, "Captioning Glasses". US Patent 6005536, 21 December 1999.

[31] G. S. Snider, "Translating eyeglasses". US Patent S20020158816 A1, 31 October 2002.

[32] T. T. P. PLC.World Patent WO2013050749 A1, 11 April 2013.

[33] Beelen, "Glass with live subtitles," 3 December 2012. [Online]. Available:

http://www.hum.leiden.edu/lucl/news-events/news/glasses-with-live-subtitles.html.

[Accessed 25 November 2013].

[34] W. Powell, 22 July 2012. [Online]. Available:

http://www.willpowell.co.uk/blog/2012/07/22/project-glass-real-time-translation-inspired/..

[Accessed 2013 November 2013 ].

[35] S. E. Inc., "STW-C140GI: Entertainment Access Glasses with Audio," 2012. [Online].

Available:

http://pro.sony.com/bbsccms/assets/files/mkt/digicinema/brochures/EntAccessGlasses-DI-

0272_2.pdf.

[36] B. Munir, 10 March 2012. [Online]. Available: http://www.uoverip.com/voice-

fundamentals-human-speech-frequency/.

[37] D. Kaplan, "Drew's Equalizer and Frequency Equalization Tutorial," 2003. [Online].

Available: http://www.dak.com/reviews/tutorial_frequencies.cfm.

[38] Google, Inc., "Tech Specs," 2014. [Online]. Available:

https://support.google.com/glass/answer/3064128?hl=en&ref_topic=3063354.

[39] N. Olivarez-Giles, "Android top mobile OS in U.S., but Apple iPhone is most popular

phone," 28 July 2011. [Online]. Available:

Page 218: Google Glass Project

209

http://latimesblogs.latimes.com/technology/2011/07/android-top-os-in-us-but-apple-iphone-

is-most-popular-phone.html.

[40] International Data Corporation (IDC), "Android and iOS Combine for 91.1% of the

Worldwide Smartphone OS Market in 4Q12 and 87.6% for the Year, According to IDC," 13

February 2013. [Online]. Available:

http://www.idc.com/getdoc.jsp?containerId=prUS23946013.

[41] Google, Inc., "Glass Developer Kit," 20 November 2013. [Online]. Available:

https://developers.google.com/glass/develop/gdk/index.

[42] Google, Inc., "Math," [Online]. Available:

http://developer.android.com/reference/java/lang/Math.html. [Accessed 4 March 2014].

[43] Google Inc, "Android Developer Page," 2014. [Online]. Available:

http://developer.android.com/develop/index.html.

[44] Google Inc, "Building Your First App," 2014. [Online]. Available:

http://developer.android.com/training/basics/firstapp/index.html.

[45] Google Inc., "Glass Developer," 19 November 2013. [Online]. Available:

https://developers.google.com/glass/develop/.

[46] Google Inc, "Activities," [Online]. Available:

http://developer.android.com/guide/components/activities.html. [Accessed 2014].

[47] Google Inc, "Intents and Intent Filters," [Online]. Available:

http://developer.android.com/guide/components/intents-filters.html. [Accessed 2014].

[48] Google Inc., "Processes and Threads," [Online]. Available:

http://developer.android.com/guide/components/processes-and-threads.html. [Accessed

Page 219: Google Glass Project

210

2014].

[49] Google Inc., "Glass Developers Develop," 19 November 2013. [Online]. Available:

https://developers.google.com/glass/develop/index. [Accessed 2014].

[50] Google Inc., "Live Cards," 27 February 2014. [Online]. Available:

https://developers.google.com/glass/develop/gdk/ui/live-cards. [Accessed 2014].

[51] Google Inc., "Immersions," 24 February 2014. [Online]. Available:

http://developers.google.com/glass/design/ui/immersions#when_to_use_them. [Accessed

2014].

[52] Google Inc., "Glass Development Kit," 20 November 2013. [Online]. Available:

https://developers.google.com/glass/develop/gdk/index. [Accessed 2014].

[53] AndroidDev101, "Episode 8: Speech Recognition....say what?!?!," 22 January 2013.

[Online]. Available: https://www.youtube.com/watch?v=6G79QhBUvHk.

[54] L. Zanolin, "Add Voice Typing To Your IME," 12 Decemeber 2011. [Online]. Available:

http://android-developers.blogspot.co.il/2011/12/add-voice-typing-to-your-ime.html.

[55] Google Inc, "FileWriter," Android, [Online]. Available:

http://developer.android.com/reference/java/io/FileWriter.html. [Accessed 20 February

2014].

[56] Google, Inc, "BufferedWriter," Android, [Online]. Available:

http://developer.android.com/reference/java/io/BufferedWriter.html. [Accessed 20 February

2014].

[57] K. Varma, "Andorid Audio Recording, Part 2," Devlper, 5 December 2010. [Online].

Available: http://www.devlper.com/2010/12/android-audio-recording-part-2/. [Accessed 5

Page 220: Google Glass Project

211

March 2014].

[58] K. Varma, "krvarma-android-samples," 5 December 2010. [Online]. Available:

https://code.google.com/p/krvarma-android-

samples/source/browse/trunk/AudioRecorder.2/src/com/varma/samples/audiorecorder/Reco

rderActivity.java. [Accessed 5 March 2014].

[59] Google Inc, "USB Host," 2014. [Online]. Available:

http://developer.android.com/guide/topics/connectivity/usb/host.html.

[60] C. Stratton, "Is it possible to use the micro USB on Google Glass to connect to another type

of micro USB device for data transfer while the Glass is in use?," 3 December 2013.

[Online]. Available: http://stackoverflow.com/questions/20359572/is-it-possible-to-use-the-

micro-usb-on-google-glass-to-connect-to-another-type-o.

[61] Newegg, "Corporate Summary," 2013 December 2013. [Online]. Available:

http://www.newegg.com/Info/AboutUs.aspx.

[62] T. S. Professionals, "A short history of The Sound Professionals and its owners," 1

December 2013. [Online]. Available: http://www.soundprofessionals.com/cgi-

bin/gold/category.cgi?category=company.

[63] Full Compass Systems, LTD, 2013 December 2013. [Online]. Available:

http://www.fullcompass.com/.

[64] A.-T. Inc., "History," 1 December 2013. [Online]. Available: http://www.audio-

technica.com/cms/site/24ca27d9a4b486fe/index.html.

[65] C. Inc., "Company Overview," 1 Dec 2013. [Online]. Available:

http://www.cui.com/company/company.

Page 221: Google Glass Project

212

[66] M. Larsen, "A Message From the President," 1 December 2013. [Online]. Available:

microphones.com/about_us.html.

[67] Google, Inc, "App Components," 2013. [Online]. Available:

http://developer.android.com/guide/components/index.html.

[68] New World Creations, 1 December 2013. [Online]. Available:

http://www.microphones.com/about_us.html .

[69] Engdahl, "Powering Microphones," 2012. [Online]. Available:

http://www.epanorama.net/circuits/microphone_powering.html. [Accessed 20 November

2013].

[70] F. Audio, "Pebble Smartwatch - Top Tech Gift 2013," 2013. [Online]. Available:

https://www.sonymobile.com/us/products/accessories/smartwatch/specifications/#tabs.

[71] Google, Inc., "FileWriter," [Online].