Original Business Proposal

AcoustiGLASS: Wearable Acoustic Pattern Recognition System for the

Hearing Impaired

Kei Kojima Kotaro Kojima

Business Concept Competition 2016-2017

DIAMOND CHALLENGE FOR HIGH SCHOOL

ENTREPRENEURS

1

1. INTRODUCTION 1.1 Problem, Pain Point, or Market Opportunity Over 360 million people in the world have little or no hearing, many of them without adequate support that is sorely needed for their disability [1]1.In fact, hearing impairment is one of the most common chronic conditions in older adults, affecting at least 29 million Americans [2]. Furthermore, over 14% of the U.S. adolescents’ population aged 12 to 19 years report hearing loss, which points to the growing prevalence of this disability condition in teenagers [3]. Unsafe use of personal audio devices, including smartphones, and exposure to damaging levels of sound at noisy entertainment venues such as sporting events have put millions of teenagers at risk of severe deafness for the rest of their lives. Hearing loss can lead to a number of devastating consequences. Exchange of information with others, an important aspect of everyday life, can be seriously impaired in individuals with hearing loss [4]. These difficulties with communication can lead to a perceived reduction in quality of life, especially among young people with hearing loss [5]. Without having the ability to hear the sounds that are surround them, deaf people are under significant emotional and physical stress. In particular, the reduced ability to identify alarming or warning sounds in a timely manner places the hearing impaired in vulnerable situations. In 1981, a 14-year-old deaf girl died in a house fire, oblivious to warnings shouted by a man and the smoke alarm in her bedroom. Last year, in Richfield, Ohio, a deaf woman was killed in a house fire due her not detecting the smoke alarm. Like these examples, countless hearing impaired lives have been jeopardized in situations where they were unaware of the developing dangers or critical situations. Hence, there is a pressing need for an innovation that increases their awareness of critical environment sounds; to give the hearing impaired safety and a peace of mind. 1.2 Proposed Solution AcoustiGLASS, a wearable acoustic pattern recognition system was designed for the hearing impaired. This opto-acoustic device identifies alarming sounds or basic human commands/greetings and notifies the user of such environmental sounds through unique light patterns (e.g., color, blink, fade) and a vibration behind the ear. The device simultaneously sends a descriptive text message of the sound object to the user’s smart device such as smartphone for fuller understanding, which serves as an instant visual display.

1 See Supporting Document 1 for citations.

Figure 1: Autonomous sound object-recognition device with optical fiber-linked LED indicator in a Google Glass-inspired sleek design. A pair of microphone behind the ears permits increased echolocation. User’s smart device becomes an instant display for describing the object. In this concept picture, siren from an approaching police car is being detected.

2

2. SYSTEM OVERVIEW The AcoustiGLASS technology is described in Figure 1. AcoustiGLASS is a sleek and user-friendly pair of glasses, equipped with built-in microphones, an integral vibrator for tactile alarm and optical fiber-linked color LEDs for visual alarm. AcoustiGLASS implements a sophisticated signal-processing algorithm that performs real-time object identification. Incoming sounds are fed into the device to generate time-varying power spectra, i.e., spectrograms, which are applied to a 2-D graphic pattern recognition procedure. When a specific pattern corresponds with one of the pre-loaded references, a visual alarm is displayed to the hearing impaired user through distinct light patterns in the user’s peripheral field of vision. Additionally, a descriptive text message is sent to their smartphone or smart watch via Bluetooth 2.1. A vibrator on the frame of the glasses is used for additional alarm. In this way, AcoustiGLASS can alert the hearing impaired wearer of cautionary or alarming sounds, forewarning them to changing situations, approaching objects, or even developing dangers. AcoustiGLASS also helps the user to respond when someone calls out to the user, particularly from the back. A pair of microphones placed on the glass frame functions like human ears in permitting echolocation, significantly increasing the ability of the hearing impaired to comprehend a critical situation in timely manner. A smartphone app developed for the device lets users to upgrade the software, update sound library, customize light patterns, and synchronize the device for user-specific sounds such as sirens of local fire trucks or doorbells. A battery-charging docking station for AcoustiGLASS turns into a nightly smoke detector with a bed/pillow shaker. Note that AcoustiGLASS differs from a speech recognition or sign language interpreting technology, which is not designed as a first responder.

3. COMMERCIALIZATION ASSESSMENT 3.1 Profile of Customer Customers would include the functionally deaf, hard-of-hearing individuals, special-needs parents, educators, schools for the deaf, and assisted living facilities.

Figure 2: AcoustiGLASS system block diagram.

3

3.2 Customer Feedback The AcoustiGLASS concept was presented to members of the Hearing Loss Association of America (HLAA) and published in HLAA’s monthly e-newsletter (March 23, 2016), which went out to about 22,000 subscribers. A number of individuals expressed strong interest in the technology, including the marketing director, David Hutchesonof HLAA. Notably, a deaf lady, R. Delgado thanked the authors for their effort in developing the technology and personally confirmed that AcoustiGLASS is definitely a product needed by deaf people. She even stated that she was willing to be a test subject to use a prototype of the system before it entered the market. 3.3 Potential Competitors and Relevant Technology Three state-of-the-art devices have been identified as potential competitors: hearing aids, personal sound amplifiers products (PSAPs), and smoke alarm aid bed shakers. Many hi-tech hearing aids have been developed and approved by the U.S. Food and Drug Association (FDA), with the primary purpose of improving hearing loss. These include Halo (Starkey Hearing Tech) and Resound Linx2 (GN ReSound), both of which are wireless hearing aids that can connect directly to any Apple or Android mobile device. The primary aim of these devices is to improve sound quality and audibility for the user and easier noisy environment listening, while allowing the user to make discreet sound enhancements through the connected mobile device. Examples of advanced PSAPs include Smart Listening System (Soundhawk, CA) and Personal Sound Amplifier CS50+ (Sound World Solutions, IL). By definition, PSAPs amplify environmental sounds for the user. PSAPs are in-ear devices that act as “smart listening” systems by connecting with the user’s smartphone using Bluetooth to modify how he or she hears through a combination of directional noise reduction and volume amplification. However, the FDA currently classifies these devices as recreational for non-hearing impaired individuals and not as a medical device. Many people with hearing impairment do use these devices because they are significantly cheaper than hearing aids. Smoke alarm aid bed-shakers are stationary devices that attempt wake a person from sleep to alert him or her to smoke and fire. An example is Safeawake Smoke Alarm (SafeAwake). It works with the house’s smoke alarm system, using a wired bed shaker and a flashing light to alarm deaf users of fire. Nonetheless, many users complain about false alarms, which grows annoying and makes the device ineffective for the purpose. Additionally, this device is not wearable and has no applications to any other kinds of sounds. 3.4 Unique Value Proposition AcoustiGLASS is an immensely beneficial innovation to the deaf and hearing impaired. While the competitors’ products are mainly designed to improve sound quality for hard-of-hearing, none of these technologies are intended to provide a solution to deafness (>90 dB) or severe hearing impairment (>40 dB) at relatively high pitches, which frequency range is often used for alarming sounds. AcoustiGLASS is the only personal, wearable alert system discreetly designed for deafness and heavy hearing loss. AcoustiGLASS offers detection of 100’s different environmental sounds with dozens of variations for each sound. Estimated retail price of AcoustiGLASS is comparable with the existing hi-tech hearing aids (Resound Linx2 hearing aids $2,350; hiHealthInnovations hi-BTE hearing aids, $1,758), PSAPs (CS50+, $349) and smoke alarm aid bed shakers (Safeawake, $250). AcoustiGLASS does not incur any fitting cost (e.g. ear mold) by a specialist that is necessary for most of hearing aids. While Google Glass was pulled out of the market due to safety and privacy concerns, an

4

enterprise project (Glass-at-Work) continues with several certified providers1 such as AMA, APX Labs, GuidiGo, and Pristine. None of the listed providers is pursuing applications in hearing aids. 4. DESCRIPTION OF REVENUE MODEL 4.1 Principal Revenue Streams Expected The principal revenues are expected to come from the sale of the product (AcoustiGLASS and accessories) directly to customers as well as through retail stores (e.g., Amazon.com, Wal-Mart), industrial supplies distributors (e.g., Grainger), and flexible spending account stores (e.g., fsastore.com). Upon FDA approval, AcoustiGLASS can be sold as hearing aids and thus revenue partially comes from health insurance’s co-payment. Software upgrades and sound library updates will be provided for free-of-charge rather than a typical subscription fee model because the system is considered a safety device. 4.1.1 Unit Variable Cost The unit cost of the product is estimated to be $207 in total before potential pretax profit and calculated as follows. Materials amount to $71.80 total: [Glass] mini vibration disk motor 2 mm (Adafruit, $1.20), commercial-grade optical fiber 20 cm ($4.50), RGB LEDs (SparkFun, $1.90), microcontroller (Tinycircuits, $4.50), memory chip (Micron Technology, $4.80), application chip (Freelance Semiconductor, $8.00), flash memory chip (Samsung, $3.80), Bluetooth chip (Cambridge Silicon Radio, $1.75), microphone chip (SparkFun, $11.90), silicon nose pads ($1.20), rechargeable battery ($10); [Docking station] USB charging port ($1.5), power cable and AC/DC converter (Texas Instruments $5), WiFi 802.11 chip (ESP8266, $1.75), bed shaker (OEM, $10). Electronic manufacturing (service contract) amounts to $90 per unit. Product assembly, i.e., framing, plastic/metal molding, fabrication, and quality control (service contract) amounts to $45 per unit. 4.1.2 Product Selling Price and Unit Profit Margin AcoustiGLASS including all accessories will retail for $420 per unit. The unit profit margin will be $180. Equation: ($420 x 92%2) – Unit Variable Cost $207 = $180.

4.2 Fixed Costs 4.2.1 Principal Startup and Development Costs (3-year time span) Principal startup and development costs include starting up the company, prototyping, validation, marketing, and business agreements. The first year is intended for starting up the company and prototyping. A provisional patent to protect the technology costs $300. Registration as a State of Ohio Limited Liability Corporation (LLC) costs $125. Prototyping would entail refining the industrial design, coding the computer program, and building printed-circuit board (PCB) of the system. Electronic manufacturing service such as Nexlogic (San Jose, CA) will be used for PCB prototyping. Plastic/metal molding of the frame will utilize a public free facility equipped with state-of-the-art graphic design software, fabricating machines, and 3D printers like Sears think[box] at Case Western University campus in Cleveland, OH. Web development of the smartphone app must also be accomplished. This prototyping will cost $7,650. In the second year, the prototype will be validated ($5,900). Marketing and advertisement will also be done in the second year ($2,400). The 1 The team considers collaboration with one of the providers as alternative business model for AcoustiGLASS commercialization. 2 Retail charge of 8% is considered (such as “referral fee” for consumer electronics charged to sellers at Amazon.com).

5

third year of this commercialization plan includes patent acquisition and business agreements ($16,000). The team will apply for $35,000 seed funding from Great Lakes Innovation & Development Enterprise (GLIDE) and acquire a $20,000 bank loan to cover the above costs. 4.2.2 Operating Costs The operating costs are estimated to be about $5,500 to $6,000 per year for the first two years. In the third year, operating costs will increase to $11,500 for the year. Operating costs include salary to the authors, office rental and utilities, insurance, various taxes (property, income, operation, etc.), and office equipment/supplies. Operating costs for fourth year, which is when sales start, are estimated to be approximately $365,000, mainly due to manufacturing service contracts ($202,860) and increased employment ($149,000). 4.2.3 Breakeven Volume The breakeven volume is expected to be approximately 840 units per year. Operating costs (sales begin at the fourth year): $164,480 / Unit profit: $146 = 1,127. See Supporting Document 2 for detailed financial analysis. 5. TECHNOLOGY PROOF OF CONCEPT A software model for acoustic pattern recognition was developed and validated in MATLAB/Simulink environment. The real-time audio processing code (converted to C/C++ language in the background) was then deployed to a research model (Raspberry-Pi 2) and tested for sound detection rate and response time (processing speed). See Supporting Document 3 for a comprehensive description of the hardware model and software validation. The code works as follows: A buffer overlaps incoming sound waves. A Periodogram function estimates the Power Spectral Density (PSD) of the wave signals through a fast Fourier transform (FFT). A second buffer function constructs the PSD data into multi-dimensional spectrogram arrays. These arrays are then cross-correlated with pre-loaded reference spectrograms in order to identify specific sounds objects (user-recorded or online sound libraries). In essence, this algorithm is a two-dimensional image cross-correlation, or image pattern matching via peak-detection (prominence, peak height, and location). This approach has proven to very accurate for the purpose as shown in our detection true/false test (see Supporting Document 3). 6. TEAM’S RELEVANT EXPERIENCES, SKILLS, AND RESOURCES Kei Kojima, team’s engineer and designer, conceived the present concept in 2014 and built the first model on Mac OS-X with Arduino Uno in 2015. Kei presented his first stand-alone hardware model in MATLAB/Simulink and received a Physical Science Grand Prize at Northeastern Ohio Science and Engineering Fair in Cleveland, OH in 2016. His project also won Broadcom MASTERS Semi-Finalist (top 300 out of 6,000 nominees). Kotaro Kojima, team’s business development lead, has become an all-state winner (out of 10,000 entries) for a novel low-emission biodiesel engine technology in Ohio Academy of Science’s entrepreneurship STEM commercialization contest in 2016. During this contest, Kotaro filed a Provisional Patent #62/324,981 (4/20/2016) on his diesel engine concept. He also became an all-state finalist in Buckeye Science and Engineering Fair at Ohio State University in 2016 with his engineering project. Kotaro has completed Introduction to Entrepreneurship course at University of Akron, OH and received an A grade. Both team members participated and received special recognition at Biomedical Entrepreneurship summer camp and National Biomechanics Day at University of Akron in 2016. The team has worked with an agent from JumpStart, an entrepreneur assistant program based in Cleveland, OH.