Edged based audio-visual speech enhancement demonstrator
Abstract
Difficulty understanding speech in noisy environments presents a significant challenge for individuals with hearing loss and is a primary factor contributing to non-adherence to hearing aid use. Recent technological advancements integrating artificial intelligence, machine learning, and smartphone technology hold promise in advancing and customizing hearing healthcare. A proposed solution is a portable hearing assistive system designed for speech enhancement in noisy settings. We anticipate that this system will enhance the auditory experience of hearing aid users. The system leverages a mobile phone’s camera, microphone, and speaker, ensuring ease of portability. Raw video and audio data are stored locally on the phone and processed by the device’s processor alongside an audio-visual speech enhancement algorithm. This algorithm is capable of identifying voice signals and lip movements using a lightweight deep neural network model, thereby optimizing memory efficiency required for real-time processing.