A highly stretchable and sensitive strain sensor for lip-reading extraction and speech recognition

Abstract

Speech recognition often relies on visual stereo images and videos, but it is often limited by light intensity and opaque occlusions such as masks (such as during the coronavirus disease 2019 (COVID-19) pandemic). Lip speech is produced by the movement of a series of muscles in the mouth. Therefore, lip-reading extraction and decoding can be achieved by attaching non-invasive sensors to capture muscle movement. Herein, we prepared a highly sensitive and stretchable CuNW-rGO/PDMS piezoresistive strain sensor based on a biomimetic pearl “brick-mortar” structure by using two-dimensional rGO and conductive stretchable one-dimensional Cu NWs as mixed “bricks” and stretchable PDMS as the “mortar”. The strain sensor shows a high sensitivity with a gauge factor of 26 in the stretching ability range of up to 70% and high durability over 1000 stretching–releasing cycles. This sensor can act as a health monitoring device when attached to human skin, detecting various human motions such as hand clenching, elbow bending, wrist pulse beating and lip-reading motion. Effective recognition of the lip reading signal was achieved by processing the lip-reading signal using a dynamic time regularization algorithm and Euclidean distance. Finally, a full system integration to real-time analyze and feedback speech information with voice broadcast and text display function based on the strain sensor was presented for accurate speech recognition, which opens up a new possibility for translating lip motion to speech or text directly by capturing lip muscle movements. This work presents a promising way to help people with vocal cord lesions or laryngeal and lingual injuries live a happy life with barrier-free communication and will enrich the diversity of lip-language translation systems in human–machine interfaces and silent speech interfaces.