Materials Horizons Emerging Investigator Series: Dr Shanshan Yao, Stony Brook University, USA

Dr Shanshan Yao received her BS and MS degrees from Xi’an Jiaotong University in 2009 and 2012, respectively. She received her PhD in mechanical engineering from North Carolina State University in 2016. She was a postdoctoral research scholar at North Carolina State University from 2017 to 2019. She is currently an assistant professor in the Department of Mechanical Engineering at Stony Brook University. Her research is primarily in the areas of functional materials, smart structures, advanced manufacturing, and soft electronics. Yao's research aims to provide new solutions and explore new applications in personal healthcare, fitness tracking, rehabilitation, soft robotics, and entertainment through combined innovations in materials engineering, mechanical design, and multi-scale manufacturing and integration. Yao received the Faculty Early Career Development (CAREER) Award from the National Science Foundation (NSF) in 2023.

Read Shanshan Yao's Emerging Investigator Series article ‘Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning’ ( https://doi.org/10.1039/D3MH01062G ) and read more about her in the interview below:

MH: Your recent Materials Horizons Communication uses a soft magnetic skin as a silent speech interface. How has your research evolved from your first article to this most recent article and where do you see your research going in future?

SY: The motivation for this work was triggered by a colleague whose spouse suffers from amyotrophic lateral sclerosis (ALS, also known as Lou Gehrig's disease) which affects speech capability. Many of the objectives of our research are to build a system that can restore the spoken communication of the spouse with loved ones and healthcare providers. Existing silent speech systems face several challenges, including bulkiness, obtrusiveness, immobility, and poor robustness against interferences. The lack of a truly efficient and unobtrusive system for interpreting silent speech precludes its continuous use and wide-scale deployment. We started our research on wearable silent speech interfaces in 2021. We first explored stretchable biopotential sensors to monitor and decode muscle activities for speech recognition. Then we developed wireless magnetic skin enabled silent speech interfaces, which require significantly fewer sensors for speech monitoring and the sensing location is much more user-friendly and socially acceptable.

MH: What aspect of your work are you most excited about at the moment?

SY: We are excited to finally develop a device that can address the concerns related to bulkiness, obtrusiveness, and poor robustness of other silent speech systems. Our method eliminates complicated setups and line-of-sight constraints required for visual and acoustic-based methods. Our system is robust under noisy and dark environments and can recognize words from different or the same viseme groups via machine learning algorithms. With only one piece of soft magnetic skin placed behind the ear, the silent speech interface allows for a wireless, unobtrusive, user-friendly system for daily use.

MH: In your opinion, what are the most important questions to be asked/answered in this field of research?

SY: In terms of hardware, although better sensors have been developed for silent speech, the hardware used for collecting speech-related data is not as soft, lightweight, and portable. More effort is required to develop fully skin-compatible and user-acceptable silent speech systems. In terms of software, despite the good stretchability and wearability of the sensors, advanced signal processing techniques for the new sensors are lacking. The machine learning algorithm should identify each phoneme in a word and have the ability to construct new words using phonemes. In the soft electronic community, most of the speech recognition methods are indeed classifiers. This method can only be used for specific applications requiring a small dataset. Signals corresponding to all phonemes/words/sentences should be collected and pre-trained. Extending the training set to hundreds or thousands of classes can be very time-consuming and inefficient. Any unseen (i.e., untrained) class beyond the training vocabulary will not be recognized. To expand the silent speech interfaces for practical applications, the phoneme-based speech recognition method is in demand. With this new method, the training set will include limited sets of words or sentences containing different phonemes. Any unseen words will be synthesized from pre-trained phonemes. We expect that the novel wearable sensors, together with more advanced machine learning algorithms, are capable of realizing silent speech recognition of large vocabularies for daily communications.

MH: What do you find most challenging about your research?

SY: The multidisciplinary nature of the research requires training and research experience in a diverse field of engineering and science, including materials science (to design new sensors), mechanical engineering (to ensure stretchability and mechanical robustness), electrical engineering (to develop data acquisition systems), computer science (to develop signal processing methods), and biomedical engineering (to understand speech articulation). My research team needs to expose ourselves to different fields. This makes the research challenging, but at the same time, makes our research exciting and interesting.

MH: In which upcoming conferences or events may our readers meet you?

SY: I will be at the International Mechanical Engineering Congress and Exposition (IMECE) conference this year. I plan to attend the Materials Research Society (MRS) and Society of Engineering Science (SES) conferences next year.

MH: How do you spend your spare time?

SY: I like travelling, baking, and aerobics in my spare time. I enjoy travelling with my son and my entire family.

MH: Can you share one piece of career-related advice or wisdom with other early career scientists?

SY: It is crucial to find an important problem you are passionate about and a problem that is worth exploring.

---