Pub Date : 2026-10-19DOI: 10.1007/s40820-026-02346-x
Ziwei Huo, Nuo Xu, Yixuan Fu, Rui Gu, Chao Liu, Zhong Lin Wang, Qijun Sun
Silent speech decoding (lip reading) represents an innovative hands-free communication method for individuals with laryngeal and lingual impairments, addressing critical daily communication challenges. Herein, a novel silent speech decoding system is proposed, which employs a self-powered, low-cost, and conformable triboelectric-based E‑skin patch and a well-trained multi-layer convolutional neural network (CNN) model. We systematically characterize the structural principles and electrical performance of the E-skin patch. A comparative analysis of labial kinematics during vocalized and silent speech is also conducted, which demonstrates strong consistency in time-frequency domain characteristics. The utilized CNN model is trained across 15 lexical categories (200 samples per category), achieving a high accuracy of 98.75% and demonstrating excellent recognition performance. Furthermore, by integrating human-machine interaction (HMI) and virtual reality (VR) technologies, we develop three representative applications, including a real-time lip language visualization system, an orofacial-controlled "Feeding Frenzy" interactive game, and a VR communication platform with avatar dialogue. This technology enables barrier-free communication for voiceless populations, demonstrating significant potential in HMI, augmentative communication systems, virtual reality integration, and biomedical rehabilitation.
{"title":"Triboelectric E-Skin Patch Decoding Silent Speech for Advanced Human-Machine Interaction and Augmented Virtual Communication.","authors":"Ziwei Huo, Nuo Xu, Yixuan Fu, Rui Gu, Chao Liu, Zhong Lin Wang, Qijun Sun","doi":"10.1007/s40820-026-02346-x","DOIUrl":"10.1007/s40820-026-02346-x","url":null,"abstract":"<p><p>Silent speech decoding (lip reading) represents an innovative hands-free communication method for individuals with laryngeal and lingual impairments, addressing critical daily communication challenges. Herein, a novel silent speech decoding system is proposed, which employs a self-powered, low-cost, and conformable triboelectric-based E‑skin patch and a well-trained multi-layer convolutional neural network (CNN) model. We systematically characterize the structural principles and electrical performance of the E-skin patch. A comparative analysis of labial kinematics during vocalized and silent speech is also conducted, which demonstrates strong consistency in time-frequency domain characteristics. The utilized CNN model is trained across 15 lexical categories (200 samples per category), achieving a high accuracy of 98.75% and demonstrating excellent recognition performance. Furthermore, by integrating human-machine interaction (HMI) and virtual reality (VR) technologies, we develop three representative applications, including a real-time lip language visualization system, an orofacial-controlled \"Feeding Frenzy\" interactive game, and a VR communication platform with avatar dialogue. This technology enables barrier-free communication for voiceless populations, demonstrating significant potential in HMI, augmentative communication systems, virtual reality integration, and biomedical rehabilitation.</p>","PeriodicalId":714,"journal":{"name":"Nano-Micro Letters","volume":"19 1","pages":""},"PeriodicalIF":36.3,"publicationDate":"2026-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149219782","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"材料科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}