A Cloud-Based Sign Language Translation System via CNN with Smart Glasses

Siwei Zhao*, Jun Wang, Kiminori Sato, Bo Wu, Xuan Huang

*この研究の対応する著者

研究成果: Conference contribution

抄録

In situations where ordinary people and the hearing-impaired person need to communicate, it is possible that the average person does not know sign language and thus communication may be impaired, which means that a technology or device to assist communication is needed. Therefore, this study develops a new cloud sign language translation system on smart device based on the Browser/Server architecture, so that when the hearing-impaired person makes a sign language movement in front of the user who using the system with a smart device (e.g., smart glasses), the screen of the smart device will display the subtitle of the sign languages. We use MediaPipe to recognize and collect sign language action data from WLASL dataset and provide it to TensorFlow’s 1D-CNN deep learning model for training, so as to realize the sign language translation function. In the test phase, we invited five experimenters to test the sign language translation system ten times for each person, and the final average accuracy rate was 72%. Through such an interpreting system, it brings a convenient and efficient communication experience to the hearing-impaired person and people who need to communicate with the hearing-impaired person, and at the same time, it can also provide support for broader sign language research and application.

本文言語English
ホスト出版物のタイトルGreen, Pervasive, and Cloud Computing - 18th International Conference, GPC 2023, Proceedings
編集者Hai Jin, Zhiwen Yu, Chen Yu, Xiaokang Zhou, Zeguang Lu, Xianhua Song
出版社Springer Science and Business Media Deutschland GmbH
ページ224-237
ページ数14
ISBN(印刷版)9789819998951
DOI
出版ステータスPublished - 2024
イベント18th International Conference on Green, Pervasive, and Cloud Computing, GPC 2023 - Harbin, China
継続期間: 2023 9月 222023 9月 24

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
14504
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

Conference

Conference18th International Conference on Green, Pervasive, and Cloud Computing, GPC 2023
国/地域China
CityHarbin
Period23/9/2223/9/24

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータサイエンス一般

フィンガープリント

「A Cloud-Based Sign Language Translation System via CNN with Smart Glasses」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル