A Cloud-Based Sign Language Translation System via CNN with Smart Glasses

Siwei Zhao*, Jun Wang, Kiminori Sato, Bo Wu, Xuan Huang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In situations where ordinary people and the hearing-impaired person need to communicate, it is possible that the average person does not know sign language and thus communication may be impaired, which means that a technology or device to assist communication is needed. Therefore, this study develops a new cloud sign language translation system on smart device based on the Browser/Server architecture, so that when the hearing-impaired person makes a sign language movement in front of the user who using the system with a smart device (e.g., smart glasses), the screen of the smart device will display the subtitle of the sign languages. We use MediaPipe to recognize and collect sign language action data from WLASL dataset and provide it to TensorFlow’s 1D-CNN deep learning model for training, so as to realize the sign language translation function. In the test phase, we invited five experimenters to test the sign language translation system ten times for each person, and the final average accuracy rate was 72%. Through such an interpreting system, it brings a convenient and efficient communication experience to the hearing-impaired person and people who need to communicate with the hearing-impaired person, and at the same time, it can also provide support for broader sign language research and application.

Original languageEnglish
Title of host publicationGreen, Pervasive, and Cloud Computing - 18th International Conference, GPC 2023, Proceedings
EditorsHai Jin, Zhiwen Yu, Chen Yu, Xiaokang Zhou, Zeguang Lu, Xianhua Song
PublisherSpringer Science and Business Media Deutschland GmbH
Pages224-237
Number of pages14
ISBN (Print)9789819998951
DOIs
Publication statusPublished - 2024
Event18th International Conference on Green, Pervasive, and Cloud Computing, GPC 2023 - Harbin, China
Duration: 2023 Sept 222023 Sept 24

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14504
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference18th International Conference on Green, Pervasive, and Cloud Computing, GPC 2023
Country/TerritoryChina
CityHarbin
Period23/9/2223/9/24

Keywords

  • Deep Learning
  • MediaPipe
  • Sign Language Translation
  • Smart Glasses
  • TensorFlow

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'A Cloud-Based Sign Language Translation System via CNN with Smart Glasses'. Together they form a unique fingerprint.

Cite this