Lipreading using deep bottleneck features for optical and depth images

Satoshi Tamura; Koichi Miyazaki; Satoru Hayamizu

Lipreading using deep bottleneck features for optical and depth images

Satoshi Tamura, Koichi Miyazaki, Satoru Hayamizu

Research output: Contribution to conference › Paper › peer-review

Abstract

This paper investigates a lipreading scheme employing optical and depth modalities, with using deep bottleneck features. Optical and depth data are captured by Microsoft Kinect v2, followed by computing an appearance-based feature set in each modality. A basic feature set is then converted into a deep bottleneck feature using a deep neural network having a bottleneck layer. Multi-stream hidden Marcov models are used for recognition. We evaluated the method using our connected-digit corpus, comparing to our previous method. It is finally found that we could improve lipreading performance by employing deep bottleneck features.

Original language	English
Pages	76-77
Number of pages	2
Publication status	Published - 2017
Externally published	Yes
Event	14th International Conference on Auditory-Visual Speech Processing, AVSP 2017 - Stockholm, Sweden Duration: 2017 Aug 25 → 2017 Aug 26

Conference

Conference	14th International Conference on Auditory-Visual Speech Processing, AVSP 2017
Country/Territory	Sweden
City	Stockholm
Period	17/8/25 → 17/8/26

Keywords

deep bottleneck feature
depth information
lipreading
multi-stream HMM

ASJC Scopus subject areas

Language and Linguistics
Otorhinolaryngology
Speech and Hearing

Cite this

@conference{ed4f7fadefd24c8b9797b673b75803ab,

title = "Lipreading using deep bottleneck features for optical and depth images",

abstract = "This paper investigates a lipreading scheme employing optical and depth modalities, with using deep bottleneck features. Optical and depth data are captured by Microsoft Kinect v2, followed by computing an appearance-based feature set in each modality. A basic feature set is then converted into a deep bottleneck feature using a deep neural network having a bottleneck layer. Multi-stream hidden Marcov models are used for recognition. We evaluated the method using our connected-digit corpus, comparing to our previous method. It is finally found that we could improve lipreading performance by employing deep bottleneck features.",

keywords = "deep bottleneck feature, depth information, lipreading, multi-stream HMM",

author = "Satoshi Tamura and Koichi Miyazaki and Satoru Hayamizu",

note = "Funding Information: A part of this work was supported by JSPS KAKENHI Grant No. 16H03211. Publisher Copyright: {\textcopyright} 2017 14th International Conference on Auditory-Visual Speech Processing, AVSP 2017. All rights reserved.; 14th International Conference on Auditory-Visual Speech Processing, AVSP 2017 ; Conference date: 25-08-2017 Through 26-08-2017",

year = "2017",

language = "English",

pages = "76--77",

}

TY - CONF

T1 - Lipreading using deep bottleneck features for optical and depth images

AU - Tamura, Satoshi

AU - Miyazaki, Koichi

AU - Hayamizu, Satoru

N1 - Funding Information: A part of this work was supported by JSPS KAKENHI Grant No. 16H03211. Publisher Copyright: © 2017 14th International Conference on Auditory-Visual Speech Processing, AVSP 2017. All rights reserved.

PY - 2017

Y1 - 2017

N2 - This paper investigates a lipreading scheme employing optical and depth modalities, with using deep bottleneck features. Optical and depth data are captured by Microsoft Kinect v2, followed by computing an appearance-based feature set in each modality. A basic feature set is then converted into a deep bottleneck feature using a deep neural network having a bottleneck layer. Multi-stream hidden Marcov models are used for recognition. We evaluated the method using our connected-digit corpus, comparing to our previous method. It is finally found that we could improve lipreading performance by employing deep bottleneck features.

AB - This paper investigates a lipreading scheme employing optical and depth modalities, with using deep bottleneck features. Optical and depth data are captured by Microsoft Kinect v2, followed by computing an appearance-based feature set in each modality. A basic feature set is then converted into a deep bottleneck feature using a deep neural network having a bottleneck layer. Multi-stream hidden Marcov models are used for recognition. We evaluated the method using our connected-digit corpus, comparing to our previous method. It is finally found that we could improve lipreading performance by employing deep bottleneck features.

KW - deep bottleneck feature

KW - depth information

KW - lipreading

KW - multi-stream HMM

UR - http://www.scopus.com/inward/record.url?scp=85133442378&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85133442378&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85133442378

SP - 76

EP - 77

T2 - 14th International Conference on Auditory-Visual Speech Processing, AVSP 2017

Y2 - 25 August 2017 through 26 August 2017

ER -

Lipreading using deep bottleneck features for optical and depth images

Abstract

Conference

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this