Word segmentation for the sequences emitted from a word-valued source

Takashi Ishida; Toshiyasu Matsushima; Shigeichi Hirasawa

doi:10.1109/CIT.2007.4385160

Word segmentation for the sequences emitted from a word-valued source

Takashi Ishida^*, Toshiyasu Matsushima, Shigeichi Hirasawa

^*Corresponding author for this work

School of Fundamental Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Word segmentation is the most fundamental and important process for Japanese or Chinese language processing. Because there is no separation between words in these languages, we firstly have to separate the sequence into words. On this problem, it is known that the approach by probabilistic language model is highly efficient, and this is shown practically. On the other hand, recently, a word-valued source has been proposed as a new class of source model for the source coding problem. This model can be supposed to reflect more of the probability structure of natural languages. We may regard Japanese sentence or Chinese sentence as the sequence emitting from a non-prefix-free WVS. In this paper, as the first phase of applying WVS to natural language processing, we formulate a word segmentation problem for the sequence from non-prefix-free WVS. Then, we examine the performance of word segmentation for the models by numerical computations.

Original language	English
Title of host publication	CIT 2007
Subtitle of host publication	7th IEEE International Conference on Computer and Information Technology
Pages	662-667
Number of pages	6
DOIs	https://doi.org/10.1109/CIT.2007.4385160
Publication status	Published - 2007
Event	CIT 2007: 7th IEEE International Conference on Computer and Information Technology - Aizu-Wakamatsu, Fukushima, Japan Duration: 2007 Oct 16 → 2007 Oct 19

Publication series

Name	CIT 2007: 7th IEEE International Conference on Computer and Information Technology

Conference

Conference	CIT 2007: 7th IEEE International Conference on Computer and Information Technology
Country/Territory	Japan
City	Aizu-Wakamatsu, Fukushima
Period	07/10/16 → 07/10/19

ASJC Scopus subject areas

Computer Science Applications
Information Systems
Software
Mathematics(all)

Access to Document

10.1109/CIT.2007.4385160

Cite this

Word segmentation for the sequences emitted from a word-valued source. / Ishida, Takashi; Matsushima, Toshiyasu; Hirasawa, Shigeichi.
CIT 2007: 7th IEEE International Conference on Computer and Information Technology. 2007. p. 662-667 4385160 (CIT 2007: 7th IEEE International Conference on Computer and Information Technology).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ishida, T, Matsushima, T & Hirasawa, S 2007, Word segmentation for the sequences emitted from a word-valued source. in CIT 2007: 7th IEEE International Conference on Computer and Information Technology., 4385160, CIT 2007: 7th IEEE International Conference on Computer and Information Technology, pp. 662-667, CIT 2007: 7th IEEE International Conference on Computer and Information Technology, Aizu-Wakamatsu, Fukushima, Japan, 07/10/16. https://doi.org/10.1109/CIT.2007.4385160

@inproceedings{1376624a4c2d46dea7e41abc625429d8,

title = "Word segmentation for the sequences emitted from a word-valued source",

abstract = "Word segmentation is the most fundamental and important process for Japanese or Chinese language processing. Because there is no separation between words in these languages, we firstly have to separate the sequence into words. On this problem, it is known that the approach by probabilistic language model is highly efficient, and this is shown practically. On the other hand, recently, a word-valued source has been proposed as a new class of source model for the source coding problem. This model can be supposed to reflect more of the probability structure of natural languages. We may regard Japanese sentence or Chinese sentence as the sequence emitting from a non-prefix-free WVS. In this paper, as the first phase of applying WVS to natural language processing, we formulate a word segmentation problem for the sequence from non-prefix-free WVS. Then, we examine the performance of word segmentation for the models by numerical computations.",

author = "Takashi Ishida and Toshiyasu Matsushima and Shigeichi Hirasawa",

year = "2007",

doi = "10.1109/CIT.2007.4385160",

language = "English",

isbn = "0769529836",

series = "CIT 2007: 7th IEEE International Conference on Computer and Information Technology",

pages = "662--667",

booktitle = "CIT 2007",

note = "CIT 2007: 7th IEEE International Conference on Computer and Information Technology ; Conference date: 16-10-2007 Through 19-10-2007",

}

TY - GEN

T1 - Word segmentation for the sequences emitted from a word-valued source

AU - Ishida, Takashi

AU - Matsushima, Toshiyasu

AU - Hirasawa, Shigeichi

PY - 2007

Y1 - 2007

N2 - Word segmentation is the most fundamental and important process for Japanese or Chinese language processing. Because there is no separation between words in these languages, we firstly have to separate the sequence into words. On this problem, it is known that the approach by probabilistic language model is highly efficient, and this is shown practically. On the other hand, recently, a word-valued source has been proposed as a new class of source model for the source coding problem. This model can be supposed to reflect more of the probability structure of natural languages. We may regard Japanese sentence or Chinese sentence as the sequence emitting from a non-prefix-free WVS. In this paper, as the first phase of applying WVS to natural language processing, we formulate a word segmentation problem for the sequence from non-prefix-free WVS. Then, we examine the performance of word segmentation for the models by numerical computations.

AB - Word segmentation is the most fundamental and important process for Japanese or Chinese language processing. Because there is no separation between words in these languages, we firstly have to separate the sequence into words. On this problem, it is known that the approach by probabilistic language model is highly efficient, and this is shown practically. On the other hand, recently, a word-valued source has been proposed as a new class of source model for the source coding problem. This model can be supposed to reflect more of the probability structure of natural languages. We may regard Japanese sentence or Chinese sentence as the sequence emitting from a non-prefix-free WVS. In this paper, as the first phase of applying WVS to natural language processing, we formulate a word segmentation problem for the sequence from non-prefix-free WVS. Then, we examine the performance of word segmentation for the models by numerical computations.

UR - http://www.scopus.com/inward/record.url?scp=38049025202&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=38049025202&partnerID=8YFLogxK

U2 - 10.1109/CIT.2007.4385160

DO - 10.1109/CIT.2007.4385160

M3 - Conference contribution

AN - SCOPUS:38049025202

SN - 0769529836

SN - 9780769529837

T3 - CIT 2007: 7th IEEE International Conference on Computer and Information Technology

SP - 662

EP - 667

BT - CIT 2007

T2 - CIT 2007: 7th IEEE International Conference on Computer and Information Technology

Y2 - 16 October 2007 through 19 October 2007

ER -

Word segmentation for the sequences emitted from a word-valued source

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this