Evaluating Image-Inspired Poetry Generation

Chao Chung Wu; Ruihua Song; Tetsuya Sakai; Wen Feng Cheng; Xing Xie; Shou De Lin

doi:10.1007/978-3-030-32233-5_42

Evaluating Image-Inspired Poetry Generation

Chao Chung Wu, Ruihua Song^*, Tetsuya Sakai, Wen Feng Cheng, Xing Xie, Shou De Lin

^*Corresponding author for this work

School of Fundamental Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

Creative natural language generation, such as poetry generation, writing lyrics, and storytelling, is appealing but difficult to evaluate. We take the application of image-inspired poetry generation as a showcase and investigate two problems in evaluation: (1) how to evaluate the generated text when there are no ground truths, and (2) how to evaluate nondeterministic systems that output different texts given the same input image. Regarding the first problem, we first design a judgment tool to collect ratings of a few poems for comparison with the inspiring image shown to assessors. We then propose a novelty measurement that quantifies how different a generated text is compared to a known corpus. Regarding the second problem, we experiment with different strategies to approximate evaluating multiple trials of output poems. We also use a measure for quantifying the diversity of different texts generated in response to the same input image, and discuss their merits.

Original language	English
Title of host publication	Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings
Editors	Jie Tang, Min-Yen Kan, Dongyan Zhao, Sujian Li, Hongying Zan
Publisher	Springer
Pages	539-551
Number of pages	13
ISBN (Print)	9783030322328
DOIs	https://doi.org/10.1007/978-3-030-32233-5_42
Publication status	Published - 2019
Event	8th CCF International Conference on Natural Language Processing and Chinese Computing, NLPCC 2019 - Dunhuang, China Duration: 2019 Oct 9 → 2019 Oct 14

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11838 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th CCF International Conference on Natural Language Processing and Chinese Computing, NLPCC 2019
Country/Territory	China
City	Dunhuang
Period	19/10/9 → 19/10/14

Keywords

AI-based creation
Evaluation
Image
Natural language generation
Poetry generation

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-32233-5_42

Cite this

Wu, C. C., Song, R., Sakai, T., Cheng, W. F., Xie, X., & Lin, S. D. (2019). Evaluating Image-Inspired Poetry Generation. In J. Tang, M-Y. Kan, D. Zhao, S. Li, & H. Zan (Eds.), Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings (pp. 539-551). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11838 LNAI). Springer. https://doi.org/10.1007/978-3-030-32233-5_42

Evaluating Image-Inspired Poetry Generation. / Wu, Chao Chung; Song, Ruihua; Sakai, Tetsuya et al.
Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings. ed. / Jie Tang; Min-Yen Kan; Dongyan Zhao; Sujian Li; Hongying Zan. Springer, 2019. p. 539-551 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11838 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Wu, CC, Song, R, Sakai, T, Cheng, WF, Xie, X & Lin, SD 2019, Evaluating Image-Inspired Poetry Generation. in J Tang, M-Y Kan, D Zhao, S Li & H Zan (eds), Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11838 LNAI, Springer, pp. 539-551, 8th CCF International Conference on Natural Language Processing and Chinese Computing, NLPCC 2019, Dunhuang, China, 19/10/9. https://doi.org/10.1007/978-3-030-32233-5_42

Wu CC, Song R, Sakai T, Cheng WF, Xie X, Lin SD. Evaluating Image-Inspired Poetry Generation. In Tang J, Kan M-Y, Zhao D, Li S, Zan H, editors, Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings. Springer. 2019. p. 539-551. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-32233-5_42

Wu, Chao Chung ; Song, Ruihua ; Sakai, Tetsuya et al. / Evaluating Image-Inspired Poetry Generation. Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings. editor / Jie Tang ; Min-Yen Kan ; Dongyan Zhao ; Sujian Li ; Hongying Zan. Springer, 2019. pp. 539-551 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d1d36357a76744e694937e69c09b7929,

title = "Evaluating Image-Inspired Poetry Generation",

abstract = "Creative natural language generation, such as poetry generation, writing lyrics, and storytelling, is appealing but difficult to evaluate. We take the application of image-inspired poetry generation as a showcase and investigate two problems in evaluation: (1) how to evaluate the generated text when there are no ground truths, and (2) how to evaluate nondeterministic systems that output different texts given the same input image. Regarding the first problem, we first design a judgment tool to collect ratings of a few poems for comparison with the inspiring image shown to assessors. We then propose a novelty measurement that quantifies how different a generated text is compared to a known corpus. Regarding the second problem, we experiment with different strategies to approximate evaluating multiple trials of output poems. We also use a measure for quantifying the diversity of different texts generated in response to the same input image, and discuss their merits.",

keywords = "AI-based creation, Evaluation, Image, Natural language generation, Poetry generation",

author = "Wu, {Chao Chung} and Ruihua Song and Tetsuya Sakai and Cheng, {Wen Feng} and Xing Xie and Lin, {Shou De}",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 8th CCF International Conference on Natural Language Processing and Chinese Computing, NLPCC 2019 ; Conference date: 09-10-2019 Through 14-10-2019",

year = "2019",

doi = "10.1007/978-3-030-32233-5_42",

language = "English",

isbn = "9783030322328",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "539--551",

editor = "Jie Tang and Min-Yen Kan and Dongyan Zhao and Sujian Li and Hongying Zan",

booktitle = "Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings",

}

TY - GEN

T1 - Evaluating Image-Inspired Poetry Generation

AU - Wu, Chao Chung

AU - Song, Ruihua

AU - Sakai, Tetsuya

AU - Cheng, Wen Feng

AU - Xie, Xing

AU - Lin, Shou De

PY - 2019

Y1 - 2019

N2 - Creative natural language generation, such as poetry generation, writing lyrics, and storytelling, is appealing but difficult to evaluate. We take the application of image-inspired poetry generation as a showcase and investigate two problems in evaluation: (1) how to evaluate the generated text when there are no ground truths, and (2) how to evaluate nondeterministic systems that output different texts given the same input image. Regarding the first problem, we first design a judgment tool to collect ratings of a few poems for comparison with the inspiring image shown to assessors. We then propose a novelty measurement that quantifies how different a generated text is compared to a known corpus. Regarding the second problem, we experiment with different strategies to approximate evaluating multiple trials of output poems. We also use a measure for quantifying the diversity of different texts generated in response to the same input image, and discuss their merits.

AB - Creative natural language generation, such as poetry generation, writing lyrics, and storytelling, is appealing but difficult to evaluate. We take the application of image-inspired poetry generation as a showcase and investigate two problems in evaluation: (1) how to evaluate the generated text when there are no ground truths, and (2) how to evaluate nondeterministic systems that output different texts given the same input image. Regarding the first problem, we first design a judgment tool to collect ratings of a few poems for comparison with the inspiring image shown to assessors. We then propose a novelty measurement that quantifies how different a generated text is compared to a known corpus. Regarding the second problem, we experiment with different strategies to approximate evaluating multiple trials of output poems. We also use a measure for quantifying the diversity of different texts generated in response to the same input image, and discuss their merits.

KW - AI-based creation

KW - Evaluation

KW - Image

KW - Natural language generation

KW - Poetry generation

UR - http://www.scopus.com/inward/record.url?scp=85075561188&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075561188&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-32233-5_42

DO - 10.1007/978-3-030-32233-5_42

M3 - Conference contribution

AN - SCOPUS:85075561188

SN - 9783030322328

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 539

EP - 551

BT - Natural Language Processing and Chinese Computing - 8th CCF International Conference, NLPCC 2019, Proceedings

A2 - Tang, Jie

A2 - Kan, Min-Yen

A2 - Zhao, Dongyan

A2 - Li, Sujian

A2 - Zan, Hongying

PB - Springer

T2 - 8th CCF International Conference on Natural Language Processing and Chinese Computing, NLPCC 2019

Y2 - 9 October 2019 through 14 October 2019

ER -

Evaluating Image-Inspired Poetry Generation

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this