Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions

Yuki Miyashita; Toshiharu Sugawara

doi:10.1007/978-3-030-30487-4_42

Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions

Yuki Miyashita^*, Toshiharu Sugawara

^*Corresponding author for this work

School of Fundamental Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

We investigate the coordination structures generated by deep Q-network (DQN) with various types of input by using a distributed task execution game. Although cooperation and coordination are mandatory for efficiency in multi-agent systems (MAS), they require sophisticated structures or regimes for effective behaviors. Recently, deep Q-learning has been applied to multi-agent systems to facilitate their coordinated behavior. However, the characteristics of the learned results have not yet been fully clarified. We investigate how information input to DQNs affect the resultant coordination and cooperation structures. We examine the inputs generated from local observations with and without the estimated location in the environment. Experimental results show that they form two types of coordination structures—the division of labor and the targeting of near tasks while avoiding conflicts—and that the latter is more efficient in our game. We clarify the mechanism behind and the characteristics of the generated coordination behaviors.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2019
Subtitle of host publication	Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings
Editors	Igor V. Tetko, Pavel Karpov, Fabian Theis, Vera Kurková
Publisher	Springer Verlag
Pages	541-554
Number of pages	14
ISBN (Print)	9783030304867
DOIs	https://doi.org/10.1007/978-3-030-30487-4_42
Publication status	Published - 2019
Event	28th International Conference on Artificial Neural Networks, ICANN 2019 - Munich, Germany Duration: 2019 Sept 17 → 2019 Sept 19

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11727 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	28th International Conference on Artificial Neural Networks, ICANN 2019
Country/Territory	Germany
City	Munich
Period	19/9/17 → 19/9/19

Keywords

Cooperation
Coordination
Divisional cooperation
Multi-agent deep reinforcement learning

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-30487-4_42

Cite this

Miyashita, Y., & Sugawara, T. (2019). Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions. In I. V. Tetko, P. Karpov, F. Theis, & V. Kurková (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings (pp. 541-554). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11727 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-30487-4_42

Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions. / Miyashita, Yuki; Sugawara, Toshiharu.
Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings. ed. / Igor V. Tetko; Pavel Karpov; Fabian Theis; Vera Kurková. Springer Verlag, 2019. p. 541-554 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11727 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Miyashita, Y & Sugawara, T 2019, Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions. in IV Tetko, P Karpov, F Theis & V Kurková (eds), Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11727 LNCS, Springer Verlag, pp. 541-554, 28th International Conference on Artificial Neural Networks, ICANN 2019, Munich, Germany, 19/9/17. https://doi.org/10.1007/978-3-030-30487-4_42

Miyashita Y, Sugawara T. Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions. In Tetko IV, Karpov P, Theis F, Kurková V, editors, Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings. Springer Verlag. 2019. p. 541-554. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-30487-4_42

Miyashita, Yuki ; Sugawara, Toshiharu. / Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions. Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, 2019, Proceedings. editor / Igor V. Tetko ; Pavel Karpov ; Fabian Theis ; Vera Kurková. Springer Verlag, 2019. pp. 541-554 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{1bb9ba427cd742158f18f649bc6ac5a1,

title = "Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions",

abstract = "We investigate the coordination structures generated by deep Q-network (DQN) with various types of input by using a distributed task execution game. Although cooperation and coordination are mandatory for efficiency in multi-agent systems (MAS), they require sophisticated structures or regimes for effective behaviors. Recently, deep Q-learning has been applied to multi-agent systems to facilitate their coordinated behavior. However, the characteristics of the learned results have not yet been fully clarified. We investigate how information input to DQNs affect the resultant coordination and cooperation structures. We examine the inputs generated from local observations with and without the estimated location in the environment. Experimental results show that they form two types of coordination structures—the division of labor and the targeting of near tasks while avoiding conflicts—and that the latter is more efficient in our game. We clarify the mechanism behind and the characteristics of the generated coordination behaviors.",

keywords = "Cooperation, Coordination, Divisional cooperation, Multi-agent deep reinforcement learning",

author = "Yuki Miyashita and Toshiharu Sugawara",

note = "Funding Information: This work was partly supported by JSPS KAKENHI Grant Number 17KT0044. Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 28th International Conference on Artificial Neural Networks, ICANN 2019 ; Conference date: 17-09-2019 Through 19-09-2019",

year = "2019",

doi = "10.1007/978-3-030-30487-4_42",

language = "English",

isbn = "9783030304867",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "541--554",

editor = "Tetko, {Igor V.} and Pavel Karpov and Fabian Theis and Vera Kurkov{\'a}",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2019",

}

TY - GEN

T1 - Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions

AU - Miyashita, Yuki

AU - Sugawara, Toshiharu

PY - 2019

Y1 - 2019

N2 - We investigate the coordination structures generated by deep Q-network (DQN) with various types of input by using a distributed task execution game. Although cooperation and coordination are mandatory for efficiency in multi-agent systems (MAS), they require sophisticated structures or regimes for effective behaviors. Recently, deep Q-learning has been applied to multi-agent systems to facilitate their coordinated behavior. However, the characteristics of the learned results have not yet been fully clarified. We investigate how information input to DQNs affect the resultant coordination and cooperation structures. We examine the inputs generated from local observations with and without the estimated location in the environment. Experimental results show that they form two types of coordination structures—the division of labor and the targeting of near tasks while avoiding conflicts—and that the latter is more efficient in our game. We clarify the mechanism behind and the characteristics of the generated coordination behaviors.

AB - We investigate the coordination structures generated by deep Q-network (DQN) with various types of input by using a distributed task execution game. Although cooperation and coordination are mandatory for efficiency in multi-agent systems (MAS), they require sophisticated structures or regimes for effective behaviors. Recently, deep Q-learning has been applied to multi-agent systems to facilitate their coordinated behavior. However, the characteristics of the learned results have not yet been fully clarified. We investigate how information input to DQNs affect the resultant coordination and cooperation structures. We examine the inputs generated from local observations with and without the estimated location in the environment. Experimental results show that they form two types of coordination structures—the division of labor and the targeting of near tasks while avoiding conflicts—and that the latter is more efficient in our game. We clarify the mechanism behind and the characteristics of the generated coordination behaviors.

KW - Cooperation

KW - Coordination

KW - Divisional cooperation

KW - Multi-agent deep reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85072870308&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072870308&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-30487-4_42

DO - 10.1007/978-3-030-30487-4_42

M3 - Conference contribution

AN - SCOPUS:85072870308

SN - 9783030304867

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 541

EP - 554

BT - Artificial Neural Networks and Machine Learning – ICANN 2019

A2 - Tetko, Igor V.

A2 - Karpov, Pavel

A2 - Theis, Fabian

A2 - Kurková, Vera

PB - Springer Verlag

T2 - 28th International Conference on Artificial Neural Networks, ICANN 2019

Y2 - 17 September 2019 through 19 September 2019

ER -

Cooperation and Coordination Regimes by Deep Q-Learning in Multi-agent Task Executions

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this