Reinforcement learning of a continuous motor sequence with hidden states

Hiroaki Arie*, Tetsuya Ogata, Jun Tani, Shigeki Sugano

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

13 Citations (Scopus)


Reinforcement learning is the scheme for unsupervised learning in which robots are expected to acquire behavior skills through self-explorations based on reward signals. There are some difficulties, however, in applying conventional reinforcement learning algorithms to motion control tasks of a robot because most algorithms are concerned with discrete state space and based on the assumption of complete observability of the state. Real-world environments often have partial observablility; therefore, robots have to estimate the unobservable hidden states. This paper proposes a method to solve these two problems by combining the reinforcement learning algorithm and a learning algorithm for a continuous time recurrent neural network (CTRNN). The CTRNN can learn spatio-temporal structures in a continuous time and space domain, and can preserve the contextual flow by a self-organizing appropriate internal memory structure. This enables the robot to deal with the hidden state problem. We carried out an experiment on the pendulum swing-up task without rotational speed information. As a result, this task is accomplished in several hundred trials using the proposed algorithm. In addition, it is shown that the information about the rotational speed of the pendulum, which is considered as a hidden state, is estimated and encoded on the activation of a context neuron.

Original languageEnglish
Pages (from-to)1215-1229
Number of pages15
JournalAdvanced Robotics
Issue number10
Publication statusPublished - 2007 Oct 1


  • Actor-critic method
  • Pendulum swing-up
  • Perceptual aliasing problem
  • Recurrent neural network
  • Reinforcement learning

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Human-Computer Interaction
  • Hardware and Architecture
  • Computer Science Applications


Dive into the research topics of 'Reinforcement learning of a continuous motor sequence with hidden states'. Together they form a unique fingerprint.

Cite this