Recognizing abruptly changing facial expressions from time-sequential face images

This paper proposes a method that can spot and recognize each facial expression from time-sequential images that contain multiple facial expressions that could abruptly change from one expression to another expression. Previously, the authors have proposed an HMM (Hidden Markov Models) based method for recognizing a spotted facial expression. In this paper, to HMM, we add states corresponding to the simultaneous motion of two different facial expressions: i.e. a muscle relaxation for one expression and a muscle contraction for another expression. Then, the added states are each linked from the HMM apex state of one expression and are linked to that of another expression. Experimental results showed that for most pairs of expressions the change in expression can be recognized accurately. In addition, recognition rate for very fast change of expressions improved significantly. The proposed method was applied to regenerate facial expressions on a synthesized character to show the method's effectiveness in obtaining facial motion information.