TOP > Research > Comprehensive List of Researchers "Information Knowledge" > Department of Media Science > Real World Data Circulation Program > HIRAYAMA, Takatsugu

Comprehensive List of Researchers "Information Knowledge"

Department of Media Science

HIRAYAMA, Takatsugu
Real World Data Circulation Program
Designated Associate Professor
Dr. of Engineering
Research Field
Image Recognition / HCI / Multimodal Data Analysis
HIRAYAMA, Takatsugu

Current Research

Design and Mining of Gaze-Based Interaction Making Cognitive State Explicit
Design and Mining of Gaze-Based Interaction Making Cognitive State Explicit

[The eyes are the windows of the mind? The eyes say more than the mouth?]
Achieving harmony between human beings and the information environment is one of the major issues in the field of computer science. Recognizing the cognitive state of people such as interest or intent by observation of their behavior is very important for achieving the harmonization. As you know, the eyes are the window of the mind. Eye movement is a clue for the recognition. So many researchers in various field such as psychophysics and engineering have analyzed the eye movement from the late 19th century.

Since the eyes are an interface to not only input vast amounts of information but also output the cognitive state, the eye movement should have a valuable role in designing a visual information-mediated human machine interaction. The machine can measure the eye movement as temporally continuous data, while the user unconsciously leaks information reflecting the cognitive state on the measurement. They will experience a natural interaction like casual communication between people. “The eyes say what the mouth doesn’t say at every moment” might be correct though people in the past said “the eyes say more than the mouth.” It is, however, difficult to recognize the cognitive state of human by only passive observation.

[Design of gaze-based interaction for probing responses reflecting cognitive state]
How do we recognize other’s cognitive state in everyday life? We adopt proactive behaviors so that we probe the cognitive state of conversational partners. My past study, for example, showed that observing their reaction timing to the probe made the recognition easy. Based on the findings, Mind Probing has been proposed as a gaze-based interaction strategy, which a machine gives the user a proactive behavior and then recognizes a cognitive state by observing his/her reaction.

[Mining of interaction among visual environment, gaze behavior, and cognitive state]
Mind Probing has the potential to be applied to not only human-to-human and human-to-machine interaction but also human-to-environment interaction. People shift gaze in a reflection of cognitive state and in response to visual stimuli present in our environment. However, eye movement and gaze behavior do not have a simple relationship to cognition because they represent low dimensional information, whereas the cognition seems to be a high dimensional internal state. A key to understanding the relationship is focusing on visual environment dynamics. Many researchers have analyzed visual attention and gaze behavior in laboratory environment with controllable visual stimuli for years. Contrary to the well-established approach, my study attempt is quantitatively describing the visual environment dynamics uncontrollable in everyday space. It opens the way for not only hypothesis testing that analyzes eye movement and gaze behavior for a specific scene but also data mining that extracts characteristic findings form big data. Through this analytic paradigm shift, my study aims to model relationships among the visual environment, the gaze behavior, and the cognitive state.

[Other research themes]
・Modeling timing structure of multimodal data
(Application: Facial image recognition to distinguish intentional smiles from spontaneous smiles, Detection of cognitively distracted driving)
・Extracting distinctive patterns from big data
(Application: Representation of individual facial features, Analysis of the knack and habit of gaze behavior, Retrieval of atypical recipes)


  • Takatsugu Hirayama received the M.E. and D.E. degrees in Engineering Science from Osaka University, Japan, in 2002 and 2005, respectively. He was a research assistant professor in the Graduate School of Informatics, Kyoto University from 2005 to 2011, and an assistant professor in the Graduate School of Information Science, Nagoya University from 2012 to 2013. He is currently a designated associate professor in the Graduate Program for Real-World Data Circulation, Nagoya University.

Academic Societies

  • ACM
  • IPSJ
  • HIS


  1. Takatsugu Hirayama, Kenji Mase, Kazuya Takeda, "Analysis of Temporal Relationships between Eye Gaze and Peripheral Vehicle Behavior for Detecting Driver Distraction,International Journal of Vehicular Technology, Vol.2013, Article ID 285927, 2013
  2. Akisato Kimura, Ryo Yonetani, Takatsugu Hirayama, "Computational Models of Human Visual Attention and Their Implementations: A Survey,IEICE Transactions on Information and Systems, Vol.E96-D, No.3, pp.562-578, 2013
  3. Takatsugu Hirayama, Jean-Baptiste Dodane, Hiroaki Kawashima, Takashi Matsuyama, "Estimates of User Interest Using Timing Structures between Proactive Content-Display Updates and Eye Movements," IEICE Transactions on Information and Systems, Vol.E-93D, No.6, pp.1470-1478, 2010
  4. Masahiro Nishiyama, Hiroaki Kawashima, Takatsugu Hirayama, Takashi Matsuyama, "Facial Expression Representation based on Timing Structures in Faces," Proceedings of IEEE International Workshop on Analysis and Modeling of Faces and Gestures (LNCS 3723), pp. 140-154, 2005
  5. Tetsuro Onishi, Takatsugu Hirayama, Takashi Matsuyama, "What does the Face-Turning Action Imply in Consensus Building Dialogue?," Proceedings of International Workshop on Machine Learning and Multimodal Interaction (LNCS 5237), pp. 26-37, 2008