Analyzing Gaze and Hand Movement Patterns in Leader-Follower Interactions During a Time-Continuous Cooperative Manipulation Task

Humans often interact with each other during daily life and many times one finds that one person (at least for some time) takes the lead while the other follows. Different from usual experimental settings in a lab, this happens without external influences and in a time-continuous manner. While knowledge that gazes onto the object to be manipulated precede a then-following manipulation action emerged long ago, little is known how gaze- and hand movement patterns develop in complex interactive scenarios where one person needs to consider the other’s action when planning their own.To address this question in this study we investigate predictive, planning-related behavior during a two-player (Leader and Follower) table-top game. In this game – called “do-undo” – the Leader has to perform pick and place actions, following simple rules, to change the configuration of objects on the table. The Follower then has to use different objects to re-create the object-configuration back to the one that had existed before the Leader had acted. We track eyes, hand-movements and individual objects on the table and determine relations between eyes and hands of both players, with emphasis on the differences in behavior of Leader and Follower. Data is recorded using a setup where gaze tracking is combined with multicamera tracking of motion of the subjects and of the configuration on the table. The game proceeds without any external trigger signals and we address the problem of finding accurate time-points for phasing of the cooperative manipulation by using touch sensors on the hands of the players to record touching and untouching events by which we can unequivocally define the different temporal (inter-)action intervals for analysis.While gaze clearly precedes manipulation in all cases, we find clear differences in eye-movement patterns of Leader and Follower. The Leader makes more and earlier gazes on the objects to be manipulated. In a substantial proportion of trials, the Leader is predicting by his gaze not only their own, but also the required potential Follower-actions before their own manipulation. Also, the gaze data shows that the Leader takes more interest in re-checking the outcome of the do-undo manipulation pair. Indications of trying to memorize a configuration are also found in the gaze pattern of both. In addition, we often find sequences of alternating gazes towards objects and location where to put, which are much more expressed by the Leader than by the Follower. Intriguingly some patterns show that players pre-plan over longer periods and not just for their next action.This clearly indicates different decision-making and planning characteristics for Leader as compared to Follower, which happen not only for own plan but also for potential action plans by the other shining a light on the complex cognitive processes that exist in daily human-human interaction.