A personality-labeled semantic dataset from facial expressions, gaze, and head movement cues

Facial expressions, eye gaze, and head movements play key roles in personality perception and social interactions. However, datasets that capture these cues with personality labels are rare, which limits advancements in computational personality recognition and expressive virtual agents. To address this, we developed a compact, semantically interpretable dataset using temporal Action Units (AUs), gaze, and head movements labeled with Five-Factor personality traits. The dataset is annotated using an iterative strategy that employs the existing “First Impressions” dataset to train a deep-learning model for automated personality labeling. The model, which combines attention with convolutional components, is iteratively refined based on user feedback from small-scale user studies. Results show that combining AUs with gaze and head motion significantly improves personality trait prediction, outperforming state-of-the-art models in accuracy. We also analyze how AUs, gaze, and head parameters contribute to the perception of personality factors. The findings offer insights applicable to the animation of expressive virtual characters and behavioral sciences.