Understanding Annotator Perception: Modeling Psychological Inference from First- and Third-Person Annotations (Student Abstract)

Abstract Large language models (LLMs) are trained on vast amounts of publicly available text. However, the current training frameworks take for granted that these annotations are accurate reflections of the authors’ true intents. This study questions that assumption by examining the gaps between writers’ actual psychological states and the inferences made by third-party annotators. We explore how readers interpret psychological cues in text and demonstrate that third-person annotations often fail to align with first-person realities. By integrating both first- and third-person annotations, we develop computational models that reveal significant biases in how psychological states are perceived and the downstream effects these perceptions have on reader behavior. Our findings challenge the foundational assumptions of LLM training, suggesting that the reliance on potentially flawed third-person annotations could impact model accuracy and real-world applications.