INFOTEC-NLP at SemEval-2025 Task 11: A Case Study on Transformer-Based Models and Bag of Words

AbstractLeveraging transformer-based models as feature extractors, we introduce a hybrid architecture that integrates a bidirectional LSTM network with a multi-head attention mechanism to address the challenges of multilingual emotion detection in text. While pre-trained transformers provide robust contextual embeddings, they often struggle with capturing long-range dependencies and handling class imbalances, particularly in low-resource languages. To mitigate these issues, our approach combines sequential modeling and attention mechanisms, allowing the model to refine representations by emphasizing key emotional cues in text.