Search papers, labs, and topics across Lattice.
This paper presents HABIBTAZ, a system designed to disentangle formal logic from content in Large Language Models (LLMs) for the SemEval-2026 Task 11, addressing the challenge of content bias in formal reasoning across 12 languages. By employing mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset and utilizing a multi-objective loss function, the system achieved top rankings with perfect scores in several subtasks, demonstrating its effectiveness in minimizing bias and maximizing accuracy. The results indicate that the proposed approach significantly enhances the formal reasoning capabilities of LLMs while maintaining robustness against content effects.
Achieving perfect accuracy and zero bias in formal reasoning tasks reveals a groundbreaking method to enhance LLMs' logical capabilities without succumbing to content biases.
While Large Language Models (LLMs) excel in many general NLP tasks, their formal reasoning capabilities are often compromised by content effects, demonstrating a measurable bias towards real-world plausibility. In this paper, we present our system for SemEval-2026 Task 11, which evaluates the ability of models to disentangle formal logic from content across 12 languages with and without distractor premises. We address this challenge using mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset of syllogistic schemes to avoid the semantic noise of LLM-augmented data. To explicitly decouple plausibility from logical structure, our training pipeline employs a multi-objective loss function combining Adaptive Group Distributionally Robust Optimization (DRO), a scheduled differentiable bias penalty, and KL-Divergence consistency regularization. Our system achieved #1 ranks and perfect Ranking Scores (100.0) with 0.00% bias and 100.0% accuracy on Subtask 1 (English), Subtask 2 (Noisy English), and Subtask 3 (Multilingual). On the highly complex Subtask 4 (Noisy Multilingual), the system achieved the 6th rank with 89.06% Accuracy and F1-score, alongside a limited 2.89% Bias and a 37.78 Ranking Score. Our dataset generation engine and codebase are publicly available to facilitate future work on robust logical reasoning.
