Information-Theoretic Modality Reliability Optimization For Robust Multimodal Transformer Models

Abstract

Abstract
Multimodal transformer models integrate heterogeneous data sources such as text, vision, and audio to enhance reasoning and decision-making. However, most existing multimodal architectures implicitly assume equal reliability across modalities, even though real-world inputs are frequently noisy, incomplete, or misleading. This assumption leads to modality dominance, error propagation, and reduced robustness. This paper proposes a novel Information-Theoretic Modality Reliability Optimization (IMRO) framework that explicitly quantifies modality trustworthiness using entropy, mutual information, and uncertainty estimation. A reliability-aware attention mechanism is introduced in which modality contributions are dynamically weighted based on their estimated information content. The framework is optimized through a reliability-regularized objective function with theoretical stability guarantees. Extensive mathematical formulations, algorithmic design, and implementation details are presented. The proposed approach improves robustness, interpretability, and training stability, making it suitable for real-world multimodal transformer deployments.