Can AI read conflicting emotional signals?
23 Oct 2026
When people communicate, words may not tell the whole story. A person may say “I’m fine” while speaking in a low, unhappy voice and looking tired. Humans can instinctively interpret these conflicting signals, but artificial intelligence (AI) still struggles to do so.
To address this challenge, a research team from the Academy of Artificial Intelligence and Advanced Technology (AIAT) at Xi’an Jiaotong-Liverpool University (XJTLU) has developed a Semantic Purification and Time-Causal Stabilization with Energy-Inspired Arbitration Framework (SCEF), a new multimodal sentiment analysis model. This model enables AI to assess the reliability of different emotional signals before combining them. By integrating language, tone, and facial expressions, the model can make more accurate judgements regardless of whether these signals agree or contradict one another.
The team’s paper was published online on 21 August 2026 in IEEE Transactions on Affective Computing, a leading journal in this field. The paper’s first author is Minghui Zhu, a PhD candidate at AIAT, and the corresponding author is Dr Yushan Pan.
The research was conducted in collaboration with Zhengzhou University of Light Industry, China; the Norwegian University of Science and Technology, Norway; and Zhejiang University, China.
The SCEF multimodal sentiment analysis model
Assessing conflicting signals
Text, audio and video can all contain noise and may convey different or even contradictory emotional cues. Multimodal sentiment analysis involves more than combining the three sources.
“If a model simply combines all sources without determining which one is more reliable, this may lead to misjudgements,” Dr Pan explains.
SCEF addresses this problem by evaluating the emotional information carried by different modalities and analysing how they interact. Rather than giving equal weight to every signal, it dynamically adjusts their influence according to the context.
For example, when verbal and non-verbal cues convey conflicting emotions, SCEF can recognise the discrepancies and assess the relative reliability of different signals before reaching a final judgement.
AI interpreting conflicting emotional cues from different sources
Testing across datasets
The team evaluated SCEF on four publicly available multimodal sentiment analysis datasets: CMU-MOSI and CMU-MOSEI for English, and CH-SIMS and CH-SIMS-V2 for Chinese. These datasets combine linguistic, vocal and facial information, allowing researchers to assess how effectively AI can integrate different sources of emotional information.
SCEF achieved competitive performance across all four datasets. On the widely used CMU-MOSI dataset, it reached 90.85% accuracy in binary positive/negative sentiment classification, compared with 89.30% for TEDT, the previous best-performing model. This represents an improvement of 1.55 percentage points in accuracy.
Inspired by statistical physics
A key innovation of SCEF is its use of the concept of “energy” from statistical physics to measure how closely different modalities align. This enables the model to determine whether information from different sources is suitable for fusion.
When signals are consistent, they can be integrated more readily. When they conflict, the model adjusts their relative influence according to their interactions, providing a more informed basis for sentiment prediction.
The researchers believe this approach could contribute to more natural and reliable human-AI interaction. Potential applications include intelligent customer service, digital humans, social robots, elderly care, and digital health. In these contexts, understanding both verbal and non-verbal cues could help AI respond more appropriately to people’s changing emotional states.
“AI is evolving from merely recognising text and images towards understanding humans,” says Dr Pan. “We hope AI can not only identify emotional cues from individual sources, but also understand the relationships and conflicts between different signals.”
By Huatian Jin
Translated by Xueqi Wang
Edited by Katharina Zhu and Xinmin Han
23 Oct 2026