Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

📦 Dataset: Tri-PvP on Hugging Face

Omni-modal LLMs (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two forms of evidence within a modality — perceptual signals (a photo/recording of a dog) and propositional signals (the claim "this is a dog") — so measured modality bias is confounded with evidence-form bias. We introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Across five OLLMs we find robust visual bias, plus a systematic asymmetry: models favor perceptual evidence in vision but propositional evidence in audio. Layer-wise linear probing and contrastive decoding show this bias emerges in early representation layers and is only partially mitigable.

Run Inference

# Gemini 3
python inference_scripts/inference.py --model gemini-3.0-flash --hf-repo ModaSense/animal --output results/

# Qwen2.5-Omni
python inference_scripts/inference.py --model qwen2.5-omni-7b --backend vllm --hf-repo ModaSense/animal --output results/

# Qwen3-Omni
python inference_scripts/inference_Qwen3Omni.py --model_name qwen3-omni-thinking --image_type perceptual --audio_type propositional --output_dir raw_output/

# Gemma 4
python inference_scripts/inference_gemma4.py --model gemma4-e4b-it --dataset ModaSense/animal --split test --output results/

# MiniCPM-o 4.5
python inference_scripts/inference_minicpm.py --model minicpm-o-4_5 --dataset ModaSense/animal --split test --output results/

Run Judge

python evaluation/llm_judge.py --input_file results/{model_name}/results.json

Citation

@inproceedings{tripvp2026,
  title     = {Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts},
  author    = {},
  booktitle = {},
  year      = {2026},
}