📢 [2025-02-02] VAP is now open-source! Explore our Visual Adversarial Perturbation (VAP) method to mitigate object hallucinations in LVMs. Check out the repo and get started! 🔥

📢 [2025-02-03] Our paper “Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs” is now available! 🎉

We evaluate eight state-of-the-art Large Vision-Language Models (LVMs) to validate the efficacy of our proposed approach. These models, spanning significant advancements in multimodal AI from September 2023 to December 2024, range from 7.1B to 16.1B parameters.

POPE Benchmark: Evaluation triplets are sampling from the MS-COCO dataset and are formatted in the following JSON structure:

• POPE GitHub Repository
• Download MS-COCO Dataset

Example JSON format:

{
  "id": 0,
  "image": "name", // The image filename from the dataset
  "question": "XXX", // The generated question related to the image
  "gt": "yes/no" // The ground truth answer (binary: "yes" or "no")
}

BEAF Benchmark: This dataset follows the format specified in the BEAF repository.

• BEAF Github Repository
• BEAF Dataset Download

This section provides detailed instructions for setting up the environments required to run various LVMs. We ensure compatibility across different models by structuring the setup efficiently, leveraging shared dependencies where applicable.

• Model: liuhaotian/llava-v1.5-7b
• Environment Setup:

cd env_setting/LLaVA
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip
pip install -e .
pip install ftfy regex tqdm
pip install protobuf transformers_stream_generator matplotlib

• Models:
  • Salesforce/instructblip-vicuna-7b
  • OpenGVLab/InternVL2-8B
  • OpenGVLab/InternVL2-8B-MPO
• Shared Environment Setup:

conda create -n internvl python=3.9 -y
conda activate internvl
pip install lmdeploy==0.5.3
pip install timm ftfy regex tqdm matplotlib
pip install flash-attn==2.3.6 --no-build-isolation

• Model:
  • lmms-lab/llava-onevision-qwen2-7b-ov
  • AIDC-AI/Ovis1.6-Gemma2-9B
• Shared Environment Setup:

cd env_setting/LLaVA-NeXT
conda create -n llava_onevision python=3.10 -y
conda activate llava_onevision
pip install --upgrade pip
pip install -e ".[train]"
pip install ftfy regex tqdm matplotlib
pip install transformers==4.47.0
pip install flash-attn==2.5.2

• Model: Qwen/Qwen2-VL-7B-Instruct
• Environment Setup:

conda create -n qwen python==3.11 -y
conda activate qwen
pip install transformers
pip install qwen-vl-utils
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install accelerate==0.26.1
pip install ftfy regex tqdm matplotlib

• Model: deepseek-ai/deepseek-vl2
• Environment Setup:

conda create -n deepseek python==3.10 -y
conda activate deepseek
cd env_setting/DeepSeek-VL2/
pip install -e .
pip install ftfy regex tqdm matplotlib
pip install flash_attn==2.5.8
pip install --force-reinstall --no-deps --pre xformers
pip install transformers==4.38.2

This setup ensures a streamlined development environment while minimizing conflicts across dependencies. For further optimizations, consider leveraging CUDA-based optimizations, distributed inference, and efficient memory management strategies.

1. Visual Adversarial Perturbation (VAP) Execution

To generate visual noise for mitigating hallucinations in LVMs, run the following command:

bash script/VAP.sh

2. Hallucination Evaluation

To assess the model’s performance on hallucination benchmarks, execute:

bash script/evaluate.sh

If you find our work helpful, please consider citing our paper:

@article{zhang2025poison,
  title={Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs},
  author={Zhang, Kejia and Tao, Keda and Tang, Jiasheng and Wang, Huan},
  journal={arXiv preprint arXiv:2501.19164},
  year={2025}
}

Your citation helps support our research and further advances the field of reliable vision-language models. 🚀