📷SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models

Abstract

Introduction

SimpleVQA

The SimpleVQA benchmark consists of 2,025 samples spanning 9 core tasks and 9 primary domains, with each question-image pair categorized into relevant subcategories, enabling a comprehensive evaluation of MLLMs across diverse knowledge areas. The dataset 9 tasks, including covers Logic & Science (LS), Object Identification Recognition (OIR), Time & Event (TE), Person & Emotion (PE), Location & Building (LB), Text Processing (TP), Quantity & Position Relationship (QPR), Art & Culture (AC), and Object Attributes Recognition (OAR). To ensure broad topic coverage, SampleVQA is structured around 9 key domains: Literature, education & sports (LES), Euro-American History & Culture (EHC), Contemporary Society (CS), Engineering, Technology & Application (ETA), Film, Television & Media (FTM), Natural Science (NS), Art (AR), Chinese History & Culture (CHC), and Life (LI).

As shown in Table 1, SampleVQA differs from existing MLLM benchmarks by focusing on factual knowledge boundaries instead of general vision language understanding. Politically sensitive and ideological content is excluded to maintain neutrality and avoid controversy. Designed for efficiency, the dataset features concise questions and standardized answers, reducing complexity in model evaluation. All samples follow a short-answer Q&A format, enabling simple and objective assessment through direct answer matching. These refinements ensure SampleVQA serves as a robust benchmark for evaluating MLLMs’ factual reasoning abilities.

SampleVQA adheres to strict criteria ensuring objectivity, temporal stability, and verifiability in its questions, images, and answers. The following guidelines define these standards. Question Guidelines. Clear and Unique Answers: Questions must have a single, undisputed answer. They should precisely define scope (e.g., "Which city?" instead of "Which location?") and specify time references (e.g., "Which year?" rather than "When?"). Evidence-Based: Each question must be supported by verifiable sources. Manually annotated questions include reference links, while automatically generated ones undergo independent validation by two AI trainers. Challenging for MLLMs: Questions are tested on GPT-4o, GPT-4o-mini, doubao-vision-pro, and ERNIE-VL. Only those that at least one model answers incorrectly are retained; others are revised. Answerable by August 2024: All questions must be answerable based on knowledge available before September 1, 2024, ensuring a fair evaluation across models with similar knowledge cutoffs. Visual Guidelines. No Direct Textual Clues: Images must not contain text revealing the answer. Authenticity: Only real, unaltered images are allowed to prevent factual distortion. Supports Question Reasoning: Each image must provide sufficient context for answering. Manually labeled samples undergo multi-annotator verification. Fixed Before August 2024: Image content must be valid and confirmable before August 2024. Answer Guidelines. Temporal Stability: Answers must remain unchanged and unaffected by new information. Time-sensitive topics (e.g.,sports, media) should specify a timeframe rather than a general answer that may change. Sufficiently Challenging: Answers are tested against four high precision MLLMs. If all models respond correctly, the question is revised to increase difficulty. Fully Objective and Evaluable: Answers must be precise, verifiable, and free from subjective interpretation.Unambiguous: Each answer must have a single, clear meaning to prevent misinterpretation.

As shown in Figure 2, the construction of SimpleVQA follows a structured five-step process: Step 1: Seed Example Collection. SimpleVQA’s seed examples are sourced from two primary channels. First, we filter images and Q&A pairs from publicly available VQA datasets that align with factual knowledge criteria. We select MMVet (English), MME (English), Dynamath (English), MMbench_CN (Chinese), and CCBench (Chinese) due to their recent construction (post-2023) and their relevance to real-world applications. Second, we collect images and relevant factual knowledge from search engines (e.g., Google, Baidu, Wikipedia), with expert annotators generating corresponding questions and answers. These data focus on enti- ties and events across multiple domains, ensuring answers are objective, fact-based, and centered on entity recognition or attribute extraction. Step 2: Data Enhancement and QA Pair Generation. Once sufficient seed examples are gathered, we employ GPT-4o (OpenAI, 2023) to refine the data and generate Q&A pairs for factual categories. For multiple-choice questions (MCQs) from sources like MMbench_CN and CCBench, to ensure answer uniqueness, we use LLMs to rephrase the original question and introduce qualifiers that precisely align with the correct response. For MME, we extract the answer entity and rewrite the question based on its attributes, ensuring a one-to-one correspondence. Datasets like MMVet, Dynamath, and CCBench, which contain discrepancies from factual Q&A formats (e.g., incorrect answer options, image descriptions, or MCQ distractors), are processed using GPT-4o to align the content with factual reasoning. These refinements produce the initial version of SimpleVQA. Step 3: LLM-Based Quality Verification. Once sufficient seed examples are gathered, we employ GPT-4o (OpenAI, 2023) to refine the data and generate Q&A pairs for factual categories. For multiple-choice questions (MCQs) from sources like MMbench_CN and CCBench, to ensure answer uniqueness, we use LLMs to rephrase the original question and introduce qualifiers that precisely align with the correct response. For MME, we extract the answer entity and rewrite the question based on its attributes, ensuring a one-to-one correspondence. Datasets like MMVet, Dynamath, and CCBench, which contain discrepancies from factual Q&A formats (e.g., incorrect answer options, image descriptions, or MCQ distractors), are processed using GPT-4o to align the content with factual reasoning. These refinements produce the initial version of SimpleVQA. Step 4: Difficulty Screening. To maximize the dataset’s utility in model evaluation, we filter out overly simple Q&A pairs. We assess responses from four mainstream MLLMs (GPT-4o, GPT-4o-mini, Doubao-vision-pro, and ERNIE-VL). Any question correctly answered by all four models is deemed too simple and excluded from the dataset, thereby maintaining a challenging benchmark. Step 5: Extracting Atomic Facts. To analyze visual comprehension and language alignment in MLLMs more precisely, we generate atomic questions from each SimpleVQA entry. An atomic fact represents the most fundamental, indivisible attribute or characteristic of an object. For instance, given the question "In what year was the person in the image born?", the corresponding atomic question is "Who is the person in the image?". MLLMs generate candidate answers, which are then reviewed and refined by professional annotators to ensure accuracy.

To ensure dataset quality, we implement a rigorous manual validation process following automated data collection. All the collaborators in this paper participated in the necessary data annotation, and we also selected three domain experts from the collaborators. Each question is independently reviewed by two expert annotators to verify factual accuracy. If either annotator finds a question unsuitable, it is discarded. Annotators fact-check answers using authoritative sources such as Wikipedia and Baidu Encyclopedia, providing at least two sup- porting URLs. If their answers differ, a third expert conducts a final review to ensure consistency and correctness. Only Q&A pairs that fully align with both human evaluations and LLM-generated responses are retained. A difficulty assessment further refines the dataset. We begin with 8,360 Q&A pairs, filtering out 22% of image-based samples that lack challenge or fail to meet predefined criteria. 1,108 pairs are removed through multi-model testing to ensure that questions pose a meaningful challenge to MLLMs. To maintain category balance, we carefully select 200 high-difficulty mathematical Q&As from 5,000 Dynamath samples, avoid- ing an overrepresentation of simpler factual questions. Through multiple validation rounds, we re- tain 2,025 high-precision Q&A pairs, accounting for 24% of the original dataset. This process ensures factual integrity, topic diversity, and appropriate difficulty levels, making SampleVQA a robust benchmark for evaluating MLLMs’ reasoning and knowledge boundaries.

As shown in Table 1, our SimpleVQA benchmark consists of 2,025 samples across 9 major tasks, 9 major domains, and 244 image types. Examples of each category can be found in Figure 2. This design facilitates a comprehensive assessment of MLLMs across different domains. Regarding the distribution of topics and image types in SimpleVQA, nine main topics are defined and subcategories are assigned based on each topic. In Table 1, we also compare SimpleVQA with several mainstream MLLMs’ evaluation benchmarks, which suggests that SimpleVQA is the first MLLMs’ benchmark that focuses on the evaluation of knowledge boundaries in factual categories. We excluded ideological and politically relevant data from the dataset to prevent social controversies and negative impacts. In addition, we implemented several optimizations to improve the efficiency of the evaluation. The dataset features concise questions and standardized answers, minimizing the input and output markers required for GPT assessment. In addition, all examples are in short-answer question-and-answer (QA) format, and they can be assessed by simple matching.

Experiments

We maintain a consistent prompt format across all experiments. The temperature and sampling parameters adhere to each LLM’s official configuration or default settings and GPT-4o serves as the primary model for evaluation and data construction.

We evaluate 18 models in total, comprising 8 closed-source and 10 open-source models, providing a diverse evaluation of model capabilities across different architectures and training paradigms. The closed-source models include GPT-4o, GPT-4o-mini, Doubao-pro-128k, Doubao-pro-32k, Gemini-2.0-flash, Claude-3.5-Sonnet, Qwen-Max, ERNIE-VL. The open-source models cover a wide range of frameworks, including InternLM2.5,Qwen2.5, Qwen2, Janus-pro-7B.

The evaluation of the SampleVQA benchmark employs a set of rigorous metrics designed to assess the accuracy, reliability, and consistency of the model’s predictions. These metrics include: (1) Correct (CO) evaluates whether the predicted answer matches the reference answer exactly without any contradiction. (2) Not Attempted (NA) identifies cases where the model does not attempt to answer, ensuring no contradictions are present. (3)Incorrect (IN) flags instances where the predicted answer contradicts the reference answer, even if resolved. (4) Correct Given Attempted (CGA) measures the proportion of correct answers among those attempted by the model, reflecting its performance when engaged. (5) F-score computes the harmonic mean between "Correct" and "Correct Given Attempted," providing a balanced evaluation that combines accuracy and attempt success.

Results on Different Tasks.Table 4 presents the performance of various closed-source and open-source vision-language models on a benchmark, highlighting their F-scores across different tasks in Chinese and English. Among the closed-source models, Gemini-2.0-flash and Doubao-vision-pro-128k show strong performance, particularly in tasks like PE and TP. In contrast, models like Claude-3.5-Sonnet and Qwen-Max exhibit moderate perfor- mance. Open-source models, such as InternVL2.5-78B-MPO and Qwen2.5-VL-72B-Instruct, demon-strate competitive results, though slightly lower than the top closed-source models, with F-scores mostly in the 40s and 50s. Notably, models like InternVL2-Llama3-76B and DeepSeek-VL2-27B show weaker performance, with F-scores in the 30s, indicating a significant gap between the highest and lowest-performing models. Overall, closed-source models tend to outperform open-source ones, particularly in specialized tasks, though some open-source models remain competitive in specific categories.

Further Analysis

Based on SimpleVQA, we conducted a comprehensive evaluation of the mainstream MLLMs, exposing serious factual problems in the model. We also conducted an in-depth causal analysis of the existing factual problems from the perspective of MLLMs’ image understanding and text generation capabilities, providing a forward-looking direction for the optimization of subsequent models. First, we identified the three most robust MLLMs through evaluation. For each Visual Question Answering (VQA) task, if the model’s response is incorrect, we simplify the question into an atomic problem related to content recognition using a prompt. This atomic problem corresponds to an atomic fact. When provided, it transforms the original question into a purely factual text-based query. If the model still cannot answer the atomic query correctly, we attribute the failure to the MLLM’s insufficient understanding of the image. Next, since some of the original questions are atomic questions, we collect cases where the atomic questions are different from the original questions and use them to extract a test set, called the Complex Fact Question (CFQ) set, to verify whether the performance of the model improves when given atomic facts. In another experiment, we incorporate the answer to the atomic question as a hint into the CFQ query and reassess the model’s response. If the model still provides an incorrect answer, we attribute the failure to a lack of background knowledge. The table below shows the results of our CFQ experiment. In this paper, we refer to the performance of Top 5 SOTA models in the main experiment and select difficult CFQ examples from all samples totaling 559, which we use as the CFQ dataset to test the picture comprehension ability and knowledge internalization ability of GPT-4o, Qwen2.5-VL-72B-Instruct and InternVL2.5-78B-MPO. The experimental results are shown in Table 5, there is a large mismatch between the model’s picture literacy and knowledge internalization abilities, and the model’s knowledge storage ability is a little better compared to picture comprehension, but it still has a lot of room for improvement.

Related Works.

Recent vision-language benchmarks have been developed to assess models’ capabilities in integrating visual and textual information across various tasks (Wu et al.,2024a,b; Zhang et al., 2024), including OCR, spatial awareness, multimodal information retrieval, and reasoning skills. For example, MMBench (Liu et al., 2023) employs multiple-choice tasks in both Chinese and English, covering a wide range of domains. MMMU (Yue et al., 2024) focuses on complex vision-language tasks, particularly those requiring advanced multimodal reasoning. MMStar (Chen et al., 2024) utilizes multi-task evaluations to test models’ ability to fuse different modalities.

Factuality refers to their ability to generate content that follow facts, including commonsense, world knowledge, and domain-specific information. This capability is typically assessed by comparing model outputs to authoritative sources such as Wikipedia or academic textbooks. Recently, Various benchmarks have been developed to evaluate factuality in LLMs (Zhong et al., 2023; Huang et al., 2023; Li et al., 2023a;Srivastava et al., 2023; Yang et al., 2018; Lin et al., 2022; Yang et al., 2024a,b). For example, MMLU (Hendrycks et al., 2021) assesses multitask accuracy across 57 diverse tasks. HaluEval (Li et al., 2023b) explores the propensity of LLMs to produce hallucinations or false information. SimpleQA (Wei et al., 2024) and Chinese SimpleQA (He et al., 2024a) have been proposed to measure the short-form factuality in LLMs.

Conclusion.

In this paper, we introduce the first Chinese-English visual question-answering benchmark, SimpleVQA, designed to evaluate the fact-based quizzing capabilities of existing Multimodal Large Language Models (MLLMs). This benchmark encompasses seven key features: Chinese-English bilingual support, multi-task and multi-scene adapt ability, high quality, challenging content, static design, and ease of evaluation. Utilizing SimpleVQA, we conducted a comprehensive assessment of 18 MLLMs, analyzing their performance in fact-based quizzes to highlight the advantages and necessity of our benchmark. Future work will focus on enhancing the factual quizzing capabilities of MLLMs and expanding SimpleVQA to include multi-language support, multi-intelligentsia, and specialized domains such as coding, e-commerce, and encyclopedias for role-playing scenarios. Building on prior research in neural network calibration, we developed a novel methodology to calibrate the visual comprehension and visual-linguistic information alignment abilities of MLLMs, identifying error sources by testing key atomic questions derived from original factual queries. A notable limitation of SimpleVQA is its focus on measuring factuality through short, verifiable answers, leaving the correlation between providing concise factual responses and generating lengthy, fact-rich answers as an open research question. We hope that the SimpleVQA will serve as a valuable tool for assessing factuality and inspire the development of more trustworthy and reliable MLLMs.