IPCAI 2026 · Diagnostic Benchmark
SurgCheck — a bias-controlled benchmark that exposes how much surgical VQA performance comes from linguistic shortcuts rather than genuine visual understanding.
Vision–language models (VLMs) show promising performance in surgical visual question answering (VQA). However, existing surgical VQA datasets often contain linguistic shortcuts, where question phrasing implicitly constrains the answer space. In safety-critical surgical settings, it remains unclear whether reported performance reflects visual understanding or reliance on such shortcuts.
We introduce SurgCheck, a diagnostic benchmark for quantifying linguistic shortcut reliance. Each surgical frame is paired with an original question containing entity names and a less-biased counterpart that removes these names while preserving identical visual content and ground-truth answers. The resulting performance gap provides a diagnostic signal of shortcut reliance. To keep less-biased questions well-defined, four grounding cues are incorporated: bounding box, arrow, spatial position, and periphrasis.
Across five VLMs we observe consistent degradation on less-biased questions despite identical visual inputs. A text-only ablation reveals minimal drops for action and target prediction — indicating these are largely driven by linguistic shortcuts rather than visual reasoning. SurgCheck shows that strong benchmark performance does not necessarily imply faithful visual understanding.
Built on scene-graph annotations from Endoscapes-SG201, each frame produces an original and a less-biased question that differ only in linguistic bias. A VLM answers, and an LLM-as-a-Judge (GPT-5-mini) scores semantic correctness.
Removing an entity name makes a question ambiguous. Grounding cues re-anchor it — ranging from explicit visual pointers to purely textual descriptions.
Pick a real surgical frame, a grounding cue, and a task. The image and answer stay identical — only the wording changes. This is exactly the manipulation SurgCheck measures.
After fine-tuning, every model drops on less-biased questions despite identical images and answers; the zero-shot setting shows the same trend, with PaliGemma2 as a marginal exception. Click a column header to sort.
▼ red = drop on less-biased · ▲ green = gain.
Action & Target rely most on entity names; Anatomy-hop & Triplet (spatial reasoning) stay robust.
Explicit visual cues (arrow, box) drop the most; periphrasis (contextual phrasing) is the most robust.
Remove the image entirely: Action & Target barely drop — proof the model was reading text, not pixels. Anatomy-hop & Triplet collapse, showing they truly need vision.
Actual fine-tuned LLaVA-OV-1.5-8B outputs. For action/target the model stays correct even with the image removed (a shortcut); spatial tasks fail without vision.
Real fine-tuned LLaVA-OV-1.5-8B predictions. The image and answer are identical — only the wording changes, yet the prediction flips. Filter by task and step through examples.
Surgical VQA is inherently prone to strong linguistic bias. Question phrasing — instrument names, hand references, and conventional instrument–action–target patterns — is highly correlated with the answer, so models can, and do, exploit textual shortcuts instead of grounding their predictions in the image.
SurgCheck makes this reliance measurable. The paired-question gaps and the text-only ablation show it is systematic: performance stays high on original questions even when the image is removed, especially for action- and target-centric tasks. In other words, strong benchmark numbers do not guarantee faithful visual understanding.
This has a direct implication for how surgical VQA models should be built. Training and evaluation should:
We hope SurgCheck supports more reliable evaluation of multimodal models and encourages methods that genuinely ground predictions in visual evidence.
SurgCheck ships as paired JSONL annotation files. The surgical frames themselves are not redistributed — download them from Endoscapes-SG201 and combine them with the annotations here.
Both options give you the same files. Structure:
data/
├── full/
│ ├── train_original.jsonl (19,604 rows)
│ ├── train_less_biased.jsonl (19,604 rows)
│ ├── test_original.jsonl ( 3,860 rows)
│ └── test_less_biased.jsonl ( 3,860 rows)
└── balanced_4cue_test/ # Sec. 3.5 cross-cue-aligned subset
├── box_{original,less_biased}.jsonl (1,898 rows each)
├── arrow_{original,less_biased}.jsonl (1,898 rows each)
├── position_{original,less_biased}.jsonl (1,898 rows each)
└── periphrasis_{original,less_biased}.jsonl (1,898 rows each)
Each *_original.jsonl and *_less_biased.jsonl file is paired line-by-line: line i shares the same image, task, cue, and ground-truth answer — only the question wording differs. Inside balanced_4cue_test/, the four cue files are additionally cross-cue aligned (line i of box / arrow / position / periphrasis points to the same (frame, category, slot)).
Base surgical frames come from Endoscapes-SG201 (Murali et al., 2023), derived from Cholec80. Please respect the upstream license.
Each annotation row carries an ori_img_tag (the base frame filename in Endoscapes, e.g. 164_2700.jpg) and an image field that may either point to the same base frame or to a cue-augmented variant with a red bounding box or arrow drawn on it (e.g. 164_2700_bbox_lt.jpg, 164_2700_arrow_cystic_duct.jpg). Cue-augmented variants are rendered from Endoscapes' scene-graph object bounding boxes — regenerate them by drawing the red box or arrow at the annotated location on the base frame.
{
"image": "164_2700_arrow_lt.jpg",
"conversations": [
{"from": "human", "value": "<image>\nYou are an assistant with surgical expertise. Given a surgical scene, answer the question in 1-5 words. Which tool, operated by the operator's left hand, is indicated by the red arrow?"},
{"from": "gpt", "value": "grasper"}
],
"ori_img_tag": "164_2700.jpg",
"main_tag": "Visual_QA",
"sub_sub_tag": "Instrument",
"sub_tag": "Arrow_Cue"
}
main_tag — Visual_QA or Reasoning_QAsub_sub_tag — task category: Instrument_Count, anatomy, Instrument, Action, Target, Triplet, anatomy_hop, CVSsub_tag — grounding cue: No_Cue, Box_Cue, Arrow_Cue, Position_Cue, Word_CueSee data/README.md for schema details, dataset quirks, and license notes.
@inproceedings{shin2026surgcheck,
title = {SurgCheck: Do Vision--Language Models Really Look at Images in Surgical VQA?},
author = {Shin, Jongmin and Kim, Ka Young and Cho, Eunki and Kim, Seong Tae and Oh, Namkee},
booktitle = {International Conference on Information Processing in Computer-Assisted Interventions (IPCAI)},
year = {2026}
}