Understanding fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human–AI interaction. Despite strong performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially for complex, articulated hand poses.
We introduce HandVQA, a large-scale diagnostic benchmark that evaluates VLMs’ understanding of detailed hand anatomy via visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), HandVQA provides 1.6M+ controlled multiple-choice questions probing spatial relationships between hand joints, including angles, distances, and relative positions.
We evaluate state-of-the-art VLMs (e.g., LLaVA, DeepSeek, Qwen-VL) in both base and LoRA fine-tuned settings. Our results reveal systematic limitations such as hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these reasoning gaps, but also provides a validated path to improvement: the 3D-grounded spatial knowledge learned from HandVQA transfers zero-shot to novel downstream tasks, improving hand gesture recognition (+10.33%) and hand–object interaction recognition (+2.63%).
Despite impressive progress on general vision-language benchmarks, current VLMs still struggle with fine-grained spatial reasoning about articulated objects such as human hands. These failures are often hidden in aggregate benchmark scores but become evident when models must reason about precise finger configurations, inter-finger distances, or spatial ordering.
As shown below, base VLMs frequently hallucinate finger configurations, misjudge inter-finger distances, and confuse spatial ordering (e.g., left/right or front/behind). These errors become especially pronounced under occlusion, viewpoint changes, or complex hand poses.
To systematically diagnose and improve these limitations, we introduce HandVQA, a 3D-geometry-grounded benchmark designed to evaluate fine-grained spatial reasoning about hands through controlled visual question answering.
Are any fingers crossing?
Base VLM:
No ✗
Spatial-aware VLM:
Yes ✓
Key challenge: Detects self-occlusion patterns.
Which pair of fingers are spread widest?
Base VLM:
Index–Middle ✗
Spatial-aware VLM:
Middle–Ring ✓
Key challenge: Tests distance reasoning.
Which fingertip is closest to the palm?
Base VLM:
Index ✗
Spatial-aware VLM:
Ring ✓
Key challenge: Combines distance reasoning from reference points.
Is the index finger crossing the middle finger?
Base VLM:
No ✗
Spatial-aware VLM:
Yes ✓
Key challenge: Requires depth and left/right ordering.
Is the thumb left or right of the index finger?
Base VLM:
Left ✗
Spatial-aware VLM:
Right ✓
Key challenge: Tests X-axis spatial reasoning.
Are the two fingers touching?
Base VLM:
No ✗
Spatial-aware VLM:
Yes ✓
Key challenge: Detect contact between fingers.
Questions are generated deterministically from 3D joint coordinates and rendered into natural-language MCQ options, ensuring explicit geometric grounding.
Hand joint map (MANO-style joints) used to compute pose descriptors.
We compute discrete labels from 3D joints and convert them into controlled MCQ options.
(Aligned cases are excluded in benchmark questions to avoid visually ambiguous labels.)
(Aligned cases are excluded in benchmark questions to avoid visually ambiguous labels.)
(Aligned cases are excluded in benchmark questions to avoid visually ambiguous labels.)
Overview of the HandVQA Pipeline. Our pipeline converts normalized 3D hand joints into interpretable VQA pairs through three deterministic stages. (1) Pose extraction: continuous pose descriptors — joint angles (θ), distances (d), and relative positions (Δx, Δy, Δz) — are computed and discretized into categorical pose descriptors. (2) Text generation: deterministic sentence templates are filled using these descriptors to produce candidate answer options, including both correct and distractor choices. (3) MCQ construction: each hand image is paired with its answer options to form structured multiple-choice VQA samples with a single correct label.
Overview of HandVQA Question Format. The benchmark decomposes hand pose understanding into five spatial reasoning sub-tasks: Angle, Distance, and Relative Position along the X, Y, and Z axes. A hand image with annotated joint indices supports multiple-choice questions for each task, generated directly from 3D joint coordinates. Correct answers are highlighted in green.
Breakdown of question types across training (left) and evaluation (right) splits. Each dataset in the HandVQA benchmark maintains a balanced distribution across five spatial reasoning tasks: angle, distance, and relative positions along the X, Y, and Z axes. This balanced composition supports fair evaluation across all pose-related subtasks.
Coming Soon!