Visual, OCR-Text, and Hybrid Evidence Retrieval for Document Visual Question Answering: A Controlled Failure Analysis

Abstract Purpose Document visual question answering (DocVQA) systems can fail because relevant evidence is not retrieved or because the answer generator fails to use correctly retrieved evidence. This study investigates how lexical, dense text, visual, and hybrid page retrieval affect evidence selection and downstream answer quality, and whether correct-page access resolves remaining generation errors. Methods BM25, BGE-M3, MiniLM, ColQwen, and reciprocal-rank fusion were evaluated on a document-disjoint DocVQA protocol. Fusion parameters were selected using development data and fixed before final evaluation. Answer generation used Qwen2.5-VL under matched evidence conditions from BM25, ColQwen, fusion, and oracle pages. Retrieval performance was additionally evaluated on MP-DocVQA without retuning. Results ColQwen achieved the strongest page-retrieval performance on DocVQA, with Recall@1 of 0.559 compared with 0.335 for BM25 and 0.274 for BGE-M3. Reciprocal-rank fusion did not improve over ColQwen. Better retrieved evidence translated into higher answer quality, with ANLS increasing from 0.127 using BM25 pages to 0.213 using ColQwen pages. Oracle evidence further increased ANLS to 0.395, showing that substantial answer-generation errors remained even when the annotated page was available. The visual retrieval advantage also transferred to MP-DocVQA. Conclusion Visual page retrieval improved evidence selection and downstream answering in the evaluated settings, but retrieval alone did not resolve DocVQA errors. Fixed fusion provided no additional benefit, while the oracle results show that answer generation remains an important source of failure.