A Neuro-symbolic Framework for Hallucination Detection in Image Captioning Using Knuth– Morris–Pratt Pattern Matching

The research on image captioning has improved significantly in terms of describe of a scene appears on an image. Thisparticular task of image caption generation is considered as most vital because of its wide scope multidimensional applicationtowards various societal usages. However, the existing caption generators could not be deployed with confidence which could serve100% accuracy because most of them suffers from hallucination. That means a large number of caption generator still strugglingto achieve the human like performance as far the semantic values or context of the sentence is concerned. We could predict thepotentiality of image captioning task that can leverage many other areas of computer science by simply observing the exponentialsgrowth in this topic. In spite of this popularity, the hallucination detection is most prominent issue nowadays that could finetunethe performance of the image caption generators. The term hallucination refers to the issue of predicting nonexistent object automatically solely due to the biasness of training data. In other words, when the caption generator automatically predicts a non-existing object from the input image this is called as object hallucination. Although the hallucination are three different types such as object hallucination, spatial hallucination and multimodal hallucination, the object hallucination is the most prominent one inthis field. To address this object hallucination problem, we have proposed a neuro-symbolic framework that not only generates thesemantically correct caption but also successfully detects the object hallucination by integrating the classic Knuth-Morris-Pratt(KMP) pattern matching algorithm. We have demonstrated this substring matching process by utilizing the KMP algorithm andfound a linear polynomial time complexity with in the controlled environment of experimental setup.