Cross-modal Retrieval based on Shared Proxies

Abstract Inconsistency of distribution and representation across different data modalities makes measuring cross-modal similarities a very difficult problem. Learning a common space that is semantically discriminative and modality invariant is the main challenge in cross-modal retrieval. Existing solutions usually employ pairwise or triplet data relationships to learn the common space, which can only capture the data similarity locally but would be unable to effectively characterize the global geometry of the common embedding space, and thus would limit the performance of cross-modal retrieval. In this paper, we introduce a shared proxy solution to cross-modal retrieval. We propose to incorporate the principles of shared proxy with neighbourhood component analysis to learn a common space for different modalities in which the distance between a sample’s representation and its corresponding proxy is minimized while the distances between a sample’s representation and the proxies not belonging to the sample are maximized. We propose the Cross-mOdal proXy learnIng (COXI) framework which integrates a cross-modal shared proxy loss, a discriminative loss and a modality invariant loss for supervised cross-modal retrieval. Extensive experiments on benchmark datasets clearly shows that COXI outperforms state of the art cross-modal retrieval techniques. Code is available on https://github.com/LigangZheng/COXI.