A multimodal fusion network based on variational autoencoder for distinguishing SCLC brain metastases from NSCLC brain metastases

Abstract Background Distinguishing small cell lung cancer brain metastases from non‐small cell lung cancer brain metastases in MRI sequence images is crucial for the accurate diagnosis and treatment of lung cancer brain metastases. Multi‐MRI modalities provide complementary and comprehensive information, but efficiently merging these sequences to achieve modality complementarity is challenging due to redundant information within radiomic features and heterogeneity across different modalities. Purpose To address these challenges, we propose a novel multimodal fusion network, termed MFN‐VAE, which utilizes a variational auto‐encoder (VAE) to compress and aggregate radiomic features derived from MRI images. Methods Initially, we extract radiomic features from areas of interest in MRI images across T1WI, FLAIR, and DWI modalities. A VAE encoder is then constructed to project these multimodal features into a latent space, where they are decoded into reconstruction features using a decoder. The encoder‐decoder network is trained to extract the underlying feature representation of each modality, capturing both the consistency and specificity of each domain. Results Experimental results on our collected dataset of lung cancer brain metastases demonstrate the encouraging performance of our proposed MFN‐VAE. The method achieved a 0.888 accuracy and a 0.920 AUC (area under the curve), outperforming state‐of‐the‐art methods across different modal combinations. Conclusions In this study, we introduce the MFN‐VAE, a new multimodal fusion network for differentiating small cell from non‐small cell lung cancer brain metastases. Tested on a private dataset, MFN‐VAE demonstrated high accuracy (ACC: 0.888; AUC: 0.920), effectively distinguishing between small cell lung cancer brain metastases (SCLC) and non‐small cell lung cancer (NSCLC). The SHapley Additive explanation (SHAP) method was used to enhance model interpretability, providing clinicians with a reliable diagnostic tool. Overall, MFN‐VAE shows great potential in improving the diagnosis and treatment of lung cancer brain metastases.