Comparing feature maps generated using UNet-like CNN, Transformer, Mamba, and hybrid architectures for general land cover mapping

This study compares feature maps produced by semantic segmentation architectures using varying combinations of convolutional neural network (CNN), Transformer, and Mamba selective state space (selective SSM) components with a goal of exploring the following question: does correlation or similarity between the generated data abstractions imply comparable predictive performance? Specifically, different encoder and decoder combinations are compared including a fully convolutional neural network (CNN)-based UNet, a Vision Transformer-based Swin-UNet, and a selective state space model (selective SSM)-based Mamba-UNet. For Swin-UNet and Mamba-UNet, we replaced the decoder with a CNN-based architecture comparable to that used within UNet. Central kernel alignment (CKA) and the Mantel test suggest a high degree of similarity between the feature maps generated by the decoder blocks and final feature map representations, respectively; however, the encoder feature maps were more dissimilar. Despite these similarities, differences in model performance were observed. The Mamba-based architectures generally yielded the lowest training losses and highest validation F1-scores. When used as a feature space for traditional machine learning (ML) algorithms, random forest (RF) and support vector machines (SVMs), the Mamba-based feature space generally provided the highest macro-averaged, class-aggregated F1-scores and map image classification efficacies (MICE). Feature reduction from 96 to 15 variables using principal component analysis (PCA), kernel PCA (kPCA), or independent component analysis (ICA) obtained comparable performance in comparison to using the original, larger feature spaces. When provided with all 480 feature maps generated by all five architectures, the Mamba-based features were found to be most important. The study documents that, despite similarities between the final set of feature maps generated by the final decoder block, different architecture combinations yielded varying levels of performance: similarity between feature spaces did not imply similar predictive performance. Further, we argue that DL-based methods can serve as a means to generate data abstractions for use in traditional ML workflows, and feature reduction can be implemented to reduce the computational load.