Deep model interpretation : analyzing internal features for understanding visual patterns encoded by deep models

This thesis aims at the analysis of representations internally encoded in Deep Neural Networks (DNNs) through interpretation tasks. In the first two works, we focus on studying interpretation methods and analyzing static internal features encoded from a single domain. Next, we analyze the dynamicity of features during the fine-tuning process. Finally, we evaluate the insights obtained from various interpretation methods and link them to semantic, human-understandable concepts. In the first work, we provide an overview of recent surveys, paying attention to the used terminology when referring to the model explanation and model interpretation tasks and point out potential discordance. Accordingly, we provide definitions in our work to clearly differentiate explanation from interpretation tasks. Building on this, we introduce the Framework for the Interpretation of Deep Convolutional Neural Networks, which includes seven factors characterizing interpretation methods. These factors serve as axes for positioning existing interpretation methods. The work mentioned above not only reveals various potential directions but also identifies the inherent computational cost problem in interpretation methods. To address this issue, in the second work, we propose a coreset-based interpretation framework that uses coreset selection techniques to sample a representative subset of the dataset to be used for interpretation tasks. Furthermore, we introduce an evaluation protocol based on representation similarity, which assesses the similarity between relevant features identified using the full dataset and those extracted using the coreset. Using this protocol, we evaluate the robustness of interpretation method outputs with respect to the size of the dataset provided to them. In the previous mentioned works, we focused on static encoded features. In the third work, we shift our attention to analyzing feature dynamics as a model transitions from a source domain so that the encoded features can be reused for a new target task, which may suffer from a domain shift. Specifically, we propose an analysis of the intermediate stages of the iterative fine-tuning process for a given model. Our quantitative analysis measures the similarity between representations from the pre-trained model and those at each stage of the fine-tuning process. Our qualitative analysis employs visualizations, such as activation heatmaps and average visualizations, to provide insights into the state of features encoded by the model at various stages of fine-tuning. Finally, we propose a unified evaluation protocol for the quantitative evaluation of interpretation methods, linking their feedback to semantic concepts.