Explaining the Imperfect: How do LLMs Respond to Smelly Code?

Code smells, indicators of suboptimal design or implementation, contribute to technical debt by impairing software comprehensibility and maintainability. While Large Language Models (LLMs) can define code smells accurately when prompted, their ability to appropriately handle smelly code when generating explanations is not well understood, despite the prevalence of smells in training datasets. This study empirically investigates how two distinct LLM types-the general-purpose chat applications (e.g., GPT-4o, GLM-4) and the open-source code-specific models (e.g., CodeT5+, CodeQwen1.5)-respond to method-level Java code containing smells. We categorized smells into Structural Complexity, Data/Type Issues, and Expression Clarity, and tasked LLMs with generating summary-length explanations for smelly and non-smelly code. Our multi-faceted evaluation, including N-gram metrics and corpus-level analysis, reveals that both Chat LLMs and open code LLMs generally exhibit limited sensitivity to the presence of these code smells in their explanatory outputs. They tend to produce consistent, fluent explanations that often do not significantly differentiate between smelly and non-smelly code. These findings underscore the need for LLMs to develop a more nuanced awareness of diverse code characteristics to effectively assist developers in understanding and addressing code quality issues.