LifeLong Learning for Large Language Models in Predicting Chemical Reaction Yields

Large Language Models (LLMs) based on transformer architectures excel at internet-scale tasks. However, real-world scientific scenarios—such as synthetic chemistry laboratories and autonomous experimental setups—typically involve incremental data generation in batches as new chemical reactions are conducted, unlike static, large-scale datasets. Motivated by scaling laws—which suggest that larger models are increasingly prone to catastrophic forgetting when learning from sequentially arriving data— we present a case study investigating continual learning for chemical reaction yield prediction using Mistral-7B, a 7.3-billion-parameter open-weight LLM. We first establish a baseline by evaluating model performance on the Suzuki Coupling Reactions dataset using both supervised full fine-tuning and Low-Rank Adaptation (LoRA), showing competitive yield prediction accuracy. To mimic real-world conditions, we adopt a task-incremental learning framework in which the model incrementally learns a new task group defined by unique pairs of reactants. This sequential learning setup enables us to directly assess the model’s ability to retain prior knowledge. We demonstrate that when the model is trained within this sequential learning paradigm with traditional procedures, it exhibits significant loss of prior knowledge, a phenomenon known as catastrophic forgetting in the continual learning community—an under-recognized challenge in the chemistry community. To understand this challenge, we first model the problem of continual learning and investigate the source of forgetting, subsequently, we incorporate experience replay to maintain near-baseline performance across tasks without forgetting. These results highlight the importance of integrating continual learning strategies into LLM-based chemical modeling pipelines, particularly as future experimental platforms increasingly generate non-stationary reaction data.