Prompt Carefully! ChatGPT Displays Rule-Based Insensitivity to Contingencies

Rule-governed behavior in humans is characterized by relative insensitivity to changes in contingencies, a phenomenon extensively documented in behavior-analytic research. The present study examined whether Large Language Models (LLMs) exhibit analogous patterns of contingency insensitivity. We employed a rock–paper–scissors task in which models repeatedly made choices between two opponents (Sam or Alex). In the first block of 40 trials, selecting the optimal opponent (e.g., Sam) produced a 70% probability of winning, a 15% probability of a tie, and a 15% probability of losing. In the second block of 40 trials, these probabilities were reversed (i.e., Alex was the optimal opponent). Four frontier LLMs in January 2026 (GPT 5.2, Claude Opus 4.5, Grok 4.1 Fast, and Gemini 3 Flash) were evaluated under a 2 × 2 experimental design manipulating (a) the presence or absence of a rule describing the opponent's skill level and (b) the extended LLM's reasoning (present vs. absent). In rule conditions, prompts specified the purported skill of the initial optimal opponent (e.g., "Sam is not very good at this game"). Results indicated that all models exhibited rule-based insensitivity to contingencies, qualitatively resembling human rule-governed behavior. However, the degree of insensitivity varied across models: GPT 5.2 and Grok 4.1 Fast showed the greatest contingency insensitivity, whereas Gemini 3 Flash and Claude Opus 4.5 were comparatively more sensitive to contingency shift. The effect of extended reasoning varied across LLMs. This study is the first to demonstrate contingency insensitivity in LLMs. These results have important implications for applied LLM contexts, where LLMs' contingency insensitivity might be detrimental.