Norm-Hierarchy Transitions In Representation Learning: When And Why Neural Networks Abandon Shortcuts

Neural networks often rely on spurious shortcuts for hundreds of epochs before discovering structured representations. Yet the mechanism governing when this transition occurs-and whether its timing can be predicted-remains poorly understood. While prior work has established that gradient descent converges to low-norm solutions [?] and that networks exhibit simplicity bias [?], neither line of work characterises the timescale of the transition from simple to structured features. We propose a unifying framework-the Norm-Hierarchy Transition-which explains delayed representation learning as the slow traversal of a norm hierarchy under regularised optimisation. When multiple interpolating solutions exist with different norms, weight decay induces a slow contraction from high-norm shortcut solutions toward lower-norm structured representations. We prove a tight bound on the transition delay: T = Θ(γ-1 eff log(V sc /V st)), where V sc and V st are the characteristic norms of the shortcut and structured representations. The framework predicts three regimes as a function of regularisation strength: weak regularisation (shortcuts persist), intermediate regularisation (delayed transition), and strong regularisation (learning suppressed). We validate these predictions across four domains: modular arithmetic (where all six predictions hold with R 2 > 0.97), CIFAR-10 with spurious features (five of six, including 78% → 10% clean accuracy as shortcut strength increases), CelebA (which occupies an intermediate position on the norm separation spectrum), and Waterbirds (where norm dynamics transfer but representational transition does not, confirming the framework's boundary). The norm-hierarchy mechanism is robust across architectures: ResNet18 with standard batch normalisation exhibits the same peak-then-decay norm dynamics as models without normalisation, achieving 78% clean accuracy. The single prediction that fails to transfer-the precise delay scaling T ∝ 1/λ-is explained by a new condition we term clean norm separation, the first formal criterion distinguishing settings where implicit bias timescales are predictable from those where they are not. The framework further predicts that emergent abilities in large language models arise when model scale reduces the norm gap below a training-budget threshold, connecting scaling laws to the same norm-hierarchy mechanism. Our results suggest that grokking, shortcut learning, delayed feature discovery, and emergent abilities are manifestations of a single mechanism: the slow traversal of a norm hierarchy under regularised optimisation.