The excess volatility puzzle explained by financial noise amplification from endogenous feedbacks

The arguably most important paradox of financial economics – the excess volatility puzzle – first identified by Robert Shiller in 1981 states that asset prices fluctuate much more than information about their fundamental value. We show that the excess volatility puzzle is associated with an intrinsic propensity for financial markets to evolve towards instabilities. These properties, exemplified for two major financial markets, the foreign exchange and equity futures markets, can be expected to be generic in other complex systems where excess fluctuations result from the interplay between exogenous driving and endogenous feedback loops. Using an exact mapping of the key property (volatility/variance) of the price diffusion process onto that of a point process (intensity of price change arrival times), together with a self-excited epidemic model, we introduce a novel decomposition of the volatility of price fluctuations into an exogenous (i.e. efficient) component and an endogenous (i.e. inefficient) excess component. The endogenous excess volatility is found to be substantial, largely stable at longer time scales and thus provides a plausible explanation for the excess volatility puzzle. Our theory rationalises the remarkable fact that small stochastic exogenous fluctuations at the micro-scale of milliseconds to seconds are renormalised into long-term excess volatility with an amplification factor of around 5 to 10 for equity futures and 2 to 3 for exchange rates, in line with models including economic fundamentals explicitly. We also introduce a quantitative diagnostic of financial market efficiency, defined as the share of price fluctuations without any apparent offspring in the self-exciting epidemic dynamics, which indicates significant temporary breakdowns of market efficiency during market crashes and the associated bursts in volatility.