Four Confidence-flattering Failures in One Production Search: A Forensic Case Study

<div> This retrospective forensic case study audits one production parameter-search pipeline for a </div> <div> cryptocurrency market maker against 20 runs, 60 token-level decisions, and 52 MB of retained </div> <div> artifacts. It begins by refuting the claim that motivated it: the search does not always adopt its </div> <div> fitted maximum. It kept the live configuration in 37 of 60 decisions, and its out-of-sample guard </div> <div> demonstrably rejects a worse fitted winner. The generalisation that provoked the audit — that an </div> <div> automated search over a noisy trading objective reports confident adoptions from pure noise — is </div> <div> false about this tool and is withdrawn. </div> <div> <br> </div> <div> Four defects were found elsewhere in the pipeline. A swept parameter that the execution engine </div> <div> ignored produced a perfectly monotone and entirely meaningless response surface. A narrow ladder </div> <div> sweep let a single outlier present as a strong parameter effect. Independent-error assumptions </div> <div> applied to overlapping windows doubled the apparent number of threshold crossings. And a validator </div> <div> checking nominal notional passed a configuration whose pre-trade rejections were, without </div> <div> exception, on one side. In each incident the defective measurement made the pipeline's output look cleaner </div> <div> or more actionable than the repaired measurement did. </div> <div> <br> </div> <div> A fifth proposed incident — an objective apparently pinned at an infeasibility ceiling — is </div> <div> withdrawn. The objective is gradient-ordered by design, and the retained artifacts cannot </div> <div> reconstruct the full evaluation population; they support exact counts only at named scopes, and the </div> <div> paper states which. </div> <div> <br> </div> <div> The audit separately finds that all 12 adoptions carrying a retained out-of-sample profit figure </div> <div> adopt a loss, despite the tool's relative-to-baseline guard functioning correctly. That is a </div> <div> labelling and operator-action problem rather than evidence that automated search is unsound. </div> <div> <br> </div> <div> These incidents were selected retrospectively from one pipeline after discovery. They do not </div> <div> estimate how often search-pipeline defects occur, nor establish that such defects are generally </div> <div> biased toward confidence. They motivate a mechanism hypothesis — that some broken measurements </div> <div> reduce observed variance or suppress contradictory structure, so that failure resembles signal — </div> <div> and testing it would require a prospective incident sample or a controlled known-null mutation </div> <div> study, neither of which this paper has. </div>