An LLM trading agent that beats the Nasdaq 100 by twenty or thirty points in a backtest is not showing you skill. It is showing you that the answer was ...
An LLM trading agent that beats the Nasdaq 100 by twenty or thirty points in a backtest is not showing you skill. It is showing you that the answer was already in the training data. The moment you run the same agent on a period the model has never seen, the edge collapses toward the index, and the size of that collapse is a direct measurement of how much the backtest was contaminated in the first place.
That is the finding buried inside the recent work from a team of Chinese researchers who ran five LLM-based trading methods on GPT-4o, whose training cutoff of October 2023 is public. They tested the agents on Nasdaq 100 constituents across two windows: Q2 and Q3 2021, comfortably inside the training window, and Q3 and Q4 2024, comfortably outside it. Both periods delivered almost identical Nasdaq 100 returns of roughly 13.5%, which is the whole point. The market did the same thing in both windows. Only the models' knowledge of that market differed.
The Number That Actually Matters
Inside the training window, the agents returned between 30% and 44%, beating the index by a wide margin. Outside it, on a market that behaved almost identically at the index level, the same agents returned between 9% and 22%. Several of them underperformed the passive benchmark they had crushed three years earlier.
Read that gap as a corruption estimate rather than a performance figure. The 2021 outperformance was not generated by reasoning about fundamentals or momentum. It was generated by a model that had, in some diffuse and unremovable way, already seen how those stocks traded. The researchers did not need to find explicit memorized price series to prove leakage. The behavior proves it: identical market, wildly different results, with the only variable being whether the period fell before or after the knowledge cutoff.
This is the same failure the San Francisco Fed found when it asked ChatGPT to forecast inflation. Ringfence the model however you like, restrict its inputs, control the prompt; the training data still bleeds through and flatters the backtest. A model cannot un-know what it learned. When the test period sits inside that knowledge, you are not measuring forecasting ability. You are measuring recall dressed up as forecasting.
Why This Is a Discount-Rate Problem, Not a Software Problem
The cross-asset read here is that the leakage is not evenly distributed across time; it is concentrated in exactly the regime the model was trained on, and that has a specific consequence for how these strategies behave when the macro backdrop shifts.
The 2021 window was a zero-rate, liquidity-flooded market. Duration was cheap, multiples were expanding, and the Nasdaq 100 was the purest expression of a discount rate pinned near zero. GPT-4o was trained on an internet saturated with commentary, price action, and post-hoc rationalization of that regime. So when a trading agent is dropped into 2021, it is not just recalling prices; it is recalling the entire behavioral logic of a low-rate melt-up that the training corpus explains in exhaustive detail.
Move the same agent to late 2024, with the 10Y anchored near 4.6% and the front end still restrictive, and the leaked knowledge is worth far less. The market is being priced off a different discount rate, and the model's absorbed intuitions about which names lead and how drawdowns resolve were formed under a regime that no longer holds. The performance decay is steepest precisely because the training regime and the test regime sit on opposite sides of a rates repricing. An LLM trading agent is, in effect, long the regime it was trained in.
That is the transmission channel worth naming. These agents will look most impressive backtested over the exact conditions that produced their training data, and they will degrade fastest when the rate environment that shaped that data inverts. The 2s10s at +0.47% today describes a curve that has already left the 2021 world behind. Any agent whose priors were set in that world is being asked to trade a market its intuitions were never built for.
The Fix Is Older Than the Technology
The researchers propose a repair, and it is worth taking seriously precisely because it is unglamorous. They ask the trading agents not only to backtest a strategy but to stress it against counterfactual scenarios: synthetic market paths the model never saw in training. Stripped of the framing, this is Monte Carlo simulation with a language-model wrapper. A strategy that survives both the historical backtest and the artificial environment has a better claim to being real rather than remembered.
Their results support it, with the honest caveat that the counterfactual test was run on five stocks and Bitcoin rather than the full Nasdaq 100. On that narrow set, forcing the strategy through synthetic environments materially improved live-period performance. The mechanism is intuitive: a counterfactual path is, by construction, outside the training data, so a strategy that works there is drawing on structure rather than recall.
The uncomfortable implication is that the entire value of an LLM trading agent may reduce to whether it can generate and reason about environments it was never trained on. Everything else is backtest theater.
What Would Break This Read
The thesis is that the performance gap is leakage, and the cleanest challenge to it is prosaic: two three-month windows are short, and 2021 and 2024 differed in more than the model's knowledge. Volatility structure, dispersion within the index, and single-name concentration were not identical, and it is possible that some of the 2024 underperformance reflects a genuinely harder alpha environment rather than the absence of leaked data. A skeptic could argue the agents simply met a tougher market, not a fairer one.
Two things make that harder to sustain. The index return was nearly the same in both windows, which removes the simplest "the market was just worse" explanation. And the same decay pattern shows up in a completely separate task, inflation forecasting, run by a different institution on a different model family. When the identical failure mode appears in both forecasting and trading, the parsimonious reading is that the common ingredient, training-data leakage, is doing the work.
The read breaks if someone reproduces this with a strictly enforced, verifiably leakage-free training set and the outperformance survives out of sample. Until that exists, treat any LLM trading result reported on an in-training-window backtest as unverified by construction. The observable condition to watch is narrow and testable: run the agent on a post-cutoff period against a matched pre-cutoff period with comparable index returns, and measure the gap. If the gap is large, the strategy is remembering. If it disappears, you have found something worth funding.
