The backtest that read a quarter of the data

NexForge can now read historical US equity bars from a local store rather than a rate-limited API. The whole reason that feature exists is to stop a backtest from silently running on less data than you asked for. It shipped with a defect that did exactly that, an order of magnitude worse than the problem it replaced.

The bug

The store holds 1-minute bars. A request for, say, 200,000 bars of hourly data has to scan some window of minutes and fold them into buckets. Scanning the symbol's entire history every time would be wasteful, so the query bounds the scan first.

It bounded it in seconds:

// bound the scan to the span the request could cover
const floor = maxTs - width * limit;

That arithmetic assumes bars are continuous — that limit bars of width width occupy width × limit seconds of wall clock. For crypto, which trades every minute of every day, that is true. For equities it is nowhere close.

A regular US trading session is 09:30 to 16:00 ET: 390 minutes out of 1,440. Weekends produce nothing. Holidays produce nothing. Early closes produce 210. So the bars a calendar day contains are roughly a quarter of the bars that arithmetic expects it to contain, and the scan floor lands about four times too close to the present.

What that measured

Measured against a store holding 196,560 bars — deliberately under the 200,000 cap, so all of them were the correct answer to the request:

Nothing downstream was wrong. The engine computed 24 metrics correctly. The equity curve was an accurate equity curve. The Monte Carlo reshuffle honestly reshuffled the trades it was given. Every component behaved exactly as designed, on the wrong input, which is why nothing anywhere had a reason to complain.

This is the specific danger of a backtest as an artifact: it produces a plausible number under every condition. There is no output that looks like an error. A strategy validated on five months of a bull leg and reported as two years of mixed conditions is not a mildly imprecise result — it is the exact overfitting failure the whole discipline of out-of-sample testing exists to prevent, arriving through the data layer instead of through the strategy.

Why four reviews missed it

The read path had been reviewed four times, including a full task review and a fix round. The bug survived all of them for a reason worth stating out loud: every test fixture was ten bars.

Ten-bar fixtures verify shape. Does it resample correctly, does it preserve gaps, does it filter to regular hours, does it return ascending buckets. They pass identically whether the scan window is right or four times too small, because with ten bars in the store the floor never excludes anything. The property that broke only exists at scale, and nothing in the suite operated at scale.

The general form: a bug in a bound cannot be caught by a fixture smaller than the bound. Any test of a limit, a page size, a cap or a window needs at least one case that seeds past it. The regression test for this one does exactly that, and it is the only test in the file that is slow.

The fix, and why it deliberately over-fetches

The scan is now bounded by sessions rather than by seconds — count how many trading days could supply the requested buckets, then floor the scan at the oldest of them:

/** Buckets an early-close session (09:30-13:00 ET, 210 minutes)
 *  yields. Used to size the scan window, so it must be the FLOOR
 *  across session lengths — a full 390-minute day only ever
 *  produces more, never fewer. */
const MIN_BUCKETS_PER_SESSION =
  { "1m": 210, "5m": 42, "15m": 14, "1h": 4, "4h": 1, "1d": 1 };

Sizing off the shortest possible session means a normal day contributes more buckets than the estimate assumed, so the scan reaches further back than strictly necessary and slice(-limit) trims the excess. That is a deliberate asymmetry:

// Sizing off the shortest possible session over-fetches on
// normal days, which is the safe direction: slice(-limit)
// below trims the excess, whereas under-fetching would return
// fewer bars than asked for with no error.

Over-fetching costs milliseconds on a local SQLite read. Under-fetching costs the correctness of every number in the report and says nothing. When the two failure directions are that unequal, the bound should be wrong in the cheap direction on purpose.

The floor itself is a count of distinct trading days present in the store, not a calendar subtraction — so holidays, halts and missing months shift it automatically rather than requiring a calendar the app would then have to maintain:

SELECT MIN(d) AS floorDay FROM (
  SELECT DISTINCT ts / 86400 AS d
  FROM bars_1m WHERE symbol = ? AND in_session = 1
  ORDER BY d DESC LIMIT ?
)

The shape to look for in your own code

Strip out the equities and this is a unit mismatch: a quantity requested in one unit (bars) and enforced in another (seconds), with a conversion factor that is only constant for markets that never close. Crypto made the conversion look correct. Equities are what the assumption was always false for.

Three questions that would have caught it, and now get asked of every bound in this codebase:

• What unit is the caller asking in, and what unit does the guard enforce? If they differ, something is converting, and that conversion has assumptions.
• Which direction does it fail? Returning less than requested with no error is categorically worse than returning more, or than failing loudly. Pick the wrong direction on purpose.
• Does any test exceed the limit? If the largest fixture is smaller than the smallest bound, the bound is untested no matter how many tests there are.

Why this is on the blog

Because it is the exact class of defect this product is supposed to protect people from. A backtest is only worth running if you can trust the sample it ran on, and the 24 metrics are all downstream of that one property. Publishing the number — 27.6% — is cheaper than asking anyone to take our word for how seriously we take it.

It also sets an expectation worth setting before we ever charge for this: when the data layer is wrong, you will read about it here with the measurement attached.