Write the experiment contract first
A regime-conditioned backtest evaluates a fully specified decision process, not just a colored return chart. Declare the inputs, transformations, label rules, decision schedule, allocation mapping, holding rules, return accounting, costs, and benchmark before inspecting evaluation outcomes. State whether the question concerns prediction, exposure management, or descriptive differences across labels. A threshold classifier can be tested without assuming a hidden-state model. Synthetic examples demonstrate mechanics only; they cannot establish historical forecasting ability or a strategy's expected performance.
Reconstruct the information timeline
Separate the period an observation describes from its release time, ingestion time, decision time, and first eligible execution time. A vintage snapshot records information available at a particular point; it is not interchangeable with today's revised history. The Philadelphia Fed's real-time-data research describes why this distinction matters for forecast evaluation. Use the archived value available at each decision, including earlier observations as they were then revised. An unavailable vintage is a documented limitation, not permission to substitute final values without qualification.
Fit everything inside the historical boundary
Scaling, feature selection, threshold estimation, imputation, and probability calibration all belong within the training boundary. Freeze the fitted objects before evaluating subsequent observations, or follow a predeclared rolling update schedule. For a separately fitted hidden-state model, filtered probabilities condition on observations through that time, whereas smoothed probabilities can use later observations. Even a filtered calculation leaks information if its parameters were estimated using the eventual test period. The statsmodels switching-model reference distinguishes these probability views; neither label guarantees a point-in-time experiment by itself.
Use chronological evaluation without boundary leakage
Reserve later time blocks for evaluation and keep a final assessment separate from configuration selection. If a training target extends into an evaluation block, remove or otherwise isolate that overlapping target according to a rule fixed in advance. Apply the same care to warmup data: earlier observations may initialize a transformation, but future observations may not. Record each attempted specification. The backtest-overfitting paper by Bailey and coauthors explains why selecting among many tried configurations changes the evidential meaning of the apparent winner.
Follow a fictional release through the ledger
In the entirely fictional timeline below, an indicator first arrives at +0.2 in arbitrary units. A declared positive-value rule assigns State A at the next decision. A later release revises that observation to -0.1, which would imply State B if reconstructed afterward. The historical decision remains A. The revision can affect later decisions under the declared policy, and a separate hindsight series may show B, but overwriting the original row would give the earlier decision knowledge it did not have. All times and values are invented.
Translate decisions into accountable returns
Measure outcomes only after the eligible action time. Define what happens when execution is delayed, an instrument is unavailable, or an order is only partially completed. Preserve the intended allocation separately from the simulated holding. Include cash returns, financing where applicable, distributions, and clearly scoped switching costs. Compare against a predeclared baseline using compatible return conventions. A same-close action needs a defensible information and execution assumption; an end-of-session label cannot silently receive the whole session's already-realized return.
Report the experiment, including its failures
Deliver a decision ledger, configuration history, vintage identifiers, train and evaluation boundaries, missing-input treatment, and gross-to-net accounting alongside summary statistics. Break results down by episodes and explain which observations dominate. Show sensitivity to reasonable timing and cost assumptions without replacing the primary specification after seeing the answer. Keep revised-data diagnostics visibly separate from the decision-time result.
Before interpreting performance, test timing invariants with synthetic records. Moving an input's release beyond the decision cutoff should prevent that decision from using it. Replacing a later revision should not alter an earlier archived action. These checks establish software behavior, not independent strategy-validation evidence. Preserve both the test design and its limited purpose.
A reproducible negative or inconclusive outcome is more informative than an unexplained favorable curve. This guide concerns research design, not investment advice or a claim that any backtest establishes future profitability.
Inspect the comparison
| Fictional event | Invented information | Permitted interpretation |
|---|---|---|
| Session 1, 08:30 UTC | First release: +0.2 arbitrary units | Eligible input after availability checks |
| Session 1, 08:31 UTC | Decision uses positive-value rule | Archive State A and its input vintage |
| Session 1, 09:00 UTC | Predeclared first eligible action | Begin outcome accounting under the execution rule |
| Session 2, 08:30 UTC | Earlier observation revised to -0.1 | Do not overwrite Session 1's archived decision |
Primary references
References support the stated methods and data conventions. Original fictional examples are not observations from the cited sources.