From b1122863c8ec0e002040fd20dbb068784533c44e Mon Sep 17 00:00:00 2001 From: =?utf8?q?J=C3=A9r=C3=B4me=20Benoit?= Date: Thu, 30 Jul 2026 22:43:55 +0200 Subject: [PATCH] docs(quickadapter): define backtest evaluation protocol (#154) * docs(quickadapter): define backtest evaluation protocol * docs(quickadapter): unify outer prediction window terminology Harmonize the lone bare "outer window" in the backtest evaluation protocol nested-HPO step with the "outer prediction window" term used elsewhere in the section, removing an ambiguous synonym for the rolling-origin evaluation window (distinct from the "outer holdout"). * docs(quickadapter): harmonize backtest-protocol terminology and qualify drawdown bootstrap - clarify the row-wise known_at_lookahead maximum spans label and weight availability values - use code font consistently for continual_learning - align the label-candle shuffling seed role with the configuration tunables table - name log returns explicitly before the positivity requirement - unify "latest retained information time" across the purge step - note the maximum-drawdown bootstrap bound is approximate for a non-smooth path-dependent functional * docs(quickadapter): gate step 8 effect-size bootstrap on stationarity credibility Make the stationary-bootstrap applicability in step 8 symmetric with the Reality Check/SPA gate in step 9: declare the result inconclusive when the paired series' stationarity and local-dependence assumptions are not credible under a fixed or moving-window scheme. Addresses the outstanding PR #154 review comment on the step 8 vs step 9 stationarity asymmetry. * docs(quickadapter): harmonize primary-metric naming and fix spelling - name the step 9 promotion bound the "net-log-return effect", consistent with the step 7-8 net-log-return inference and the positive-is-favorable effect terminology (removes the one-off "net-growth" synonym for the same estimand) - correct "relabelled" to "relabeled" for US-English consistency --- README.md | 224 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+) diff --git a/README.md b/README.md index a3ae17a..5407b91 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ - [QuickAdapter](#quickadapter) - [Quick start](#quick-start) - [Configuration tunables](#configuration-tunables) + - [Backtest evaluation protocol](#backtest-evaluation-protocol) - [ReforceXY](#reforcexy) - [Quick start](#quick-start-1) - [Supported models](#supported-models) @@ -148,6 +149,229 @@ docker compose up -d --build | freqai.optuna_hyperopt.reset_label_study_on_schema_mismatch | true | bool | Reset a persisted `label` study when its selection schema is missing, invalid, or incompatible. `true` keeps the historical destructive reset, deleting the study before recreating it; `false` preserves its trials and stored metadata, permits caller-managed reuse in memory, and does not persist selected params until the schema is reconciled. Both fail closed: an inspection error, or (under `true`) a deletion error, aborts study creation. Has no effect when `continuous=true` or outside live/dry-run modes, where studies are always reset. | | freqai.optuna_hyperopt.vary_model_seed_by_trial | true | bool | Add `trial.number` to each regressor's configured model seed (or its default seed of `1`) during HPO. `true` samples model randomness across trials and preserves the historical behavior; `false` evaluates every trial and the final fit with the same model seed. This does not change `freqai.optuna_hyperopt.seed`. | +### Backtest evaluation protocol + +Use this protocol before adopting a change to a QuickAdapter default. It is an +evaluation procedure, not evidence that any current default is optimal. +When `freqai.data_split_parameters.test_size` yields a non-empty outer holdout, +QuickAdapter's `holdout_rmse` is the weighted prediction error of the selection +model on the original label scale. When the effective validation size is +nonzero, the final model is subsequently refitted. With `test_size=0`, the +holdout is empty, `holdout_rmse=inf` is an unavailable-metric sentinel rather +than a measured error, and the final refit is skipped. The label is smoothed +Zigzag morphology rather than a return. Consequently, a finite `holdout_rmse` is +a diagnostic, not an economic objective or a promotion criterion. + +#### Scope of a native FreqAI backtest + +A native FreqAI backtest trains and predicts each sliding model window before +`backtesting_fit_live_predictions()` replays `fit_live_predictions()` over the +assembled prediction frame. It can evaluate fixed training, feature, regressor, +prediction-threshold and strategy settings with FreqAI's rolling prediction +windows. It also exercises the replayed thresholds and strategy decisions. + +The replay cannot make a label-HPO result or other adaptive state affect a model +window that has already been trained. A native backtest therefore does not +evaluate QuickAdapter's complete live feedback loop. Evaluate that loop with a +chronological runner which trains, predicts, updates state and only then advances +to the next window, or with a forward dry-run. Keep native-backtest and +full-loop results in separate report sections. This repository does not provide +that chronological runner. See the [FreqAI running guide][freqai-running] and +the [FreqAI callback order][freqai-source]. + +#### Procedure + +1. **Pre-register the decision.** Before inspecting candidate results, freeze the + incumbent, candidate change, pair universe, data source and hash, outer + timerange, rolling-window schedule, point-in-time pair-eligibility and + missing-data rules, random seeds, HPO budget, costs, baselines, minimum + coverage, primary metric and tested families or model universes. For every + comparison, freeze the positive-is-favorable estimand, practical margin and + its units, test statistic and one-sided confidence-bound construction; also + freeze each raw or universe p-value construction, the familywise error rate + and correction method. Count every configuration inspected, including failed + and abandoned trials. Reserve a final chronological confirmation period that + is not used to choose features, parameters, margins or the candidate. +2. **Build one paired rolling-origin path.** At every outer prediction window, + train each candidate only on data available before that window and score all + candidates on the same timestamps. Fit scalers, label transforms, feature + selection and regressor HPO inside the corresponding training window. + Calibrate prediction thresholds only from prior out-of-sample predictions + available at the decision time. Use distinct FreqAI identifiers and storage + for every candidate/seed; never share model, prediction or Optuna state + between them. +3. **Purge labels and impose an embargo gap.** Keep `causal_mode` enabled. Remove + every training row whose label-information interval overlaps the next + validation or prediction window. In the unsliced window frame, take the + row-wise maximum of the `known_at_lookahead` values of all emitted labels + and weights, add the row's local position, and map that availability + position to its information time. Treat every non-finite or out-of-frame + availability position as unavailable and purge it; never clip it to a + boundary timestamp. Verify that the latest retained information time + precedes the scoring window. Then apply a pre-registered non-negative gap + between the latest retained information time and that window to cover any + additional data publication or execution delay. If an inner procedure uses + observations after a validation block, also embargo the observations + immediately after that block; the preferred rolling-origin design never + trains on future observations. Record the purged row count and effective + gap for every split. Use an external runner when the native window schedule + cannot express the pre-registered gap. + These controls follow the purging and embargo concepts in [Advances in + Financial Machine Learning][afml]. +4. **Separate the experiment from HPO.** First compare the incumbent and + candidate with label and model hyperparameters fixed, changing only the + component under test. If HPO is part of the proposed behavior, run a second + nested experiment: tune regressor parameters only on time-ordered inner + training/validation splits, then evaluate once on the untouched outer + prediction window. QuickAdapter's dynamic label HPO is different: it selects + label parameters from causal morphology objectives in `fit_live_predictions()`; + reproduce that deployed selector and judge it only by later outer economic + results. Do not reinterpret label trials as regressor-validation trials. Any + warm-start state used by an inner trial or selection model must have been + fitted only on that split's training prefix; otherwise disable + `continual_learning` during selection. Evaluate a pre-registered + `continual_learning` treatment only on later outer prediction windows. +5. **Pair stochastic runs.** Use the same seed list for every candidate and at + least five complete paired seeds for a promotion decision; fewer seeds are + diagnostic only. Do not replace or omit a pre-registered seed: a + candidate-attributable training, prediction or runtime failure fails its + operational promotion gate. An incumbent-attributable failure makes the + comparison inconclusive unless a deployable fallback and its finite scoring + rule were pre-registered. Retry the same seed only for a documented + infrastructure failure, under one pre-registered retry budget and policy + applied to both arms; exhausted retries make the experiment inconclusive. + Record the Optuna sampler and label-candle shuffling seed roles, regressor + seed and bootstrap seed separately. QuickAdapter currently derives both + Optuna roles from `freqai.optuna_hyperopt.seed`. Sequential HPO removes + parallel trial-scheduling nondeterminism and improves sampler replayability; + exact replay also requires deterministic backend settings and fixed software + and hardware. Otherwise, treat full searches as stochastic repetitions. If + parallel HPO is the deployed behavior, also report its trial order. +6. **Model execution costs explicitly.** Except for the single pre-registered + component under test, hold position sizing, protections, order types and fill + rules constant. Run a declared base-cost case and at least one adverse case + covering fees, spread, slippage and market impact; + include funding or borrow costs when applicable. Use + `--enable-protections`, explicit `--fee`, `--cache none` and + `--timeframe-detail ` backed by downloaded detail candles + where feasible. Apply non-fee costs through a pre-registered conservative + price or external fill model; this repository does not provide that model. + Freqtrade's candle simulator cannot prove queue position, fill probability or + capacity, so validate material fill/callback behavior in forward dry-run. See + the [backtesting assumptions][freqtrade-backtesting]. `--cache none` disables + Freqtrade's backtest-result cache, not model artifacts under a reused FreqAI + identifier. +7. **Measure portfolio economics and model health.** Preserve the whole + portfolio's net marked-equity path at the strategy or detail timeframe. + Aggregate a copy at fixed UTC daily boundaries for paired net-log-return + inference and consistent attribution of trades crossing model windows. + Before computing log returns, require finite, strictly positive marked + equity at every timestamp in every paired run. Any candidate breach is an + automatic failed promotion gate for that cost case, remains counted, and + must not be relabeled an invalid seed or omitted. An incumbent breach makes + the paired log statistic undefined and the experiment inconclusive unless a + finite treatment was pre-registered. + Compute drawdown from the full intraday path instead of daily marks or + averaged interval drawdowns. Derive this path externally when the backtest + output does not expose it; do not substitute closed-trade balance. Report net + return, drawdown, exposure, turnover, trades, partial exits, exit reasons and + long/short contribution. Also report per-window `holdout_rmse`, raw and + post-pipeline row counts, feature count, latest retained information time, + pivot prevalence, threshold support and training latency. Compute any + additional label-prediction metric after inverse transformation to the + original label scale. Compare against the incumbent and, when relevant to the + change, fixed-label/fixed-model ablations. Cash and buy-and-hold are context + baselines, not substitutes for the paired incumbent comparison. +8. **Quantify both temporal and seed uncertainty.** Across the complete + pre-registered family or model universe, use one hierarchical paired + stationary bootstrap. In each replication, resample a list of paired seed + indices of the original length with replacement, draw one + stationary-bootstrap sequence of calendar-day blocks, and apply the same seed + selection and block sequence to every run entering every comparison. + For drawdown, concatenate the selected days' full intraday equity-return + segments before recomputing the statistic. Because maximum drawdown is a + non-smooth path-dependent functional, read its bootstrap bound as + approximate rather than a calibrated interval. Pre-register an independent + pilot period, a block-length selection rule applied to representative paired + differentials, and a conservative longer-dependence sensitivity range before + running the pilot; fix the resulting settings before opening evaluation + results and require the decision to hold throughout the range. Use at least + 10,000 replications per setting and record the bootstrap seed. Report the + paired effect, one-sided confidence bound and seed distribution. Declare the + result inconclusive when the pre-registered minimum number of effective + blocks, trades, scored days or valid seeds is not met, when resampling is + degenerate, or when the paired series' stationarity and local-dependence + assumptions are not credible under a fixed or moving-window scheme. The + stationary bootstrap preserves local dependence under its assumptions; it + does not create information absent from a short backtest [Politis and + Romano][stationary-bootstrap]. See [Politis and White][block-length] for + data-driven block-length selection. +9. **Control selection and decide once.** Orient effects so positive values + favor the candidate. A promotion requires the one-sided lower confidence + bound for the primary net-log-return effect to exceed its practical margin, + the lower bound for incumbent-minus-candidate maximum drawdown to exceed the + negative degradation margin, all operational constraints to pass, and the + conclusion to hold in every pre-registered cost case and on the untouched + confirmation period. Use each comparison's positive-is-favorable effect and + pre-registered practical margin to test `H0: effect <= margin` against + `H1: effect > margin`. For each candidate, combine its mandatory statistical + gates and cost cases into an intersection-union p-value: the maximum component + p-value. Do not correct these conjunctive components against one another; keep + deterministic operational constraints as pass/fail. Apply + [Holm adjustment][holm] across the resulting candidate-level or confirmatory + claims that offer alternative routes to promotion. Report every component + statistic and raw p-value, the candidate-level p-value, Holm-adjusted p-value + and familywise decision. + A [Reality Check][white-reality-check]/[SPA test][hansen-spa] may instead test + a compatible model-universe claim, but its single universe p-value does not + identify a promotable model; a specific promotion still requires the + model-specific multiplicity-adjusted inference above. Reality Check/SPA + applies only to a time-indexed additive performance-differential series whose + estimand is its per-period expectation; it does not directly test + path-dependent statistics such as maximum drawdown. Express every universe + margin in the same per-period units, subtract it observation by observation, + and use the published null construction. For these published tests, + pre-aggregate complete seed runs at each timestamp by a pre-registered + function to form one differential series per candidate, then bootstrap only + the time index with common blocks across the complete universe; the + hierarchical seed resampling in step 8 is separate effect-size uncertainty. + Include SPA studentization and sample-dependent recentering. Apply the tests + only when their stationarity assumptions are credible and the scheme is fixed + or moving-window; the published SPA test does not cover recursively estimated + state. Report the seed aggregation, universe statistic and bootstrap p-value. + A [Deflated Sharpe Ratio][dsr] or [Probability of Backtest + Overfitting][pbo] analysis is a useful selection-bias audit, not a replacement + for chronological confirmation. If any gate fails, keep the incumbent and + report the result as inconclusive or rejected. + +Before reporting, run [lookahead analysis][lookahead-analysis] and [recursive +analysis][recursive-analysis] with a startup window long enough for every +informative timeframe. Freqtrade supports FreqAI lookahead analysis, but its +documented target indicators from `set_freqai_targets()` can be false positives; +ignore only those target-column flags, not feature or signal differences. Before +each command, set `freqai.identifier` to a value never used by another run and +verify that `/models/` is absent. Use separate fresh +identifiers for the two analyses: `--cache none` only bypasses the backtest-result +cache. +Archive a timestamped manifest containing the Git commit, image digest, +Freqtrade and dependency versions, configuration and data hashes, exact +commands, identifiers, seeds, HPO trial histories, per-window cutoffs, costs +and result hashes. + +[afml]: https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086 +[block-length]: https://doi.org/10.1081/ETC-120028836 +[dsr]: https://doi.org/10.3905/jpm.2014.40.5.094 +[freqai-running]: https://www.freqtrade.io/en/stable/freqai-running/ +[freqai-source]: https://github.com/freqtrade/freqtrade/blob/2026.6/freqtrade/freqai/freqai_interface.py#L396-L402 +[freqtrade-backtesting]: https://www.freqtrade.io/en/stable/backtesting/ +[hansen-spa]: https://doi.org/10.1198/073500105000000063 +[holm]: https://www.jstor.org/stable/4615733 +[lookahead-analysis]: https://www.freqtrade.io/en/stable/lookahead-analysis/ +[pbo]: https://doi.org/10.21314/JCF.2016.322 +[recursive-analysis]: https://www.freqtrade.io/en/stable/recursive-analysis/ +[stationary-bootstrap]: https://doi.org/10.1080/01621459.1994.10476870 +[white-reality-check]: https://doi.org/10.1111/1468-0262.00152 + ## ReforceXY ### Quick start -- 2.53.0