fix(reforcexy): align training, inference and reward analysis contracts
* fix(reforcexy): account for portfolio transition returns
* fix(reforcexy): align training inference and reward analysis contracts
* fix(reforcexy): resolve statistical reporting and type-narrowing findings
- BH correction filters undefined p-values out of the finite family;
non-applicable tests keep significant flags None and NaN adjusted values.
- Bootstrap percentile CIs validate finite ordered bounds only; constants
keep exact degenerate bounds in both diagnostic modes without widening.
- Distribution diagnostics mark constant columns as N/A (skewness/kurtosis
None, normality tests and Q-Q omitted) and reject fabricated fallbacks.
- Explicit independent_observations contract across hypothesis tests,
bootstrap, shift metrics and report sections; trajectory report is
descriptive (no inferential p-values/CIs unless declared independent).
- Distribution shift computes exact ks_2samp statistics on raw values;
KS p-value reported only for independent observations.
- PBRS certification requires ordered-episode evidence: local identity
F = gamma*next - prev, continuity, discounted boundary residual;
non-canonical configurations are never certified from raw sums.
- Report renders actual bootstrap resample count, independent sections
and N/A constant rows; transforms rename bounded asinh to softsign_sqrt
without aliases; loss-potential bound docs corrected.
- Narrow Optional callback/env/attr types and force float arrays so the
BasedPyright snapshot holds 142 diagnostics; ruff clean.
* docs(reforcexy): align HPO continuation contract with merged behavior
* fix(reforcexy): harden inference contracts and reward accounting
Isolate deployed policy on continuation, account open positions at
terminal liquidation, guarantee Python bool terminated, handle OHLC
drop without holdout, prune no-learning DQN trials, preserve replay
buffer and chronology across restarts and splits, propagate complete
observation validity, reset temporal caches on discontinuities, fix
terminal neutral potential in analysis, align history events with
transitions, enforce loss amplification threshold, honor simulated
unrealized PnL, keep idle duration past threshold, reject missing
real_episodes files and unknown overrides. Transport inference dates
through dk.data_dictionary with explicit length alignment guard.
* fix(reforcexy): address review follow-ups
* refactor(reforcexy): consolidate constants and helpers
Remaining 4 are by-design, not user-facing message-convention items:
- 3 internal 'persisted ...' ValueErrors are immediately re-raised by
the conforming 'Training [%s]: cannot continue ...' DependencyException
wrapper; prefixing them would double-scope the surfaced text.
- FileExistsError(path) carries no literal message.
fix(reforcexy): port applicable QuickAdapter PR #261 fixes (#263)
* fix(reforcexy): port applicable QuickAdapter PR #261 fixes
- Align prediction history to requested candle dates and exact strategy
indexes (recorded/produced provenance, dedupe, exact-index attach).
- Freeze deployment feature coordinates for continual learning with
independent cache/disk state restoration and explicit rejection of
legacy or incompatible artifacts.
- Keep raw OHLC environment prices out of feature preprocessing; cold
HPO trials use fresh candidate pipelines while the final continuation
reuses the frozen deployment pipeline on the raw split.
- Return the current finite evaluation best checkpoint (best_model.zip)
with final-policy fallback.
- Optional per-pair produced-observation action statistics with session
warmup, numeric coercion and population standard deviation.
- Refresh BasedPyright snapshot; 2 argument-type errors resolved.
* fix(reforcexy): align resumed gamma and invalidate incompatible HPO state
* docs(reforcexy): clarify continuation and objective compatibility
* docs(reforcexy): condense continuation guidance
* fix(reforcexy): preserve replay buffer on best checkpoint selection
fix(quickadapter): preserve zero power mean exponent in label selection
_validate_power_mean_p returns 0.0 for p=0 (a finite float accepted by
README), and the falsy fallback silently rewrote it to 1.0, turning the
documented generalized-mean p=0 (geometric) into arithmetic. Use an
explicit None check for the fallback; map-based metrics are unchanged.
Verified in the QA image (Freqtrade 2026.8): p=0 now matches
geometric_mean selection (winner 1, identical distances), while p=None
keeps the arithmetic default.
chore(basedpyright): drop resolved reward_space_analysis ExtensionArray.size diagnostic
pandas-stubs 3.0.5.260914 resolves the previously-recorded
reportAttributeAccessIssue on ExtensionArray.size at
reward_space_analysis.py:2720, so the accepted ReforceXY snapshot no
longer matches current diagnostics. Remove the stale entry to unblock
the BasedPyright (ReforceXY) required check.
fix(qa): tolerate tool version drift in snapshot comparison
Compare snapshots with basedpyrightVersion normalized out: a tool upgrade that changes no diagnostics no longer fails the check. The recorded version stays in snapshots and --write output as provenance, is still validated nonempty on both sides, still appears in the failure diff, and both versions are reported in the success message. Diagnostics, file inventory and schema stay byte-exact, so changed output still fails closed. Covers scripts/test_check_basedpyright.py (stdlib unittest): version-only drift matches, diagnostic change mismatches, empty version and missing key rejected.
fix(renovate): anchor managed version values to digit-start
Require managed version values to start with a digit in both Dockerfile ARG and compose matchStrings. Kills the false-positive class where any *_version key with a non-version scalar (e.g. latest, stable) becomes a pypi lookup. Verified no-op on the current corpus: identical 26-capture set before/after.
chore(qa): refresh BasedPyright snapshots for 1.40.1
Repin accepted snapshots for the basedpyright 1.40.0 -> 1.40.1 bump. Diagnostics are unchanged in both projects (242 quickadapter / 159 reforcexy) — only the version marker moves. Snapshots generated with the same --pull docker build as CI.
feat(docker): manage xgboost installation and version
Pin xgboost as a managed build dependency, harmonized with the existing catboost/ngboost pattern: ARG xgboost_version in quickadapter/Dockerfile, matching build arg in docker-compose.yml, and automatic updates via the existing renovate custom regex manager (pypi datasource).
The freqtrade stable_freqai base image ships xgboost 3.2.0; this pins 3.4.1 so the QA and production images track the latest release independently of base image lag. ReforceXY is untouched: it does not use xgboost. BasedPyright snapshots unchanged (242/159 diagnostics verified with the rebuilt QA images).
chore(qa): refresh BasedPyright snapshots for 1.40.0
Repin accepted snapshots for the basedpyright 1.40.0 bump (pyright 1.1.412). QuickAdapter diagnostics are unchanged; ReforceXY keeps one message-only update from the re-pulled base image (base class step() return dict value inferred as dict[str, float] instead of dict[str, Unknown]) with the same rule, location, and severity. No added or removed diagnostics. Snapshots generated with the same --pull docker build as CI.
Jérôme Benoit [Fri, 28 Aug 2026 16:18:38 +0000 (18:18 +0200)]
fix(quickadapter): gate final take-profit on retracement
Replace the elapsed-time final take-profit exit with a persisted volatility-scaled retracement, harden its state invariants, and harmonize final-exit terminology.
Jérôme Benoit [Sun, 23 Aug 2026 21:14:34 +0000 (23:14 +0200)]
chore(quickadapter): update label_weighting and label_smoothing templates
- set label_smoothing method to kaiser_bessel_derived with window_candles 6
- add commented alternative label_weighting example (amplitude strategy,
epsilon_gaussian fill, knn bandwidth)
- align commented per-label beta example with canonical default (8.0)
Ports QuickAdapter's _install_date_pred_dedup_patch verbatim to ReforceXY. The duplicate-date_pred / validate="m:1" MergeError originates in the shared FreqaiDataDrawer (freqtrade 2026.7) and affects every FreqAI model; ReforceXY runs in its own process and never imports QuickAdapter, so it needs its own identical copy.
Jérôme Benoit [Wed, 12 Aug 2026 20:54:36 +0000 (22:54 +0200)]
fix(quickadapter): guard against freqtrade duplicate date_pred regeneration (#202)
* revert(scripts): drop historic-predictions dedup tool
Reverts the squash-merged PR #201 (commit c20d23a2). A one-shot on-disk
deduplication cannot durably fix the issue: freqtrade 2026.7 regenerates the
duplicate date_pred in memory on restart. It is superseded by an in-memory
guard in the QuickAdapter model (next commit).
* fix(quickadapter): guard against freqtrade duplicate date_pred regeneration
freqtrade 2026.7 FreqaiDataDrawer.set_initial_return_values trims the new
prediction window by position (new_pred.iloc[len(common_dates):]) instead of by
date_pred value. When the persisted history overlaps the tail of the analysis
window (the normal restart geometry), the trim misaligns and the following
pd.concat re-creates duplicate date_pred rows, which the validate="m:1" merge in
attach_return_values_to_return_dataframe rejects with MergeError, blocking the
pair. A one-shot on-disk dedup cannot fix this since the duplicate is
regenerated in memory on restart.
Install an idempotent in-memory guard from the model module that wraps
set_initial_return_values, append_model_predictions and
attach_return_values_to_return_dataframe to keep date_pred unique, preferring a
real prediction over a zero/NaN placeholder and preserving NaT rows. Validated
against freqtradeorg/freqtrade:stable_freqai (2026.7) across all restart
geometries.
* fix(quickadapter): keep original row order in date_pred dedup
Sort the kept rows by their original position (_order) instead of by
date_pred (_dp). The date sort pushed interior NaT rows to the tail and could
change which rows tail(N) returns to the strategy, dropping a real candle and
injecting a placeholder. Ordering by original position preserves freqtrade
insertion order minus the removed duplicates; date_pred uniqueness is already
guaranteed by drop_duplicates.
* style(quickadapter): type the date_pred dedup wrappers
Annotate the three FreqaiDataDrawer wrappers to match the module conventions
and freqtrade signatures (data_drawer.py:286/343/413).
* docs(quickadapter): sharpen date_pred duplicate attribution in guard docstring
State that set_initial_return_values counts the overlap but trims new_pred by
position, so a crash-persisted or overlapping store breaks the contiguous
head-prefix assumption and reintroduces duplicate date_pred; drop the
inaccurate "normal restart geometry" framing.
Lead with the robust primary cause: a store already holding duplicate
date_pred rows survives set_initial_return_values unchanged (its pd.concat
preserves hist_preds) and trips the validate="m:1" merge. Demote the
positional-trim path to a secondary case and drop the inaccurate claim that it
needs non-monotonic dates: a gapped or tail-overlapping store triggers it with
strictly monotonic dates too (proven). Anchor the data_drawer.py source lines.
Move the freqtrade-bug rationale from the pure _dedupe helper (contract only:
unique non-NaT date_pred, most-informative row, NaT preserved, order kept) to
_install, where the patch rationale belongs, and condense it with source
anchors. Net -13 doc lines; matches the module terse-docstring style.
* fix(quickadapter): drop NaT date_pred and restore chronological dedup order
Two correctness defects in the date_pred dedup guard reached the validate="m:1"
merge in attach_return_values_to_return_dataframe (data_drawer.py:429-431):
- The early return only checked duplicate valid dates, so a frame with >=2 NaT
date_pred was returned unchanged; the merge then raised MergeError ("Merge
keys are not unique in right dataset") because pandas counts repeated null
keys as non-unique, even with no NaT on the left.
- Ordering the kept rows by original position (_order) left the frame
non-chronological, so the tail(N) trim in set_initial_return_values and
append_model_predictions could keep stale dates and evict recent ones.
Drop NaT date_pred rows (a NaT matches no candle -- freqtrade always assigns
real dates at set_initial_return_values, data_drawer.py:305 -- and only harms
the m:1 merge) and order survivors chronologically by normalized date_pred; the
most-informative row per timestamp is still kept. This supersedes the _order
sort introduced in f4b95a7b: removing the NaT rows makes that reorder
unnecessary and the tail chronologically correct.
Add scripts/historic_predictions_deduplicate.py to repair FreqAI
historic_predictions.pkl stores that a crash-recovery backfill can leave with
duplicate date_pred rows, which break the prediction merge for that pair.
- Keeps the most informative row per date_pred (informative cells, then non-null
count, then most-recent write); deduplicates only non-NaT keys.
- Dry-run by default; --apply writes atomically (temp + fsync + os.replace,
preserving mode and best-effort owner) after copying the original aside as
.original-<stamp>. Exits 1 when any target is skipped as unreadable.
- Repairs the sibling historic_predictions.backup.pkl as well.
- Portable single-file tool with a bundled test suite run via uv.
Jérôme Benoit [Wed, 5 Aug 2026 11:46:54 +0000 (13:46 +0200)]
style(optuna): apply ruff format to best-params I/O
Reflow the lock helper, save fchown block and legacy resolver wrapping
to satisfy ruff format at the default 88-column width, matching the rest
of both files (main was format-clean). Whitespace and redundant grouping
parentheses only; the AST is unchanged.
Jérôme Benoit [Tue, 4 Aug 2026 23:16:21 +0000 (01:16 +0200)]
fix(optuna): harden best-params I/O against deployment failure modes
Address three P2 review findings; the same latent behaviors were ported
from QuickAdapter to ReforceXY, so all fixes are applied to both.
- Cross-uid save (C2): make owner preservation best-effort. A non-root
process on a bind mount with a differently owned existing file lacks
CAP_CHOWN; the fchown now swallows PermissionError (logged at debug)
so the atomic write still completes, as the previous in-place write
did. fchmod and the successful-chown path are unchanged.
- Read-only mount (C3): a shared (read) lock no longer forces O_CREAT.
On a read-only mount where the lock file is absent, the load reads
without a lock; os.replace atomicity guarantees a consistent read.
Exclusive locks keep O_CREAT and still fail closed on read-only
filesystems. S_ISREG/O_NOFOLLOW/O_NONBLOCK guards are preserved.
- Legacy warm-start (C1): a base-only legacy best-params file is now
loaded when exactly one configured pair maps to that base (read in
place, no rename, read-only safe). Ambiguous cases (more than one pair
sharing a base) still warn and return None, preserving #180 safety.
The configured pair list is threaded into the QuickAdapter module-level
loader; a legacy payload still passes the same validation.
Mirror the rationale already documented on the shared QuickAdapter
helper: _quarantine_path appends the tag and timestamp after the
complete filename (extension included) so live-artefact globs never
match quarantined files. Docstring only; no behavior change.
The best-params quarantine in Utils re-inlined the timestamp + tie-break
path computation that the journal quarantine already implemented in
QuickAdapterRegressorV3, duplicating the algorithm across the two
modules.
- Add a shared, parameterized _optuna_quarantine_path(path, now, *, tag,
limit) in Utils (the module both call sites can import).
- Delegate from _quarantine_corrupt_optuna_best_params (Utils) and from
_optuna_quarantine_journal (regressor); each keeps its own domain
constants (journal vs best-params), logging and rename handling, so
quarantine names, log output and error semantics are unchanged.
- Remove the regressor's private _optuna_quarantine_path copy.
repositories / freqai-strategies.git / log
