fix(quickadapter): guard against freqtrade duplicate date_pred regeneration (#202)
* revert(scripts): drop historic-predictions dedup tool
Reverts the squash-merged PR #201 (commit
c20d23a2). A one-shot on-disk
deduplication cannot durably fix the issue: freqtrade 2026.7 regenerates the
duplicate date_pred in memory on restart. It is superseded by an in-memory
guard in the QuickAdapter model (next commit).
* fix(quickadapter): guard against freqtrade duplicate date_pred regeneration
freqtrade 2026.7 FreqaiDataDrawer.set_initial_return_values trims the new
prediction window by position (new_pred.iloc[len(common_dates):]) instead of by
date_pred value. When the persisted history overlaps the tail of the analysis
window (the normal restart geometry), the trim misaligns and the following
pd.concat re-creates duplicate date_pred rows, which the validate="m:1" merge in
attach_return_values_to_return_dataframe rejects with MergeError, blocking the
pair. A one-shot on-disk dedup cannot fix this since the duplicate is
regenerated in memory on restart.
Install an idempotent in-memory guard from the model module that wraps
set_initial_return_values, append_model_predictions and
attach_return_values_to_return_dataframe to keep date_pred unique, preferring a
real prediction over a zero/NaN placeholder and preserving NaT rows. Validated
against freqtradeorg/freqtrade:stable_freqai (2026.7) across all restart
geometries.
* fix(quickadapter): keep original row order in date_pred dedup
Sort the kept rows by their original position (_order) instead of by
date_pred (_dp). The date sort pushed interior NaT rows to the tail and could
change which rows tail(N) returns to the strategy, dropping a real candle and
injecting a placeholder. Ordering by original position preserves freqtrade
insertion order minus the removed duplicates; date_pred uniqueness is already
guaranteed by drop_duplicates.
* style(quickadapter): type the date_pred dedup wrappers
Annotate the three FreqaiDataDrawer wrappers to match the module conventions
and freqtrade signatures (data_drawer.py:286/343/413).
* docs(quickadapter): sharpen date_pred duplicate attribution in guard docstring
State that set_initial_return_values counts the overlap but trims new_pred by
position, so a crash-persisted or overlapping store breaks the contiguous
head-prefix assumption and reintroduces duplicate date_pred; drop the
inaccurate "normal restart geometry" framing.
* docs(quickadapter): correct date_pred duplicate attribution
Lead with the robust primary cause: a store already holding duplicate
date_pred rows survives set_initial_return_values unchanged (its pd.concat
preserves hist_preds) and trips the validate="m:1" merge. Demote the
positional-trim path to a secondary case and drop the inaccurate claim that it
needs non-monotonic dates: a gapped or tail-overlapping store triggers it with
strictly monotonic dates too (proven). Anchor the data_drawer.py source lines.
* docs(quickadapter): tighten date_pred dedup guard docstrings
Move the freqtrade-bug rationale from the pure _dedupe helper (contract only:
unique non-NaT date_pred, most-informative row, NaT preserved, order kept) to
_install, where the patch rationale belongs, and condense it with source
anchors. Net -13 doc lines; matches the module terse-docstring style.
* fix(quickadapter): drop NaT date_pred and restore chronological dedup order
Two correctness defects in the date_pred dedup guard reached the validate="m:1"
merge in attach_return_values_to_return_dataframe (data_drawer.py:429-431):
- The early return only checked duplicate valid dates, so a frame with >=2 NaT
date_pred was returned unchanged; the merge then raised MergeError ("Merge
keys are not unique in right dataset") because pandas counts repeated null
keys as non-unique, even with no NaT on the left.
- Ordering the kept rows by original position (_order) left the frame
non-chronological, so the tail(N) trim in set_initial_return_values and
append_model_predictions could keep stale dates and evict recent ones.
Drop NaT date_pred rows (a NaT matches no candle -- freqtrade always assigns
real dates at set_initial_return_values, data_drawer.py:305 -- and only harms
the m:1 merge) and order survivors chronologically by normalized date_pred; the
most-informative row per timestamp is still kept. This supersedes the _order
sort introduced in
f4b95a7b: removing the NaT rows makes that reorder
unnecessary and the tail chronologically correct.