]> Piment Noir Git Repositories - freqai-strategies.git/commit
feat(scripts): add historic-predictions deduplication tool
authorJérôme Benoit <jerome.benoit@piment-noir.org>
Wed, 12 Aug 2026 17:00:57 +0000 (19:00 +0200)
committerGitHub <noreply@github.com>
Wed, 12 Aug 2026 17:00:57 +0000 (19:00 +0200)
commitc20d23a2cd3dab2c9a8a77b4e8b66feaad9eb35b
tree332fdaf157e1c406dcc500747793652d5f28936a
parent5bdd8f0fb2ba12c4bf41f2fc822a3e2ab9fb8da3
feat(scripts): add historic-predictions deduplication tool

Add scripts/historic_predictions_deduplicate.py to repair FreqAI
historic_predictions.pkl stores that a crash-recovery backfill can leave with
duplicate date_pred rows, which break the prediction merge for that pair.

- Keeps the most informative row per date_pred (informative cells, then non-null
  count, then most-recent write); deduplicates only non-NaT keys.
- Dry-run by default; --apply writes atomically (temp + fsync + os.replace,
  preserving mode and best-effort owner) after copying the original aside as
  .original-<stamp>. Exits 1 when any target is skipped as unreadable.
- Repairs the sibling historic_predictions.backup.pkl as well.
- Portable single-file tool with a bundled test suite run via uv.
README.md
scripts/historic_predictions_deduplicate.py [new file with mode: 0755]
scripts/pyproject.toml [new file with mode: 0644]
scripts/tests/test_historic_predictions_deduplicate.py [new file with mode: 0755]