Jérôme Benoit [Mon, 22 Jun 2026 12:31:01 +0000 (14:31 +0200)]
fix(quickadapter): quarantine corrupt optuna journal log and recover (#102)
When `JournalFileBackend`'s replay encounters a corrupt journal record, optuna raises during `JournalStorage.__init__` (Class A/B: immediate `KeyError`/`json.JSONDecodeError`) or defers the raise to the next `_sync_with_backend` (Class C1/C2/C3: truncated tail / malformed last line / bare trailing newline at EOF). Both paths previously caused HPO for the affected pair to be silently skipped on every fit cycle.
Wrap `JournalStorage` construction in a narrow `try / except (KeyError, ValueError, json.JSONDecodeError)`, atomically rename the corrupt log aside as `optuna-<COIN>.log.corrupt-<UTC_µs>`, log a `WARNING`, and retry once on a fresh path. A bounded O(1) tail probe runs BEFORE construction to detect deferred-raise EOF corruption. `OSError` is intentionally excluded so filesystem failures stay operator-actionable.
The recovery follows RocksDB's documented WAL recovery philosophy (quarantine + restart), preserves forensic evidence (rename, no `unlink`), and keeps the live-journal glob `optuna-*.log` from matching quarantined artefacts.
Jérôme Benoit [Mon, 22 Jun 2026 10:09:19 +0000 (12:09 +0200)]
refactor(quickadapter): dispatch optuna_create_sampler via match + assert_never (#103)
Follow-up to #101. Converts the `optuna_create_sampler` if/elif/else dispatch chain to a `match`/`case` statement using value patterns (`QuickAdapterRegressorV3._OPTUNA_SAMPLERS.<name>`) -- the per-field singleton `Literal[...]` typing introduced by #101 unlocks pyright/mypy exhaustiveness narrowing, so the final `case _: assert_never(sampler)` type-checks as `Never` and catches any future extension of `OptunaSampler` that forgets to add a corresponding match branch.
The `case None:` branch is structurally required (not stylistic): without it, after the four `Literal[...]` value patterns, pyright/mypy would narrow `sampler` to `None`, not `Never`, and `assert_never(sampler)` would fail to type-check. Its presence is what makes the "5th sampler addition -> type error at assert_never" claim effective. The error message inside `case None:` is preserved verbatim from the prior `else: raise ValueError(...)` for user-facing wire compatibility.
Behavior delta is confined to non-`Literal` string inputs (e.g. dynamically-injected `"garbage"`): prior code raised `ValueError` with the supported-values list; new code raises `AssertionError` with the standard `typing.assert_never` message. In practice this path is unreachable from every current in-repo call site -- the sole caller `optuna_create_study` already validates `sampler not in samplers` and raises `ValueError` with the supported-values list before dispatching, so misconfigured `_optuna_config["sampler"]` values surface the old error shape from the upstream gate. The new `AssertionError` would only manifest via a hypothetical future caller that bypasses `optuna_create_study`, which does not exist today.
Pattern parity: the same `assert_never` exhaustiveness idiom is already used in this file for `support_policy` dispatch; value-pattern syntax matches the 8 call sites migrated by #101.
Closes the deferred follow-up from PR #101 (issue #88).
Jérôme Benoit [Mon, 22 Jun 2026 09:32:44 +0000 (11:32 +0200)]
refactor(quickadapter): consolidate Optuna sampler tuples to NamedTuple (#101)
PR #81 consolidated `_OPTUNA_NAMESPACES` to `Utils._OptunaNamespaces` (a `NamedTuple` with per-field singleton `Literal` types). Propagate the same pattern to the three sibling sampler tuples in `QuickAdapterRegressorV3.py`:
Each new `_Optuna*Samplers` NamedTuple class lives at module level immediately before `class QuickAdapterRegressorV3`, matching the `_OptunaNamespaces` adjacency in `Utils.py`. Per-field types are singleton `Literal["..."]` (not the `OptunaSampler` union) to unlock pyright/mypy narrowing for a future `assert_never` migration. The class-private `_OPTUNA_*_SAMPLERS` instance constants remain class-attributes of `QuickAdapterRegressorV3` to preserve every consumer's `QuickAdapterRegressorV3._OPTUNA_*` access pattern.
Each `_Optuna*Samplers` redeclares its field defaults (per-class `Literal["tpe"] = "tpe"` etc.), replacing the prior tuple-slice derivation `_OPTUNA_HPO_SAMPLERS = _OPTUNA_SAMPLERS[:2]` and the 6-line custom reordering of `_OPTUNA_LABEL_SAMPLERS`. Trade-off: minor string-literal duplication across 3 classes is accepted for harmonization with the `_OptunaNamespaces` template; the single source of truth for the 4 valid tokens stays the `OptunaSampler = Literal[...]` alias unchanged at module level.
Migrates 8 positional-indexing call sites (`_OPTUNA_*_SAMPLERS[N] # "name"`) to attribute access (`.<name>`); drops the 6 surviving inline annotations at the call sites (2 sites carry no annotation pre-refactor; the multi-line site collapses 3 source lines into 1) plus the 4 analogous annotations inside the deleted 6-line `_OPTUNA_LABEL_SAMPLERS` custom-ordering block. The `_OPTUNA_HPO_SAMPLERS_SET` and `_OPTUNA_LABEL_SAMPLERS_SET` frozenset companions are kept unchanged (still used in O(1) membership testing at `optuna_samplers_by_namespace`); their type annotation `Final[frozenset[OptunaSampler]]` is preserved.
Non-migration sites confirmed unchanged: the frozenset companion construction, the `', '.join` error-message iteration over `_OPTUNA_SAMPLERS`, and the `_SET` membership references. All iterate over the NamedTuple instance and produce byte-identical output.
Add `NamedTuple` to the existing `from typing import (...)` block (alphabetical, between `Literal` and `Optional`). `assert_never` already imported, kept in anticipation of the deferred follow-up.
Per-field singleton `Literal[...]` unlocks `assert_never` exhaustiveness on the `optuna_create_sampler` dispatch chain -- left to a follow-up PR since the migration (`else: raise ValueError(...)` becoming `assert_never(sampler)`) changes the user-facing error contract on the unreachable branch.
The AGENTS.md *Canonical defaults* principle and the README documented enum order are encoded in the field-declaration order of each `_Optuna*Samplers`. NamedTuple remains a `tuple` subclass, so `[0]` indexing, `len(...)`, `frozenset(...)`, `', '.join(...)`, and iteration all keep their existing semantics. No behavior change.
Reviewed by two pre-implementation design passes (3-oracle on v1; Metis + Momus + meta-Oracle on v2) and a 3-oracle review on the live PR, each citing upstream evidence from `freqtrade/freqai/` confirming no external consumer.
Follow-up from PR #81 review (Oracle harmonization dimension).
Prose state-form (`Utils.py`, `QuickAdapterRegressorV3.py`):
- `_normalize_label_column_name` docstring: ``Raises ValueError when
the result contains `&` or `%` after sigil strip``.
- Deprecated-config-key warning aligns with the sibling pattern at
the adjacent branch: ``f"{old_path!r} is deprecated, use
{new_path!r} instead"``.
- `sanitize_and_renormalize` docstring states ``mean(out) == 1`` as the
rescale invariant.
- Optuna-label throttle log reads ``callback throttled,
{N} candles until next emission``.
- Fit-live-predictions warmup log reads ``Fit live predictions not
warmed up: {N} candles until warmup completion``.
Docstrings on validator/composer helpers (3 functions lacking a
docstring at HEAD):
- `_apply_support_policy`: documents the ``policy='raise'`` /
``policy='fallback'`` dispatch contract.
- `_compose_train_weights_with_support`: documents the support-gating
flow (None-label-weights branch routes through
``_apply_support_policy`` when ``strategy != 'none'``; main branch
composes and validates the summary against three thresholds).
- `_validate_optuna_label_best_params`: enumerates the rejection
paths and the optional ``expected_selection_metadata`` drift gate.
Harmonization (post-merge carry-over):
- `LABEL_WEIGHT_SUFFIX` renamed to `_LABEL_WEIGHT_SUFFIX`
(no external consumer; symmetric with
`_LABEL_KNOWN_AT_LOOKAHEAD_SUFFIX`).
- `safe_distribution_fit` call-site contexts harmonized with the
PR #97 / PR #99 ``[<pair>] <event>`` convention:
`f"[{pair}] di_values_weibull_fit"` and
`f"[{pair}] label_norm_fit:{label_col}"`.
Python idioms:
- `_adapt_label_generator` rejects any 3rd positional parameter
whose name is not ``logger``, regardless of whether the parameter
is required or has a default. A defaulted non-``logger`` 3rd
positional raises ``ValueError`` at registration. The 3-arg
pass-through is reached only when ``positional[2].name == "logger"``.
- `_build_sample_weight_inputs` switches the two `logger.debug`
calls to lazy ``%s`` formatting so the f-string body is not
materialized when the debug level is disabled.
Consolidates SAFE + DRY + COSMETIC findings from the 4-axis review of
`add1fb7..d6a718f` (PRs #90, #94, #95, #96, #97 + 2 style + d6a718f).
Migration (PR #94 carry-over to `QuickAdapterV3.py`):
- `_TRADE_DIRECTIONS_SET` and `_ORDER_TYPES_SET` are
`Final[frozenset[T]]` constants adjacent to the canonical
`_TRADE_DIRECTIONS` / `_ORDER_TYPES` tuples; the strategy reads
them directly at all 3 call sites. The
`@staticmethod @lru_cache(maxsize=None)` set-views are absent at
HEAD.
Context-prefix harmonization (PR #97 carry-over):
- The 2 `sanitize_and_renormalize` calls in
`QuickAdapterRegressorV3._apply_pipelines` carry the `[<pair>]`
prefix (`f"[{pair}] post_feature_pipeline:train"` / `:test`),
matching the PR #97 convention at the 4 `compose_sample_weights`
call sites.
Alias removal (PR #96 carry-over):
- `Utils.label_known_at_column_name` is absent at HEAD. The
underlying column has zero external callers in the repo, and
PR #96 shifted semantics (absolute index -> per-row offset);
no back-compat alias is warranted.
DRY (`d6a718f` carry-over):
- The 8-tuple swap blocks in `_make_train_test_split_datasets`
and `_make_timeseries_split_datasets` use pythonic parallel
pair-swap (`a, b = b, a` per slot pair); 4 lines per slot pair
instead of the 18-line 8-tuple parallel assignment.
Dead constant removal:
- `QuickAdapterRegressorV3._AGGREGATE_DISTANCE_METRICS_SET` is
absent at HEAD; the constant has no caller. The
`_CLUSTER_DENSITY_DISTANCE_METRICS_SET` block comment lists the
7 aggregate metrics inline for reference (`harmonic_mean`,
`geometric_mean`, `arithmetic_mean`, `quadratic_mean`,
`cubic_mean`, `power_mean`, `weighted_sum`).
Cosmetic:
- `_LABEL_KNOWN_AT_LOOKAHEAD_SUFFIX` placement is adjacent to
`LABEL_WEIGHT_SUFFIX` (both are label-aux column suffixes).
- `_known_at_lookahead` returns `int64` on both single-series and
multi-series paths (symmetric dtype contract).
- Schema-version reset log: `f"v{existing_schema_version!r}"`
renders unambiguously for non-int corrupt values (booleans,
strings).
Jérôme Benoit [Mon, 22 Jun 2026 01:54:59 +0000 (03:54 +0200)]
fix(quickadapter): route reversed train weights through support_policy (#98)
PR #85 added `_compose_train_weights_with_support` (gates training-set weights through `support_policy`) and `_compose_eval_weights` (eval-side, deliberately bypasses `support_policy`). The `reverse_train_test_order` path in `_make_train_test_split_datasets` and `_make_timeseries_split_datasets` swapped slices AT THE FINAL `build_data_dictionary` call -- AFTER weight composition -- so the actual-train slot received weights composed by `_compose_eval_weights` (silent bypass), and the actual-test slot received weights composed by `_compose_train_weights_with_support` (wrong direction, typically a no-op under `support_policy='fallback'` default).
Reachable only under `causal_mode=false` (deprecated; acausal baselines only) since `causal_mode=true` rejects `reverse_train_test_order=true` upfront.
Fix: perform the train/test slice swap BEFORE weight composition so the `train_*` and `test_*` identifiers map to their actual training roles throughout. Both call sites converge to a single `dk.build_data_dictionary` return; context strings in `support_policy` log/raise messages now reflect the true train/test role.
Add an upfront `ValueError` in `_make_train_test_split_datasets` when `test_size=0` AND `reverse_train_test_order=True`, mirroring the existing `causal_mode`/reverse rejection pattern. The `timeseries_split` path already rejects `test_size < 1` upstream of the swap.
Behavior change in the deprecated path: `support_policy='raise'` now correctly raises on actual-train insufficient support; `support_policy='fallback'` now correctly warns.
Reviewed by three parallel Oracle passes (math + algorithmics + scope/reachability; Python state-of-the-art + harmonization + implementation elegance; documentation + terminology + completeness) at design stage and again post-implementation, each citing upstream evidence from `freqtrade/freqai/`.
Follow-up from PR #80 review, deferred during PR #90.
Jérôme Benoit [Mon, 22 Jun 2026 01:34:16 +0000 (03:34 +0200)]
fix(quickadapter): prefix `compose_sample_weights` and `sanitize_and_renormalize` log entries with caller context (#97)
`compose_sample_weights` and `sanitize_and_renormalize` accept a
required keyword-only `context: str` parameter; every warning, error,
and inner `sanitize_and_renormalize` call uses `context` as its sole
prefix.
- `compose_sample_weights(..., *, logger, context, on_collapse=...)`:
`context: str` keyword-only, required. The 4 internal
warnings/errors (shape-mismatch `ValueError`, all-dropped
`LabelWeightSupportError`, sparse-mass warning, collapse-on-
survivors `LabelWeightSupportError` and fallback warning) prefix
with `{context}:`. Internal `sanitize_and_renormalize` calls
compose sub-contexts: `{context}:base_only`,
`{context}:label_weighted`, `{context}:base_fallback`. The sparse-
mass message reads `sparse weighting mass`, accurate for both
train and eval paths.
- `sanitize_and_renormalize(..., *, logger=None, context: str)`:
`context` keyword-only, required. The 5 warnings/errors
(`drop_mask` shape `ValueError`, `drop_mask` dtype `ValueError`,
rescale-overflow warning, weights-collapsed warning,
mask-covers-all warning) prefix with `{context}:`. The redundant
`(context=%s, ...)` subfield is dropped.
- 5 `compose_sample_weights` call sites in
`QuickAdapterRegressorV3.py` pass `context=context`. The 2
external `sanitize_and_renormalize` call sites
(`post_feature_pipeline:train`, `post_feature_pipeline:test`)
already pass `context=`.
Log format goes from
`compose_sample_weights: sparse training mass (59/2603 rows ...)`
to
`[ETH/USDT] train_test_split:train: sparse weighting mass (59/2603 rows ...)`.
The pair, split method, and train/eval side are traceable in the
log line.
Jérôme Benoit [Mon, 22 Jun 2026 01:21:28 +0000 (03:21 +0200)]
refactor(quickadapter): migrate _*_set() lru_cache family to Final[frozenset] (#94)
Each `@staticmethod @lru_cache(maxsize=None) _X_set()` family member
maps to a class-level `_X_SET: Final[frozenset[...]]` adjacent to its
backing tuple. Derived members express their content through set
algebra over the new `_*_SET` constants.
Call sites at `(cls|QuickAdapterRegressorV3)._X_set()` resolve to the
matching `_X_SET` reference. `optuna_samplers_by_namespace` returns
`tuple[frozenset[OptunaSampler], OptunaSampler]` to match the new
constant types. The surviving `@lru_cache(maxsize=8)` on
`optuna_samplers_by_namespace` falls outside the `_*_set` family and
stays in place.
Jérôme Benoit [Mon, 22 Jun 2026 01:20:11 +0000 (03:20 +0200)]
refactor(quickadapter): rename known_at_index to known_at_lookahead (#96)
PR #95 retained the historical column name `<label>_known_at_index` for what is now a per-row label lookahead in candles, to keep that hotfix strictly minimal. This PR converges the column suffix, the helper, the dataclass field, the static method, and the per-call-site locals onto `_known_at_lookahead`, with a retro-compat alias on the only externally-named public helper (`label_known_at_column_name = label_known_at_lookahead_column_name`).
The auxiliary `<label>_known_at_*` column is regenerated on every training run inside `set_freqai_targets`; FreqAI persists only the fitted model and `extra_returns_per_train`, never auxiliary dataframe columns -- the rename invalidates no on-disk artifact.
Reviewed by three parallel Oracle passes (math + claims-coherence; Python state-of-the-art + harmonization; documentation + terminology + PR-description coherence), each citing upstream evidence from `freqtrade/freqai/freqai_interface.py`, `data_kitchen.py`, and `data_drawer.py`. Consensus fixes were applied: README `causal_mode` formula symbol bound to the column token (`row-wise max(<label>_known_at_lookahead)`) to colocate definition with usage.
The two causal-guard local variable pairs were also harmonized to the local `train_<noun>` family (`train_known_at_lookahead`, `train_known_at_position`) used by the surrounding `_make_*_datasets` methods.
Jérôme Benoit [Mon, 22 Jun 2026 01:02:56 +0000 (03:02 +0200)]
fix(quickadapter): use slice-invariant lookahead for causal guard (#95)
* fix(quickadapter): use slice-invariant lookahead for causal guard
PR #78 stored '<label>_known_at_index' as 'arange(len) + horizon +
kernel_half_width' -- absolute positions in the dataframe passed to
'set_freqai_targets'. freqtrade's 'dk.slice_dataframe' (a '.loc' filter)
runs AFTER 'set_freqai_targets' and drops warmup rows but preserves
column values, so those pre-slice positions survived into the post-slice
'unfiltered_df'. The causal guard then compared them against
'first_test_position' derived from 'np.arange(len(unfiltered_df))' --
local post-slice positions in a different coordinate system. The unit
mismatch wiped out most or all training rows on every pair.
Production crash on 2026-06-22 (XRP/USD): "removed 2621
causal-unsafe train rows" followed by "causal guard removed all
train rows, skipping".
Fix: the column now stores a per-row label lookahead (in candles),
invariant under 'dk.slice_dataframe'. Consumers combine the row's
local position with the lookahead to recover the local known-at
position before comparing to 'first_test_position'. Column name
'<label>_known_at_index' is retained for this hotfix; a rename to
'<label>_known_at_lookahead' (with rétro-compatible alias) is left
to a follow-up PR per AGENTS.md 'small, verifiable changes'.
Touches:
- Utils.py: producer rewritten to store a constant per-row lookahead;
'LabelData' and 'label_known_at_column_name' docstrings document the
new contract; '_LABEL_KNOWN_AT_SUFFIX' carries an inline disambiguation.
- QuickAdapterV3.py: smoothing-lookahead advance comment harmonized to
the canonical 'per-row label lookahead (in candles)' phrasing.
- QuickAdapterRegressorV3.py: '_known_at_index' docstring rewritten;
'train_test_split' and 'timeseries_split' causal-mode branches add
'train_positions + delta' before the '< first_test_position' check;
'timeseries_split' hoists 'train_positions' for symmetry with
'train_test_split'.
- README.md: 'causal_mode' tunable description reflects the new
comparison semantic.
Reviewed by three parallel Oracle passes (math/algo/scope,
Python state-of-the-art / harmonization, documentation /
terminology / concision) with cross-validation; one false alarm
on a missing position-only fallback in 'timeseries_split' was
resolved by confirming 'TimeSeriesSplit.gap' enforces the
chronological purge at the sklearn layer.
* docs(quickadapter): shrink _known_at_index docstring to LabelData pointer
Per multi-oracle PR #95 review (Oracle 3 §8.1): paragraph 1 of
_known_at_index duplicated the slice-invariance rationale already
canonical on LabelData.known_at_index. Replace with a thin pointer per
AGENTS.md *No duplication: maintain single authoritative documentation
source; reference other sources rather than copying.*
Consolidates P1/P2 findings from `chatgpt-codex-connector` review
comments on PRs #78, #79, #80, #81, and PR #90.
Utils.py + label generation:
- `_generate_extrema_label` accepts `logger: Logger | None`; the
`LabelGenerator` type signature, `generate_label_data` dispatcher,
and `QuickAdapterV3.set_freqai_targets` caller propagate the logger.
`_generate_extrema_label` has no `F821 logger` undefined-name path.
- `register_label_generator` routes the input through
`_adapt_label_generator`. The adapter detects the canonical
`(dataframe, params, logger)` shape by a positional parameter named
`logger` at index 2 (with or without a default); other generators
with 2 required positional parameters are wrapped via
`functools.wraps` (preserves `__name__`/`__doc__`/`__wrapped__`) to
drop the logger argument at dispatch, with defaulted positionals
after index 1 left at their defaults. `ValueError` is raised at
registration for `*args`, `**kwargs`, keyword-only `logger`, fewer
than 2 required positionals, more than 3 required positionals, and 3
required positionals whose third name is not `logger`.
- `safe_divide` denominator zero-check uses exact-zero
(`denominator_arr != 0.0`); subnormal and satoshi-scale denominators
pass the gate, and non-finite division outputs coerce to `fallback`
via the post-division finite-mask.
Causal label split lookahead:
- `QuickAdapterV3.get_label_horizon_candles` recomputes the horizon
from the current `label_period_candles` (via
`get_label_period_candles`); init omits `label_horizon_candles`, so
HPO period updates propagate to the horizon. The regressor's
`_optuna_label_params` init likewise omits `label_horizon_candles`.
- `QuickAdapterV3.set_freqai_targets` advances `<label>_known_at_index`
by the smoothing kernel half-width after smoothing. The
`Utils.get_smoothing_kernel_half_width(config, *, series_length)`
helper reuses `get_odd_window`/`get_even_window`/`get_savgol_params`
shared with `smooth()` and dispatches on kernel routing:
- `filtfilt`-routed zero-phase kernels (members of
`SMOOTHING_KERNELS`: `gaussian`, `kaiser`, `kaiser_bessel_derived`,
`triang`): half-width `effective_window - 1` (forward+backward
pass extends dependency to the full filter length on both sides).
- Single-pass centered windows (`smm`, `sma`, `savgol`): half-width
`effective_window // 2`.
- `gaussian_filter1d`: `int(4.0 * sigma + 0.5)` matching
`scipy.ndimage` default truncation.
The helper returns 0 when `smooth()` itself short-circuits:
`series_length < max(window_candles, 3)` (top-level no-op) and, for
the filtfilt/savgol routes, `series_length < effective_window`
(downstream short-series no-op in `zero_phase_filter` /
`savgol_filter`). The `series_length` parameter is keyword-only and
required.
Label-weighting support policy:
- `QuickAdapterRegressorV3._compose_train_weights_with_support` routes
the zero-pivot case (label-weighting strategy configured but no
label weights available) through `_apply_support_policy`; the
support policy governs the contract: `raise` raises, `fallback`
warns.
Label Optuna selection hardening:
- `_OPTUNA_LABEL_SELECTION_SCHEMA_VERSION` is `2`, co-located with
`_OPTUNA_LABEL_BEST_PARAMS_SCHEMA_VERSION` in `Utils`. The two
constants are independent: wire format and selection algorithm
carry separate version axes.
- `_optuna_label_selection_metadata` rejects non-finite `label_weights`
/ `label_p_order` with `ValueError`; downstream dict equality on the
selection_metadata is NaN-safe.
- `_validate_optuna_label_best_params` accepts an
`expected_selection_metadata` keyword. It rejects files that are
not a `{schema_version, params, selection_metadata}` dict, files
with mismatched `schema_version`, files missing or with an invalid
`selection_metadata.schema_version`, and -- when
`expected_selection_metadata` is provided -- files whose stored
`selection_metadata` differs from the caller's current view. Legacy
unversioned best-params files (no `schema_version`) are rejected
outright. `QuickAdapterRegressorV3.optuna_load_best_params` passes
its `_optuna_label_selection_metadata()` view for the label
namespace; `QuickAdapterV3.optuna_load_best_params` omits the
keyword, since the strategy reads only `label_period_candles`,
`label_horizon_candles`, and `label_natr_multiplier` from the
best-params file.
- The study schema-migration branch resets the Optuna label study
whenever `existing_schema_version` is not an `int` (rejects `bool`
and other types) or differs from `target_version`; an unversioned
study (legacy pre-metadata) and a version-mismatched study are
treated identically. Trials selected under a different metric
whitelist or weighting scheme cannot be reused.
- `_optuna_label_selection_metadata` includes `label_weights` and
`label_p_order`; `_calculate_distances` consumes both, so the
idempotent `set_user_attr` write detects drift on those tunables.
- `_calculate_distances` validates `label_weights` length against the
original objective count (raises on mismatch) only on the slicing
path (when `objective_indices is not None` and the count differs).
The sliced vector falls back to uniform weights only when
`np.all(sliced == 0.0)` (the user's positive weights all on
dropped objectives); slices containing negative or non-finite
values flow through to `_validate_label_weights`.
- `_validate_label_selection_metric` accepts an `aggregate_allowed`
parameter; cluster/density category callers pass `False`,
restricting the valid set via the cached
`_cluster_density_distance_metrics_set()` classmethod
(SciPy-compatible non-probability metrics). The aggregate metrics
(`harmonic_mean`, `geometric_mean`, `arithmetic_mean`,
`quadratic_mean`, `cubic_mean`, `power_mean`, `weighted_sum`) live
in the cached `_aggregate_distance_metrics_set()` classmethod
derived by set-algebra from
`_distance_metrics_set() - _scipy_metrics_set() -
_probability_distance_metrics_set()`. README cluster/density metric
rows list the SciPy-compatible set. `compromise_programming`/
`topsis` accepts the full aggregate set via
`_calculate_trial_distance_to_ideal`.
Out of scope:
- Two PR #80 findings: `reverse_train_test_order` support-policy
routing; post-`feature_pipeline.fit_transform` support recheck.
Jérôme Benoit [Sun, 21 Jun 2026 21:45:02 +0000 (23:45 +0200)]
fix(quickadapter): stabilize label optuna selection (#81)
Multi-objective `label` Pareto best-trial selection in the
QuickAdapter regressor.
- Probability-style metrics (`jensenshannon`, `hellinger`,
`shellinger`) rejected for `label_distance_metric` /
`label_cluster_metric` / `label_density_metric` — Pareto objective
matrices are unbounded floats, not probability vectors. Invalid
values fall back to `euclidean` with a warning.
- Constant objective dimensions dropped before trial-distance
computation: a constant objective dimension is non-informative and
would bias the geometry. Tolerance at
`_NON_CONSTANT_OBJECTIVE_ATOL: Final[float] = 1e-8`.
- User-supplied `label_weights` matching the original objective count
slice to align with the non-constant subset; mismatched sizes flow
to `_validate_label_weights(mode="raise")`.
- Deterministic best-trial tie-break by `(distance, trial.number)`,
independent of `study.best_trials` ordering.
- All-constant Pareto front falls back to the lowest `trial.number`.
- Persisted Optuna `label` study user-attr `selection_metadata` nests
`schema_version`
(`_OPTUNA_LABEL_SELECTION_SCHEMA_VERSION: Final[int] = 1`) and
`method_config`. Studies without a recorded `schema_version` are
tagged at the current version on next `optuna_create_study` (trials
preserved); studies recording a different version are reset. The
`selection_metadata` write is idempotent: skipped when unchanged,
warned on diff.
- `OptunaNamespace` Literal and `_OPTUNA_NAMESPACES` (a `NamedTuple`
of `hp` and `label` with per-field literal types) live in `Utils`,
with `.hp` / `.label` accessors at all call sites
(`QuickAdapterRegressorV3`, `QuickAdapterV3`). No per-class tuple
alias, no inline `# "hp"` / `# "label"` annotations.
Jérôme Benoit [Sun, 21 Jun 2026 20:37:10 +0000 (22:37 +0200)]
fix(quickadapter): restore _format_collection ordering before register decorators
Rebase of the label-weight-support-policy PR onto main (atop the style line-wrap commit) reapplied the _format_collection hunk at the wrong location, placing the helper AFTER its @_format_value.register(list|tuple|set|dict|np.ndarray) decorators. Move the definition back to its canonical position above the dispatchers.
Jérôme Benoit [Sun, 21 Jun 2026 20:26:45 +0000 (22:26 +0200)]
feat(quickadapter): add label weight support policy (#85)
Per-row sample-weight composition decoupled from the final per-split
compose. Causal-guard filtering operates on raw base/label weights.
Configurable support thresholds gate the train-weight compose.
- `SampleWeightInputs` dataclass carries `(base, label,
label_weighting_config)` from `_build_sample_weight_inputs`;
`compose_sample_weights(base, label)` runs AFTER
`train_test_split`/`TimeSeriesSplit` AND AFTER causal guards.
`__post_init__` validates 1-D shape, base/label shape parity,
required `label_weighting_config` keys, and `support_policy` enum
membership.
- Thresholds in `freqai.label_weighting`:
`min_pivot_equivalent_count` (default 3),
`min_positive_label_weight_fraction` (default 0.01),
`min_effective_sample_size` (default 3.0; Kish ESS).
- `support_policy: enum {fallback, raise}` (default `fallback`)
drives the failure mode when any threshold trips. Eval (test/val)
weights bypass this policy by design.
- `compose_sample_weights` takes `on_collapse: Literal["raise",
"fallback"]` (default `"raise"`); the train path lets collapse raise
through `LabelWeightSupportError` so `support_policy` catches it,
the eval path passes `"fallback"` to preserve the label-derived
drop mask.
- `_apply_support_policy` (typed `policy: LabelWeightSupportPolicy`)
dispatches `fallback` and `raise` branches via `match`/`case` with
`assert_never` exhaustiveness; `compose_sample_weights` applies the
same pattern on `on_collapse`.
- `_shuffle_split_rows` 4-tuple shuffler covers label weights.
- `_filter_train_by_mask` accepts optional `train_label_weights` for
uniform causal-guard filtering across base+label weight arrays.
- `LabelWeightSupportSummary`, `_effective_sample_size` (Kish ESS),
and `summarize_label_weight_support` documented as operational spec.
README documents the four `label_weighting` rows;
`config-template.json` records the `fallback` default.
Shared finite-sample, guarded distribution-fit, safe divide/log-ratio,
and sigmoid-domain helpers. Log/division feature paths route through
the helpers; distribution fits guard empty, non-finite, and constant
samples.
- `Utils.py` helpers: `FiniteSample` dataclass with `finite_sample`;
`safe_distribution_fit` (documented fallback-length contract);
`safe_divide`; `safe_log_ratio`.
- `nan_average` finite/zero-weight guards; documented divergence from
`np.nanmean` (strips +/-inf as well as NaN; bounded for current
callers).
- `_clip_sigmoid_domain` in `LabelTransformer.py` guards
`sp.special.logit` against values outside the open `(-1, 1)` domain
during `sigmoid` inverse normalization.
- `feature_engineering_expand_basic` and Utils log/divide sites
(`top_log_return`, `bottom_log_return`, `price_retracement_percent`,
`ewo` normalize, `zigzag` log prices, KC/BB/VWAP widths) route
through the safe helpers.
- DI Weibull and label `norm` fits in `fit_live_predictions` use
`safe_distribution_fit`; DI cutoff fallback at
`_DI_CUTOFF_DEFAULT: Final[float] = 2.0`.
Jérôme Benoit [Sun, 21 Jun 2026 18:01:23 +0000 (20:01 +0200)]
feat(quickadapter)!: add causal label split foundation (#78)
Causal split guards on QuickAdapter training. Default causal mode
rejects `data_split_parameters.shuffle=true`,
`feature_parameters.shuffle_after_split=true`, and
`feature_parameters.reverse_train_test_order=true`.
- `feature_parameters.causal_mode` (default `true`): guard toggle.
`false` is deprecated.
- `feature_parameters.label_horizon_candles` (default
`label_period_candles`): candles after a label row before its label
is considered known by causal split guards. Fallback chain
`label_horizon_candles` -> `label_period_candles` -> `1`.
- `<label>_known_at_index` columns expose `LabelData.known_at_index`
per-row; multi-label boundary via element-wise max across present
columns.
- `timeseries_split` `gap` auto-set from `label_horizon_candles` under
causal mode; explicit `gap < label_horizon_candles` rejected.
- Persisted Optuna `label` best-params JSON has shape
`{schema_version, params}`
(`_OPTUNA_LABEL_BEST_PARAMS_SCHEMA_VERSION = 2`). Unversioned files
identified by shape; version-mismatched files emit distinct
"missing" vs "incompatible" warnings.
- `_label_aux_column_name` shared sigil-stripping helper backs
`label_weight_column_name` and `label_known_at_column_name`;
uniform collision guard against `&`/`%` and empty stem.
- `QuickAdapterRegressorV3.version = 3.12.0`.
BREAKING CHANGE: `feature_parameters.causal_mode` defaults to `true`.
Configs with `data_split_parameters.shuffle=true`,
`feature_parameters.shuffle_after_split=true`, or
`feature_parameters.reverse_train_test_order=true` raise at training
time.
Add a fourth off-pivot weighting mode that superposes the epsilon
floor and the gaussian bumps additively, and fix a related defect in
_scatter_weights that allowed pivot rows to sit *below* the off-pivot
field whenever that field at the pivot's index could legitimately
exceed the pivot's raw weight.
with phi = eps * B(W) (B mean or median, eps in [0, 1]; phi = 0 on
empty pivots or non-finite baseline), and per-pivot sigma_p from
_compute_pivot_sigmas (fixed or k-NN). The combined formulation reuses
both existing closed forms verbatim:
Bound: phi <= f(i) <= phi + max_p w_p. The new mode reduces to pure
gaussian when eps = 0 (bit-identical). The reduction is a per-row max
over per-pivot Gaussian bumps; phi is the epsilon floor.
Before this change _scatter_weights wrote out[p] = w_p unconditionally,
so a pivot whose raw weight was below the off-pivot field at its
index appeared as a sharp dip relative to its neighbors. Two
manifestations of the same defect class:
- 'epsilon' / 'epsilon_gaussian' (sub-floor): a pivot with
w_p < phi (e.g. W = (0.001, 1.0, 1.0) with eps = 0.5 and
B = median, phi = 0.5) sat at 0.001 while neighbor rows sat at phi.
- 'gaussian' / 'epsilon_gaussian' (sub-bump): a weak pivot with a
strong neighbor (e.g. W = (0.001, 1.0) at indices (0, 1), sigma = 1)
sat at 0.001 while the off-pivot field at the pivot's own index was
1.0 * exp(-0.5) ~= 0.6065 (the neighbor's gaussian bump).
Both cases are corrected by a single uniform change: _scatter_weights
now writes out[p] = max(w_p, fill[p]) so pivot rows are never written
below the off-pivot field. 'zero' is bit-identical (fill is always 0,
so max(w_p, 0) = w_p when w_p >= 0). 'gaussian' in the sparse-pivot
regime (the typical configuration, especially with k-NN bandwidth) is
also bit-identical because fill[p] equals w_p when no neighbor's bump
at p exceeds w_p.
Implementation
--------------
- _scatter_weights: pivot rows take np.maximum(weights, fill_weights)
unconditionally. Off-pivot rows unchanged.
- _compute_epsilon_floor (renamed from _epsilon_floor): extracted
helper that returns phi (mean / median / fallback). Reused by
'epsilon' and 'epsilon_gaussian'. Parameter baseline narrowed to
the FillEpsilonBaseline Literal type.
- _compute_gaussian_bumps (renamed from _gaussian_bumps): extracted
adapter over _gaussian_fill_weights. Reused by 'gaussian' and
'epsilon_gaussian'. logger is kwarg-only.
- compute_label_weights: dispatcher gains the FILL_METHODS[3] branch.
The combined branch computes bumps once and adds phi in-place via
np.add(out=fill_weights), keeping peak memory at the existing
(chunk, M) buffer; phi is constant in p so the post-reduction add
is algebraically identical to adding inside the chunk loop while
saving O(chunk * M) writes. ValueError messages tightened to
include 'supported values are ...' for parity with
_compute_pivot_sigmas and _aggregate_metrics.
- LabelTransformer.py: extends FillMethod Literal and FILL_METHODS
tuple with 'epsilon_gaussian' at index 3. No new tunables, no new
validators (the existing _EnumValidator(FILL_METHODS) picks up the
new value automatically; existing range / type validators on
fill_epsilon / fill_sigma_* / fill_bandwidth_* apply unchanged).
- QuickAdapterV3.py: logging block refactored from if/elif chain to
parallel if blocks keyed on tuple membership so epsilon and sigma
parameter groups emit independently for each mode that uses them.
Documentation
-------------
README cells updated with set-membership 'Ignored when ...' clauses
matching the new index sets (epsilon | epsilon_gaussian for the
floor parameters, gaussian | epsilon_gaussian for the kernel
parameters). The fill_method description names the additive
composition explicitly and the pivot-row lift invariant
(out[p] = max(w_p, f(p))).
Verified manually on the host via AST extraction harness (no automated
test infrastructure exists in quickadapter/):
- zero mode: bit-exact with prior code (fill is 0, max(w_p, 0) = w_p).
- gaussian mode, sparse pivots: bit-identical to prior code (no
neighbor's bump at p exceeds w_p, so the lift is a no-op).
- gaussian mode, neighbor-dominated regime: pivot rows lifted to the
local field max, fixing the sub-bump dip. Verified with the
counterexample W = (0.001, 1.0) at indices (0, 1), sigma = 1:
legacy out[0] = 0.001, fixed out[0] = 1.0 * exp(-0.5) ~= 0.6065.
- epsilon back-compat (above-floor pivots): phi = eps * mean(W)
reproduced; pivots above phi unchanged.
- epsilon pivot-dip fix: W = (0.001, 1.0, 1.0), eps = 0.5,
baseline = median; legacy out[0] = 0.001, fixed out[0] = phi = 0.5.
- epsilon_gaussian with eps = 0: bit-identical to pure gaussian.
- epsilon_gaussian additive decomposition: out_eg - out_g = phi at
every off-pivot row.
- epsilon_gaussian pivot-row lifted: W = (0.001, 1.0, 1.0) at
well-separated indices (e.g. (0, 100, 200)), eps = 0.5,
baseline = median, sigma = 2.0; out[0] = phi + 0.001 ~= 0.501
(was 0.001 before the scatter fix).
- empty pivots: all four modes return all-zero.
- negative pivot weights still rejected by _gaussian_fill_weights.
- knn bandwidth + epsilon_gaussian: finite, bounded below by phi.
- ValueError messages on invalid fill_method / fill_epsilon_baseline
include 'supported values are ...'.
Jérôme Benoit [Thu, 4 Jun 2026 22:29:58 +0000 (00:29 +0200)]
feat(label_weighting): adaptive k-NN bandwidth for gaussian off-pivot fill (#77)
* feat(label_weighting): adaptive k-NN bandwidth for gaussian off-pivot fill
Address the crushing of weaker pivots by stronger neighbors when pivots
fall within ~sigma_candles of each other in fill_method='gaussian'. The
per-row max aggregator preserves the upper bound Out[i] <= max_p w_p
but a wide constant sigma lets a strong neighbor's Gaussian dominate a
weak pivot's tail.
Add a k-nearest-neighbor bandwidth selector (Loftsgaarden &
Quesenberry 1965; Silverman 1986, paragraph 5.2) that adapts each
pivot's sigma to local pivot density:
where d_k(p) is the index distance to the k-th pivot neighbor. The
upper bound on Out[i] is preserved (no over-amplification) and dense
clusters automatically contract their Gaussians to stop overlapping.
Implementation:
- Pivots are emitted chronologically by zigzag, so the 1D k-NN reduces
to a sliding k-window over sorted indices, O(M) without a spatial
index.
- _gaussian_fill_weights accepts a per-pivot sigma vector via NumPy
broadcasting; the existing chunked exp/multiply/max kernel is
unchanged.
- Default fill_bandwidth='fixed' preserves byte-for-byte the previous
algorithm.
Adds "uniform" to WEIGHT_STRATEGIES (between "none" and the metric
names) which assigns weight=1.0 to every detected pivot. Off-pivot rows
remain governed by the existing fill_method (zero / epsilon / gaussian),
so uniform + gaussian collapses cleanly to a pure proximity kernel
around each pivot. Naming follows sklearn convention
(KNeighbors(weights="uniform"), DummyClassifier(strategy="uniform")).
* chore(quickadapter): bump strategy and regressor version 3.11.11 -> 3.11.12
* refactor(weights): hoist indices_array and valid_mask in compute_label_weights
Compute indices_array and valid_mask once at the top of the function
instead of after the strategy dispatch. The uniform branch can now use
indices_array.size instead of len(indices), and the duplicate
np.asarray / valid_mask construction lower in the function is removed.
Saves one np.asarray and one mask computation per call.
Drop the redundant indices: list[int] parameter now that the only
caller (compute_label_weights) hoists indices_array and valid_mask.
The function takes them positionally and uses indices_array.size for
size checks, removing three len(indices) calls and the optional-kwarg
fallback paths.
* refactor(weights): pipeline API consolidation pass
- compute_label_weights: drop Optional placeholder, accept
Sequence[int] | NDArray[np.integer] for indices
- standardize Optional[X] -> X | None across module (PEP 604)
- _impute_weights: positional call instead of keyword on single arg
- _pivot_equivalent_count: remove unreachable threshold <= 0 branch
(survivors.size > 0 implies survivors.max() > 0 because the input
has been sanitized to non-negative values upstream)
- _scatter_weights: drop dead 'if not np.any(valid_mask)' early
return; vectorized assignment is a no-op when the mask is all-False
- sanitize_and_renormalize: clarify empty-input semantics in docstring
* fix(weights): zero leading and trailing non-finite runs in _impute_weights
The boundary mask only covered the strict tip positions (index 0 and -1),
so multi-element non-finite runs at the boundary were median-imputed
instead of zeroed. With input [NaN, NaN, 1.0, 2.0, NaN, NaN] the function
returned [0.0, 1.5, 1.0, 2.0, 1.5, 0.0] instead of [0, 0, 1.0, 2.0, 0, 0],
silently extending pivot weight to the unconfirmed boundary candles.
Use np.argmax on the finite mask to detect the leading and trailing
non-finite runs and zero the entire run, matching the docstring contract.
* fix(weights): floor stacked metrics in geometric and harmonic aggregation
Power means with p<=0 collapse to 0 on a single zero in the stack:
pmean([1, 0, 3], p=-1) = 0.0 and pmean([1, 0, 3], p=0) = 0.0. Combined
with compose_sample_weights' (arr <= 0) drop_mask predicate, a single
metric returning 0 on a pivot silently drops that row entirely.
Floor stacked_metrics at np.finfo(float).tiny only inside the
geometric_mean and harmonic_mean branches so all-positive pivots
survive aggregation. arithmetic_mean, quadratic_mean, weighted_median
and softmax branches are untouched.
* fix(weights): log when out-of-range pivot indices are dropped
compute_label_weights silently filters out pivot indices outside
[0, n_values) via valid_mask. This made upstream contract violations
invisible: a stale or off-by-one index list would simply produce zero
training weight on those rows with no diagnostic.
Emit logger.warning with the count and dropped fraction whenever
n_dropped > 0 so the upstream caller can spot the issue.
* refactor(weights): collapse 4x label-config validators into a registry
Replace the four near-identical _validate_*_params + get_label_*_config
pairs with a single _LABEL_KIND_REGISTRY mapping each kind name to
(specs, defaults). _label_kind_validator builds the validator on the
fly and get_label_kind_config dispatches to _get_label_config with the
appropriate spec/default pair. The four public get_label_*_config
helpers remain as thin wrappers so existing callers in QuickAdapterV3
and QuickAdapterRegressorV3 are unaffected.
_LabelTransformerConfig.from_dict (LabelTransformer.py) is intentionally
out of scope: it would require propagating a logger through
BaseTransform's freqtrade-side interface, which is upstream-controlled.
* docs(weights): drop verbose empty-input note from sanitize_and_renormalize
The added sentence paraphrased the existing collapse line and restated
obvious facts about zero-length vectors without contractual information.
Revert to the concise pre-W1 docstring.
* docs(weights): drop get_label_kind_config docstring for consistency
The 4 sibling get_label_*_config wrappers have no docstrings; their
parameter names and the registry name self-document the contract. Drop
the redundant docstring on get_label_kind_config to match the family
style.
* fix(weights): drop tiny floor in geometric/harmonic aggregation
The floor at np.finfo(float).tiny added in b1f86a0 preserved pivots
whose metrics included an exact zero, but a zero metric is itself a
'signal absent' marker that downstream compose_sample_weights drops
via the (arr <= 0) mask. Floor was masking the intended drop.
Restore the upstream pmean behavior so a zero in any geometric or
harmonic input produces an exact 0.0 combined weight, allowing
drop_mask to drop the pivot as designed.
Jérôme Benoit [Mon, 25 May 2026 16:05:22 +0000 (18:05 +0200)]
feat(quickadapter): log new label_weighting fill_* tunables
Mirror the existing softmax_temperature conditional-logging pattern:
always log fill_method, then log fill_epsilon and fill_epsilon_baseline
only under fill_method == "epsilon", and fill_sigma_candles only under
fill_method == "gaussian". Keeps the per-column "Weighting:" log block
consistent with the resolved config and avoids printing tunables that
have no effect under the active mode.
Jérôme Benoit [Mon, 25 May 2026 15:52:05 +0000 (17:52 +0200)]
feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting (#74)
* feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting
Adds three off-pivot weighting modes behind a new fill_method tunable in
freqai.label_weighting:
- zero (default): current hard-zero behavior, retained for backward
compatibility.
- epsilon: off-pivot rows receive a flat baseline
fill_epsilon * <baseline>(pivot_weights), where <baseline> is mean or
median, controlled by fill_epsilon_baseline.
- gaussian: off-pivot rows receive a per-row weight from a heatmap-style
decay max_p w_p * exp(-(i-p)^2 / (2 sigma^2)), controlled by
fill_sigma_candles (>= 0.5).
The default is zero so existing configs without the new keys behave
identically. Switching fill_method materially changes per-leaf weight
mass and may require GBM hyperparameter retuning; flagged in the README
description column.
Implementation:
- Adds FillMethod/FILL_METHODS and FillEpsilonBaseline/FILL_EPSILON_BASELINES
Literal types and tuples in LabelTransformer.py.
- Extends DEFAULTS_LABEL_WEIGHTING with the four new keys and their
defaults.
- Extends _WEIGHTING_SPECS in Utils.py with corresponding _EnumValidator
and _NumericValidator entries (epsilon in [0, 1], sigma_candles >= 0.5).
- Refactors _scatter_weights to accept fill_weights as a precomputed
array plus optional indices_array/valid_mask kwargs; preserves
pre-existing length-mismatch ValueError and empty-input early-return
semantics.
- Adds _gaussian_fill_weights helper with in-place pipeline keeping peak
memory at one (chunk, M) buffer; chunk-by-N keyed on
_GAUSSIAN_FILL_CHUNK_BUDGET = 50_000_000 cells (~400 MB peak); emits a
density warning when M / N > 0.1; rejects negative pivot weights.
- Adds *, logger: Logger keyword-only parameter to compute_label_weights
and updates the single call site in QuickAdapterV3.py.
- Replaces the raw nonzero count in compose_sample_weights with a
pivot-equivalent count helper (_pivot_equivalent_count) so the sparse
training mass warning stays meaningful under epsilon / gaussian.
Documentation:
- Four new rows added to the README configuration tunables table under
Label weighting; fill_method flagged as requiring trained-model
deletion when changed.
- Four new keys added to config-template.json under label_weighting.
Verified manually on host via AST extraction harness (no automated test
infrastructure exists in quickadapter/):
- STATIC_OK: defaults + tuples assertions pass.
- SPOTCHECK_4..9: cluster amplification (out[50] ~= 8.0), sigma < 0.5
rejected, negative pivot weights rejected, density warning emitted at
M/N=0.2, empty pivots return zeros, mean/median epsilon ratio = 20.8x.
- SPARSE_4A..C: sparse-mass warning fires under zero mode + sparse
pivots and gaussian sigma=0.5 underflow regime; silent under broad
gaussian fills.
* fix(quickadapter): harden sanitize_and_renormalize against rescale overflow and drop_mask contract violations
Two production-quality safeguards on the load-bearing primitive used by
the new fill_method dispatch in PR #74, plus one cosmetic comment
cleanup.
1. Subnormal-total rescale overflow guard:
When the sum of sanitized weights falls into a subnormal range
(e.g. a single 1e-310 survivor among zeros, n=1000), n/total
overflows to +Inf and safe * Inf propagates Inf to every nonzero
entry, producing mean(out) = NaN and silently violating the
documented mean=1 invariant. The fix computes the rescale factor
into a local c, checks np.isfinite(c), and falls through to the
existing uniform-fallback path with a distinct warning message
('rescale factor non-finite') so operators can distinguish this
regime from the existing 'weights collapsed' case. Bit-identical
on all common paths; c -> 0 underflow is unreachable
(min c = 1/DBL_MAX > 0).
2. drop_mask shape and dtype assertions:
sanitize_and_renormalize is now load-bearing for compose_sample_weights
under all three fill_method modes (zero/epsilon/gaussian). Numpy
broadcasts a (k, n)-shaped mask silently, breaking the (n,) output
contract. Shape and dtype precondition checks raise ValueError
early with prefixed messages matching the function's existing
logger style. Dtype check uses np.issubdtype(..., np.bool_) so
any boolean alias (bool, np.bool_, 'bool') is accepted; integer
masks are rejected.
3. LabelTransformer.py: replace 'current behavior' comment with
'default' on FILL_METHODS[0] since the comparison no longer makes
sense once the PR is merged.
Verified manually:
- REVIEW_FIX_1A..C: bool, np.bool_, 'bool' all accepted; int rejected.
- REVIEW_FIX_2A..B: subnormal-overflow path emits the new distinct
warning; real collapse path emits the original warning.
- REVIEW_FIX_3_OK: docstring contradiction removed.
- REGRESSION_OK: bit-identical common path.
- All original PR #74 verifications still pass (SPARSE_4A..C,
SPOTCHECK_4..9).
* fix(quickadapter): short-circuit compute_label_weights on empty pivot weights
When metrics[strategy] is empty but indices is non-empty, the new
fill_method dispatch in epsilon/gaussian arms slices weights[valid_mask]
before _scatter_weights can short-circuit, raising IndexError on a
size-0 / N-mask shape mismatch. Pre-PR _scatter_weights returned the
default-filled array silently in this case (preserved invariant noted
inline at the empty-input early return).
Add a short-circuit before the dispatch so the contract is consistent
across all three fill methods.
Also trim _gaussian_fill_weights docstring to match the codebase style
(neighboring private helpers carry no docstring or a single short
paragraph) and drop a redundant in-line comment that the in-place
np.multiply(out=buf) pattern already conveys.
Verified on the AST-extraction harness (pre-fix reproduction → fix
verification): 12 contract assertions across 4 edge cases x 3 fill
methods, plus crossmode + non-empty differentiation, all pass; PR #74
SPOTCHECK_4..9, SPARSE_4A..C, REVIEW_FIX_*, REGRESSION_OK still pass.
* chore(quickadapter): bump strategy and regressor version 3.11.10 -> 3.11.11
* refactor(quickadapter): polish label_weighting docs, comments, and sparse-mass diagnostic
Three coordinated polish edits following final-review feedback:
1. _pivot_equivalent_count: replace the 0.5 * median threshold with
_PIVOT_EQUIVALENT_MAX_FRACTION * surviving max (default 0.1). The
median-based heuristic saturated at N under epsilon mode (off-pivot
floor dominates the median once N >> M), silencing the warning the
docstring claimed to provide. The max-relative threshold separates
pivot-class rows from off-pivot fill across the bimodal regimes
fill_method introduces. Constant is module-level and named so the
choice is auditable; warning text now self-describes the threshold
('rows above 10% of surviving max').
2. _scatter_weights: trim the 'Order matters...' comment from 3 lines
to 1 line. The shorter form pins the intentional ordering without
paraphrasing git history; future 'validate inputs first' refactors
are still flagged.
3. README: extend the fill_method row with a concise retuning hint
(per-leaf regularization + Optuna study reset) so the operator
guidance surfaces in user docs, not only in the planning artefact.
Tighten fill_sigma_candles description to match neighboring-row
density.
Verified manually:
- SPARSE_4A..C: original PR cases still pass.
- SPARSE_4D: epsilon+sparse pivots (M=20, N=1000) now correctly fires
the sparse-mass warning (was silenced with median-based threshold).
- SPARSE_4E: zero+skewed pivots ([1,1,...,10]) still fire under the
new threshold (no regression on the skew case).
- SPOTCHECK_4..9, BUG_74_FIX_*, REVIEW_FIX_*, REGRESSION_OK: all
unchanged.
Jérôme Benoit [Mon, 25 May 2026 12:01:55 +0000 (14:01 +0200)]
style(weights): use !r repr for strategy in compute_label_weights ValueError
Replace hardcoded 'none' literal with f"{strategy!r}" to match the
codebase convention of !r-formatting external string identifiers in
error messages and avoid silent drift if WEIGHT_STRATEGIES[0] changes.
Jérôme Benoit [Mon, 25 May 2026 03:18:52 +0000 (05:18 +0200)]
fix(plot): switch raw direction/weight to bar with steelblue color
Raw discrete signals (direction +/-1/0, weight at pivots) are sparse and
read better as bars than lines. Steelblue contrasts with the orange
smoothed lines for clearer visual separation.
Jérôme Benoit [Mon, 25 May 2026 02:38:25 +0000 (04:38 +0200)]
fix(weights): canonical sanitize_and_renormalize and compose_sample_weights
Derived from independent dual-oracle mathematical specification with
proofs (mean=1 invariant, drop preservation, idempotency, collapse
degradation chain).
sanitize_and_renormalize:
- Fix latent bug: fallback path with non-empty drop_mask returned ones
zeroed at drop_mask but did not renormalize, breaking the mean=1
contract. The fallback now renormalizes so mean(out) == 1 holds on
surviving rows.
- Replace .copy()+mutation with np.where for drop_mask application.
compose_sample_weights:
- Replace the post-compose combined.sum() guard (which duplicated the
predicate sanitize_and_renormalize re-evaluates internally) with a
single survivor-aware predicate covering drop_mask | ~isfinite | <=0
in one pass. The check is the explicit branch point for the base-
weights fallback when the label-weighted product collapses on
surviving rows; this preserves the recency signal and the label-
derived drop_mask instead of degrading to uniform.
- Warn when nonzero/n falls below SPARSE_TRAINING_MASS_THRESHOLD (5%,
module-level constant) so operators can spot the sparse-training
regime that pivot-only weights produce on long series with few pivots.
QuickAdapterV3._log_strategy_configuration:
- Warn at startup when label_smoothing.method is 'smm' or 'savgol'
(with polyorder>=2) combined with a non-'none' label_weighting
strategy, since these kernels can collapse a sparse weight signal
and trip the all-rows-dropped guard.
Jérôme Benoit [Mon, 25 May 2026 01:39:28 +0000 (03:39 +0200)]
fix(weights): pivot-only sample weights when label_weighting is active
Replace the full-series median fill with 0.0 in compute_label_weights so
non-pivot rows carry no sample weight when a label_weighting strategy is
configured. The median fill predates PR #72: when weights multiplied the
label, the fill was inert (label=0 × median=0). Once weights became the
sample_weight kwarg of model.fit, the fill silently leaked the median
into the training loss for every non-pivot row, diluting the pivot
detection signal the model is being trained for.
Concretely, training now concentrates on pivots and their smoothed
neighborhoods (via label_smoothing), and the raw/smoothed weight plots
both render clean profiles starting from zero.
strategy='none' (the default) is unaffected: compute_label_weights still
returns a uniform 1.0 vector and every row contributes equally.
Jérôme Benoit [Mon, 25 May 2026 00:45:18 +0000 (02:45 +0200)]
fix(weights): guard DI_values None and read label_frequency_candles from freqai_info
- fit_live_predictions: pred_df.get('DI_values') returns None when
feature_parameters.DI_threshold is 0 or absent (the default), causing
AttributeError on the subsequent .mean()/.std() calls. Fall back to
zeros instead.
- _label_frequency_candles: read from self.freqai_info['feature_parameters']
(matching every other access in the file) instead of self.config, which
is the top-level config dict and never contains feature_parameters
directly. The previous code silently ignored user-provided values and
always fell back to the default 'max(2, 2 * len(self.pairs))'.
Jérôme Benoit [Mon, 25 May 2026 00:31:49 +0000 (02:31 +0200)]
fix(weights): tighten observability and edge-case handling in label pipeline
- sanitize_and_renormalize accepts logger/context kwargs and warns on
uniform-fallback collapse; six call sites in QuickAdapterRegressorV3
thread their stage label (train_test_split / post_feature_pipeline /
timeseries_split, train|test).
- Warn at startup when label_prediction.method='none' for any label, since
populate_entry_trend would silently never trigger.
- Replace .notna() with np.isfinite() in the smoothed-weight clip so +Inf
produced by smoothing kernels is also zeroed instead of relying on the
downstream drop_mask in compose_sample_weights.
- _impute_weights tracks boundary NaN separately so injected zeros do not
bias the interior median; finite endpoints are now preserved.
Jérôme Benoit [Mon, 25 May 2026 00:04:41 +0000 (02:04 +0200)]
refactor(weights): collapse compose_sample_weights to single-target API
LABEL_COLUMNS is single-target by design, so the dict-shaped per-label
map and row-wise aggregation in compose_sample_weights were dead
plumbing. Flatten the signature to a single label_weights vector and
read LABEL_COLUMNS[0] directly in _compose_per_row_weights. Drop the
duplicate-column guard (unreachable under single-target). Align caller
naming on base_weights to match the callee parameter. Add a defensive
check that LABEL_COLUMNS[0] is in dk.label_list to fail loudly if the
project label constant ever diverges from freqtrade's runtime view.
Jérôme Benoit [Sun, 24 May 2026 23:44:44 +0000 (01:44 +0200)]
refactor(quickadapter): drop unused sample_weighting tunables
The sample_weighting.{aggregation,softmax_temperature} options were inert:
LABEL_COLUMNS is single-label by design and compose_sample_weights only
ever sees one per-label vector, making row-wise aggregation an identity
operation regardless of the configured mode.
Removes the config block, validation specs, getter, and README entry;
compose_sample_weights keeps its kwargs with safe defaults (arithmetic_mean,
T=1.0) so the call site stays trivial.
Jérôme Benoit [Sun, 24 May 2026 23:09:27 +0000 (01:09 +0200)]
feat(weights): per-label sample weights propagated to model.fit(sample_weight=...) (#72)
* chore(quickadapter): bump strategy and regressor version 3.11.8 → 3.11.9
* feat(weights): add compose_sample_weights helper with mean=1 multiplicative composition
AFML §4.10 / mlfinpy canonical: per-label mean=1 normalization, multiplicative composition with temporal decay, geometric-mean aggregation for multi-label, NaN/inf handling, all-zero degenerate fallback. Validated locally with pytest (evidence: .omo/evidence/task-5-{red,green}.txt).
* fix(weights): persist label weights into <label>_weight column instead of rescaling target
Removes statistically incorrect target rescaling (label = direction × weight). Persists raw direction labels and a separate <label>_weight column for downstream sample_weight composition. Validated locally with pytest (evidence: .omo/evidence/task-6-{red,green}.txt).
* feat(weights): add _strip_label_weight_columns helper for find_labels collision avoidance
* feat(weights): compose per-label weights with temporal decay before model.fit
* feat(weights): integrate sample_weight composition into both train() data split paths
Add _train_default() mirroring BaseRegressionModel.train() with _compose_train_weights inserted between make_train_test_datasets and _apply_pipelines. Routes train_test_split path through _train_default instead of super().train(). Inserts _compose_train_weights before _apply_pipelines in timeseries_split path. Calls _strip_label_weight_columns(dk) at top of train() for both branches. Validated locally with pytest + structural AST checks (evidence: .omo/evidence/task-9-{pytest,structural}.txt).
* refactor(weights): align _train with BaseRegressionModel.train
Rename _train_default to _train to match the upstream method name (with
underscore prefix to mark it as the internal mirror, since the public
train() method routes between data split paths).
Mirror BaseRegressionModel.train line-for-line with _compose_train_weights
as the single intentional insertion between make_train_test_datasets and
the pipeline application:
- Drop ensure_datetime_series wrapper around unfiltered_df['date']:
upstream calls .iloc[].strftime() directly.
- Drop **kwargs from self.fit(dd, dk) to match upstream signature.
- Use dk.data_dictionary['train_features'].columns for feature count log,
matching upstream source of truth.
- Apply the same cosmetic alignment to the timeseries_split path for
consistency between both train code paths.
- Add docstring documenting the mirror relationship and the single
functional difference.
* refactor(weights): drop dead apply_label_weighting wrapper
Now that QuickAdapterV3.set_freqai_targets persists raw label direction
into the label column and weights into <label>_weight (consumed by
sample_weight downstream), the apply_label_weighting wrapper that
multiplied label values by their weights is no longer used.
- Drop Utils.apply_label_weighting (returned (weighted_label, weights)).
- Drop Utils._apply_label_weights (the values × weights helper).
- Switch QuickAdapterV3.set_freqai_targets to call compute_label_weights
directly (already used internally by the removed wrapper).
* refactor(weights): simplify MAXIMA/MINIMA plot columns for binary direction
Now that the label column holds raw direction in {-1, 0, +1}, the MAXIMA
and MINIMA plot columns reduce to a where(direction>0, 0.0) /
where(direction<0, 0.0) projection.
The previous magnitude-aware logic (plot_eps padding, mask of zero values
on positive direction, etc.) was tailored to the weighted label
amplitudes and is now dead code:
- extrema.abs().where(extrema.ne(0.0)).min() always evaluates to 1.0,
so plot_eps is always max(0.5, _PLOT_EXTREMA_MIN_EPS) = 0.5.
- direction.gt(0) & extrema.eq(0.0) is always False because direction>0
implies extrema = +1, never 0. The .mask() branches never trigger.
Drop the now-unused _PLOT_EXTREMA_MIN_EPS class constant.
* refactor(weights): rename smooth_label to smooth for genericity
The function applies generic smoothing kernels (gaussian, kaiser, triang,
smm, sma, savgol, gaussian_filter1d) to any pd.Series. The 'label'
suffix narrowed it to label-specific use, but it now also smooths the
<label>_weight column (next commit). Drop the suffix; the smoothing
config dict is still named label_smoothing because the per-column
config map remains label-keyed.
* feat(weights): smooth <label>_weight column with the same kernel as label
Per-label weights are pointwise: only pivot indices carry the
metric-derived weight, while non-pivot indices are filled with the
median weight (see compute_label_weights / _build_weights_array).
The label column is smoothed with smooth() to spread pivot signals
over neighbouring candles. Without smoothing the weight column, the
smoothed label values around a pivot keep the constant median weight,
so the model treats high-amplitude pivot neighbours and the pivot
itself as equally important during training.
Apply smooth() to the <label>_weight column with the exact same
per-column smoothing config as the label, so the weight profile
follows the label profile candle-for-candle.
The smooth() positional argument list was redundant with the
col_smoothing_config dict keys; collapse both calls to **kwargs
unpacking. _SMOOTHING_SPECS keys exactly match smooth() parameter
names, so the unpack is type-safe.
compose_sample_weights already replaces non-finite or non-positive
values with 1.0, which absorbs any sign overshoot from kernels like
savgol at series edges.
* feat(weights): add direction and weight subplots showing raw + smoothed signals
Replace the MAXIMA/MINIMA bar visualization with two new subplots that
show both the raw and the smoothed direction/weight curves:
- Subplot 'direction' overlays raw direction (extrema_direction) and
smoothed direction (smoothed_extrema).
- Subplot 'weight' overlays raw weight (extrema_weight) and smoothed
weight (smoothed_extrema_weight).
Each visualization column is captured at the right point in
set_freqai_targets:
- Raw columns (EXTREMA_DIRECTION_COLUMN, EXTREMA_WEIGHT_COLUMN) are
written before the smooth() call.
- Smoothed columns (SMOOTHED_EXTREMA_COLUMN, SMOOTHED_EXTREMA_WEIGHT_COLUMN)
are written after the smooth() call.
Drop MAXIMA_COLUMN and MINIMA_COLUMN constants — they were only used by
the old min_max bar subplot. The new subplots convey the same direction
information plus the per-pivot weight magnitude that the legacy
weighted_label visualization showed (before the sample-weight refactor).
All four visualization column names lack the '&' prefix, so FreqAI's
find_labels auto-detection ignores them; they cannot leak into model
targets.
* refactor(weights): align visualization column names on extrema_<axis>[_smoothed]
Three-agent audit (explore + librarian + oracle) found the previous viz
column names suffered from three inconsistencies:
- 'extrema_direction' (raw) carries the axis word, but 'smoothed_extrema'
drops it, breaking the 2x2 grid (axis × stage).
- Stage qualifier appears as PREFIX ('smoothed_extrema') for one column
and as SUFFIX ('smoothed_extrema_weight') for another.
- 'smoothed_extrema_weight' mixes prefix stage word with suffix axis word.
Production codebases (statsmodels Kalman, bukosabino/ta MACD, mlflow
NPMI, FreqAI's own _mean/_std) overwhelmingly use suffix-decorated
processed forms with the raw form as the plain base. FreqAI's internal
pattern is suffix (&s-extrema_weight, &s-extrema_mean, &s-extrema_std);
align with it.
Result: every viz column follows extrema_<axis>[_smoothed]. The 2x2
grid is uniform, sort-order groups raw and smoothed pairs together,
and the pattern is internally consistent with FreqAI's existing
suffix-based derivations.
* fix(weights): address PR #72 review comments
Three-agent cross-validation (explore + librarian + oracle) of Copilot
review comments produced these verdicts:
C1 (Utils.py:compose_sample_weights) — REAL BUG. Replacing 0-valued
weights with 1.0 silently undoes sklearn / AFML §4.10's canonical 'drop
this sample' semantic. sklearn's _check_sample_weight_equivalence,
DecisionTree _splitter.pyx, HistGBM docs, LightGBM #5553/#905, XGBoost
#3787 and mlfinlab time-decay all converge on the same contract:
sample_weight=0 means 'this sample contributes nothing'. Preserve zeros
via a drop_mask that is OR'd across labels (any label saying 'drop'
wins), then re-applied after the geometric-mean composition. Non-zero
non-finite or negative values still collapse to 1.0 (geometric mean's
neutral element) since they represent undefined weights, not exclusions.
C3/C4 (QuickAdapterRegressorV3.py:_train, timeseries_split) — REAL
REGRESSION. Commit 9953f0c removed ensure_datetime_series with the
rationale 'mirror BaseRegressionModel.train exactly'. But
ensure_datetime_series was introduced in commit ce843f9 specifically
as a workaround for freqtrade issue #13107 (int64 epoch-ms date
columns from feather/parquet handlers). Mirror the algorithm, retain
project-specific safety patches. Restore ensure_datetime_series in
both train paths.
C2/C7 (_label_weight_column_name unused) — DRY violation. Both call
sites in _compose_train_weights now use the helper instead of inline
f-strings.
C8 (train() docstring) — Inaccurate. The default path was claimed to
'Delegate to BaseRegressionModel.train()' but actually routes to
self._train() (a mirror with weight composition). Fix docstring to
reflect actual control flow.
C5/C6 (**kwargs forwarding) — FALSE POSITIVE. freqai_interface never
passes kwargs to model.train(); upstream BaseRegressionModel.train
also calls self.fit(dd, dk) without kwargs. The current code matches
upstream and the call chain is dead in practice.
* refactor(weights): factor train paths and relocate methods for structural coherence
Three-agent structural audit (explore + librarian + oracle) identified
five issues; fixes that don't fight existing conventions:
1. _train and the timeseries_split inline branch in train() shared
~30 lines of identical scaffolding (filter_features, dates logging,
fit_labels guard, weight composition, pipeline application, fit,
timing logs). Extract _train_common(unfiltered_df, pair, dk, split_fn)
that owns the full mirror; _train_default and _train_timeseries_split
become 4-line dispatchers passing the split callback. train() routing
collapses to a clean two-line if/elif.
2. _label_weight_column_name, _strip_label_weight_columns and
_compose_train_weights were inserted into the middle of the class
constants block (between _TEST_SIZE and _SQRT_2), interrupting the
constant-block coherence. Move them to the private instance method
zone, immediately after _apply_pipelines (their natural neighbour).
3. _compose_train_weights duplicated the train/test weight extraction
loop verbatim. Factor into a static _extract_split_weights helper
that takes a split index and returns the per-label weight map; both
train and test call sites become single expressions.
4. The four visualization column constants (EXTREMA_DIRECTION_COLUMN,
EXTREMA_DIRECTION_SMOOTHED_COLUMN, EXTREMA_WEIGHT_COLUMN,
EXTREMA_WEIGHT_SMOOTHED_COLUMN) were 75 lines below EXTREMA_COLUMN /
LABEL_COLUMNS, separated by the LabelData dataclass and label
generator registry. Move them next to EXTREMA_COLUMN where they
logically belong.
5. Revert the train() docstring bullet to its upstream form. The
modification introduced RST cross-reference syntax inconsistent with
the surrounding plain-text docstring style.
* refactor(weights): centralize LABEL_WEIGHT_SUFFIX in Utils
The "_weight" suffix was duplicated as a class constant in
QuickAdapterRegressorV3 and as 5 hardcoded f-strings in QuickAdapterV3.
Three-agent audit (explore + librarian + oracle) converged on moving
this column-naming convention to Utils.py:
- PEP 8 default for constants is module-level; class-level is the
exception for class-private semantics.
- Both consumers already import column-naming constants from Utils.py
(LABEL_COLUMNS, EXTREMA_COLUMN, EXTREMA_WEIGHT_COLUMN, etc.). The
suffix belongs with them.
- Production precedents (sklearn UNUSED/WARN/UNCHANGED, mlflow
_SAMPLE_WEIGHT/_TRAINING_PREFIX, lightgbm _DatasetNames, pandas
LOCAL_TAG) all place cross-module string tokens at module level.
- The constant describes a dataframe schema contract (column names),
not model behaviour. Schema concerns belong in the schema module.
Add LABEL_WEIGHT_SUFFIX to Utils.py next to LABEL_COLUMNS. Remove the
class-level _LABEL_WEIGHT_SUFFIX in QuickAdapterRegressorV3 and import
the module-level constant. Replace 5 hardcoded f-strings in
QuickAdapterV3 with a local label_weight_col binding using the
imported constant.
The private _label_weight_column_name helper is kept in the regressor
since it is used twice (in _strip_label_weight_columns and
_compose_train_weights) and still adds a thin DRY layer over the
suffix synthesis.
* style: apply ruff formatting
* feat(weights): add label_weight_column helper with regex prefix strip and collision assertion
* fix(weights): preserve drop_mask and prevent NaN in compose_sample_weights fallback
* refactor(weights): adopt label_weight_column helper for canonical training column
* feat(weights): add _build_per_row_weights helper for pre-split weight composition
* feat(weights): add _make_default_split_datasets mirror with sklearn-key whitelist
* refactor(weights): make timeseries split helper accept external weights parameter
* refactor(weights): refactor _train_common chain and delete obsolete weight helpers
* refactor(weights): replace train() if/elif with dispatch dict and add weight-column uniqueness check
* fix(weights): harden compose_sample_weights for degenerate inputs
- Extract _sanitize_and_renormalize private helper with four-guard chain
(positive sum, finite sum, finite ratio, finite scaled) and uniform
fallback. Used at empty-map fast path and at final fallback site.
- Empty label_weights_map now sanitizes raw temporal (A0-3).
- Subnormal temporal no longer overflows: ratio + scaled both checked
for finiteness before returning (A0-2).
- Drop predicate unified as 'arr <= 0 or non-finite' instead of exact
zero, eliminating the discontinuity at zero from smoothing artifacts
(A0-4); negatives now drop, no longer rescued to 1.0 (subsumes A1-10).
- Surviving positive values floored at np.finfo(float).tiny to prevent
subnormal arithmetic in the geo-mean log step.
- drop_mask covering all rows now raises ValueError instead of silently
returning all-zero weights that crash XGBoost / sklearn HGBR (A0-1).
- Up-front per-label shape validation raises a precise error instead of
letting numpy broadcasting fail mid-computation (P2 #11).
Address audit findings A1-2, P2 #12, A1-12 on branch
feat/per-label-sample-weights.
Add 11-line docstring covering: output invariant (mean=1), per-label
sanitization predicate, aggregation operator, drop semantics, error
conditions, and the bounded full-series-median leakage in
compute_label_weights.
* refactor(weights): inline data-split dispatch with match/case
Address audit findings A1-1, A1-7, A1-6 (obsolete), P2 #14 on branch
feat/per-label-sample-weights. Conflict C1: A1-7 wins over A1-3
(YAGNI on subclass extension; LSP traceability preserved).
- Delete _DATA_SPLIT_DISPATCH class attribute (4 LOC).
- Delete _data_split_methods_set lru_cached helper (4 LOC dead code).
- Delete _train_default and _train_timeseries_split wrappers (~30 LOC
pure boilerplate dispatching to _train_common).
- Inline match/case on method name in train(); pick the right
_make_*_split_datasets via a local split_builder; nested split_fn
closes over dk. Net -37 LOC; full LSP traceability.
- Add SplitFn module-level type alias used in _train_common signature.
* fix(weights): reject bool config values in numeric validators
Python's `bool` is an `int` subclass, so `isinstance(True, int)` is
true and config values of `true`/`false` silently passed through as
`1`/`0` in the validators for n_splits, gap, max_train_size,
test_size and weight_factor. This commit closes that footgun:
- Add static helpers _coerce_int (always returns int, raises on bool
or non-int) and _coerce_optional_int (returns Optional[int]) to
centralize the validation; both echo the offending raw value via
`{value!r}` so the diagnostic shows True/False rather than 1/0.
- Apply _coerce_int to n_splits and gap, _coerce_optional_int to
max_train_size in _make_timeseries_split_datasets.
- Add explicit bool guard for test_size in both default-split and
timeseries-split paths; previously test_size=true would slip past
isinstance(_, int) and silently train on 1 sample.
- Add explicit bool guard for weight_factor before the >0 comparison.
* fix(weights): preserve mean=1 invariant across pipeline stages
Address audit finding A1-5 on branch feat/per-label-sample-weights.
compose_sample_weights guarantees sum(w)==N (mean(w)==1) over the full
training series, but this invariant breaks twice downstream: (1) the
train/test split partitions weights into disjoint subsets whose means
no longer equal 1, and (2) feature_pipeline.fit_transform may drop
rows via SVM/DBSCAN, drifting the means further. XGBoost
min_child_weight, LightGBM min_sum_hessian_in_leaf and L2
regularization are all sensitive to absolute weight scale.
- Add static helper _renormalize_to_unit_mean with the same four-guard
chain as _sanitize_and_renormalize (positive sum, finite sum, finite
ratio, finite scaled, uniform fallback).
- Apply at four sites: before dk.build_data_dictionary in both
_make_default_split_datasets and _make_timeseries_split_datasets,
and after feature_pipeline.fit_transform / .transform in
_apply_pipelines (train and test sides).
* feat(weights): make label-weights aggregation configurable
Address audit finding A1-11 on branch feat/per-label-sample-weights.
Conflict C3: switch default to arithmetic_mean (matching
_compute_combined_label_weights), expose all 6 aggregations via
existing _aggregate_metrics infrastructure.
The hardcoded geometric mean over per-label normalized arrays was
mathematically conservative (one weak label dominates) and inconsistent
with the project's _compute_combined_label_weights default
(arithmetic_mean). For PR #44's correlated multi-target labels
(amplitude, time_to_pivot, efficiency, natr all derived from zigzag),
geomean over-counts redundant evidence and silently degrades to ~0
when any single factor is small. AFML \xc2\xa74.4 recommends arithmetic-mean
equivalents for correlated meta-labels.
- Add aggregation parameter to compose_sample_weights with default
COMBINED_AGGREGATIONS[0] ("arithmetic_mean"); also expose
softmax_temperature.
- Delegate the row-wise aggregation step to _aggregate_metrics, reusing
the existing 6-operator infrastructure with uniform unit coefficients.
- Read both knobs from feature_parameters in _build_per_row_weights:
label_weights_aggregation and label_weights_softmax_temperature.
* refactor(weights): align naming on compose/sigil/base_weights
Address audit findings A1-9, A1-14, A1-15, A1-16 on branch
feat/per-label-sample-weights.
- Rename _LABEL_WEIGHT_PREFIX_PATTERN to _FREQAI_LABEL_SIGIL_PATTERN:
the regex strips the freqtrade-native '&' sigil, not a label-weight
prefix; the new name describes what is matched (Utils.py).
- Rename compose_sample_weights parameter `temporal` to `base_weights`:
the parameter accepts any base vector (recency weights or uniform
ones), not exclusively temporal data (Utils.py).
- Rename _build_per_row_weights to _compose_per_row_weights:
standardize on the 'compose' verb to mirror compose_sample_weights;
this helper is the orchestrator that calls the kernel
(QuickAdapterRegressorV3.py).
- Rename _build_weights_array to _scatter_weights: the function
scatters sparse pivot weights into a dense default-filled array,
not a generic 'build' (Utils.py).
- Rename eval_set_and_weights to make_test_set_and_weights: aligns
with FreqAI's 'test_*' data_dictionary vocabulary while avoiding
the 'test_' prefix that pytest auto-discovers (the verb 'make_'
also clarifies it as a constructor, not a test) (Utils.py + caller).
Address audit findings A1-8 and A1-13 on branch
feat/per-label-sample-weights.
- Extract the train/test shuffle pattern into a static
_shuffle_in_unison helper. Each call shuffles features, labels and
weights with the same random seed in lockstep. The shuffle block in
_make_default_split_datasets shrinks from ~28 lines (two duplicated
5-line idioms x train+test) to two helper invocations.
- Fix the dk.data_dictionary vs dd inconsistency at the feature-count
log line: read from local dd (the pipeline's return value) rather
than dk.data_dictionary (a side-effect set by _apply_pipelines).
Address audit P2 polish items #2, #4, #8, #13, #15 on branch
feat/per-label-sample-weights. P2 #21 and #22 (logger telemetry)
deliberately skipped per the no-new-comments/no-new-infrastructure
constraint; the existing fail-fast ValueError on degenerate inputs
already surfaces the most critical failure modes loudly.
- Counter-based duplicate-label diagnostic now names the offending
weight columns instead of merely raising on a length mismatch
(P2 #2).
- Widen shuffle seed space from random.randint(0, 100) (101 distinct
seeds, birthday collisions at sqrt(101) ~ 10) to randint(0, 2**31-1)
at both _shuffle_in_unison call sites (P2 #4).
- dsp = dict(self.config['freqai']['data_split_parameters']) replaced
with self.data_split_parameters (the safe pre-populated FreqAI
attribute used everywhere else in the file) (P2 #8).
- Cache label_weight_column with @lru_cache(maxsize=16): the helper
is pure on its single string argument and called in tight loops at
training; matches the file's existing convention for similar helpers
(P2 #13).
- Rename loop variable w to label_values in compose_sample_weights;
the outer scope spans ~20 lines and prior single-letter w obscured
the role (P2 #15).
* docs(weights): align train() and helper docstrings with current behavior
- Rewrite train() docstring to describe match-based dispatch and the per-row
weight composition flow through _train_common; remove stale delegation claim.
- Sync _compose_per_row_weights docstring: aggregation default is
arithmetic_mean, not geometric_mean.
- Fix AFML citation in compose_sample_weights from section 7.4 to chapter 4.
- Document _aggregate_metrics softmax branch as a per-column convex
combination with explicit T->0 and T->+inf limits.
- Promote Utils._sanitize_and_renormalize to public sanitize_and_renormalize.
- Drop QuickAdapterRegressorV3._renormalize_to_unit_mean (cross-file
duplication of the mean=1 invariant); replace 6 call sites with the
unified helper. Sites that previously skipped per-element sanitization
now also reject non-finite or non-positive entries.
- Collapse triple-guard ladder in sanitize_and_renormalize to a single
finite-positive total check; surviving 'safe' is provably finite-nonneg
so intermediate isfinite checks were dead code.
- Tighten _shuffle_in_unison signature from Any to concrete pd.DataFrame
and NDArray types.
- Drop empty-fold sentinel in _make_timeseries_split_datasets and raise
ValueError on degenerate generator output instead of silently producing
empty index arrays.
* feat(weights): validate label_weights tunables via label_weighting block
- _compose_per_row_weights now consumes get_label_weighting_config (which
validates aggregation against COMBINED_AGGREGATIONS and enforces
softmax_temperature > 0 via _WEIGHTING_SPECS) instead of reading raw
feature_parameters.label_weights_*.
- Add CONFIG_MIGRATIONS entries auto-migrating
freqai.feature_parameters.label_weights_aggregation and
freqai.feature_parameters.label_weights_softmax_temperature to the
freqai.label_weighting block; users get one warning per key.
- Add module-level _logger to Utils.py and warn on
compose_sample_weights silent fallback so collapsed-aggregation paths
are observable.
- _make_timeseries_split_datasets honors reverse_train_test_order for
parity with _make_default_split_datasets and raises ValueError on
shuffle_after_split=True (chronological + shuffle is incoherent and
would leak future data into training).
* fix(weights): seed shuffle deterministically from data_split_parameters.random_state
Replace global random.randint() with a random.Random instance derived from
data_split_parameters.random_state. When the user provides a random_state
(whitelisted in _SKLEARN_TRAIN_TEST_SPLIT_KEYS), train and test shuffles
become reproducible end-to-end; when absent, behavior remains
non-deterministic. The single parent RNG draws two independent sub-seeds
so train and test shuffles stay decorrelated.
* refactor(weights): rename for naming coherence
- Rename _make_default_split_datasets to _make_train_test_split_datasets
to restore the case-key/method-name grep-line in train()'s match
dispatch.
- Rename label_weight_column to label_weight_column_name across Utils.py,
QuickAdapterV3.py and QuickAdapterRegressorV3.py: the helper returns
a column-name string, not a column accessor; the new name matches the
*_COLUMN constant convention used elsewhere in Utils.py.
- Drop redundant 'dk.data_dictionary = dd' in _apply_pipelines:
build_data_dictionary already self-assigns the dict on dk and dd is
the same object reference.
* style(weights): group EXTREMA_* constants and separate LABEL_* declarations
* docs(weights): rewrite _make_train_test_split_datasets docstring without history narration
Replace the deviation list and PR-history reference with a concise
description of what the function IS (sklearn-key whitelist, honored
tunables, weight propagation contract).
* fix(weights): resolve KeyError in label_weighting config consumption
_compose_per_row_weights passed self.config (root) to
get_label_weighting_config and accessed weighting_config['aggregation']
directly; the helper expects freqai.label_weighting and returns
{default, columns}. Fix consumes self.freqai_info['label_weighting']
and reads ['default']['aggregation'] / ['default']['softmax_temperature'].
* fix(weights): honor drop_mask in sanitize_and_renormalize fallback
When total <= 0 or non-finite, the helper returned np.ones_like(arr)
ignoring drop_mask, resurrecting dropped rows with weight=1. Fallback
now zeros drop_mask rows before returning.
* fix(config): rename reverse_test_train_order to reverse_train_test_order
Match the canonical key name from upstream freqtrade and from the code
in QuickAdapterRegressorV3._make_train_test_split_datasets and
_make_timeseries_split_datasets. The previous template key was silently
ignored.
* fix(strategy): clip smoothed weight column to non-negative finite
Some smoothing methods (savgol, filtfilt) can ring negative on positive
input. Clip the smoothed weight series to >= 0 and replace non-finite
with 0 before assigning to the dataframe so compose_sample_weights does
not silently drop rows that were positive before smoothing.
* fix(weights): apply project _TEST_SIZE default when data_split_parameters omits test_size
The whitelist comprehension that builds sklearn_kwargs only preserved keys
present in data_split_parameters; the local test_size variable computed via
dsp.get(..., _TEST_SIZE) was never injected back. Configs without an explicit
test_size silently fell through to sklearn's stock 0.25 default instead of
the project's 0.1.
Replace bare 'shuffle' insertion with setdefault for both shuffle and
test_size so sklearn_kwargs always carries the project defaults.
Update _make_train_test_split_datasets docstring to reflect the actual
default behavior.
* refactor(strategy): rename smoothed_weights to smoothed_label_weights
Aligns naming with surrounding label_weights variable, EXTREMA_WEIGHT_SMOOTHED_COLUMN
constant and compute_label_weights helper.
* feat(weights): split per-row aggregation into freqai.sample_weighting block
Cross-metric (per-pivot) and cross-label (per-row) compositions are
distinct distributions. Decouple their tunables:
- freqai.label_weighting.{aggregation,softmax_temperature} stay for
cross-metric aggregation in compute_label_weights (combined strategy).
- New freqai.sample_weighting.{aggregation,softmax_temperature} for
cross-label composition in
QuickAdapterRegressorV3._compose_per_row_weights.
Add _SAMPLE_WEIGHTING_SPECS, DEFAULTS_SAMPLE_WEIGHTING and
get_sample_weighting_config helper routed through _get_label_config so
the returned shape ({default, columns}) matches the get_label_*_config
family.
* refactor(weights): pass logger as parameter and log per-label weight column status
Drop the module-level _logger introduced in Utils.py; compose_sample_weights
now takes a keyword-only logger argument, matching the caller-passes-logger
pattern used by every other helper in the file.
In _compose_per_row_weights, log per-label weight column resolution at
debug when columns are present (static across retrains) and at warning
when none are found (unexpected configuration; falls back to temporal
weights only).
* fix(weights): use shape-consistent empty containers for test_size=0 sentinels
Replace np.zeros(2)/pd.DataFrame() sentinels with iloc[:0] / weights[:0]
slices so test_features, test_labels and test_weights all have 0 rows
with preserved column names and dtype. Behavior unchanged because
_apply_pipelines skips test-side processing when test_size == 0, but
shape-consistent containers respect the declared types and avoid
surprising downstream consumers.
Jérôme Benoit [Sat, 23 May 2026 22:23:35 +0000 (00:23 +0200)]
fix(zigzag): default normalize to False to prevent label magnitude leak (#71)
When set_freqai_targets is invoked by FreqAI's backtesting loop, the dataframe
passed to _generate_extrema_label spans the full historical window
(right-truncated to the current train-window stop), not just train_period_days.
With normalize=True, zigzag applies a global minmax scaling across all detected
pivots in that wider window to amplitudes, amplitude_threshold_ratios,
volume_rates and speeds. The resulting label magnitudes therefore depend on the
global pivot distribution, including pivots outside the current training slice
— a magnitude leak from out-of-train data into training labels.
Switching the zigzag default to normalize=False emits raw log-amplitude values
(|log(P2/P1)|) and defers any scaling to LabelTransformer, which is fitted
strictly on the train slice and is therefore leak-free. The two existing call
sites — _generate_extrema_label (label generation) and label_objective (Optuna
hyperopt) — both want the unnormalized output, so the redundant
normalize=False kwargs are dropped at the call sites in favor of the default.
Strategy and regressor patch versions are bumped to 3.11.8.
Caveat: with apply_label_weighting strategy="combined", the metrics now sit on
heterogeneous scales (raw log-amplitudes ~[0.005, 0.5] mix with bounded ratios
in [0, 1] like efficiency_ratio). Users relying on "combined" aggregation
(power means, weighted_median, softmax) may need to introduce metric-specific
rescaling on the train slice before aggregation. Direction-only (strategy=
"none") and single-metric strategies (e.g. strategy="amplitude") are
unaffected.
Jérôme Benoit [Fri, 1 May 2026 19:03:15 +0000 (21:03 +0200)]
fix: pin pandas>=3.0 in Dockerfile and bump version to 3.11.7
Prevent silent pandas downgrade to 2.x during pip install, which
causes dtype mismatches with freqtrade 2026.4 date handling code.
Includes epoch-ms range validation in ensure_datetime_series.
Jérôme Benoit [Fri, 1 May 2026 14:03:12 +0000 (16:03 +0200)]
fix: validate epoch-ms range before converting int64 date columns
Reject int64 values outside [2010, 2035] epoch-ms range to fail fast
on corrupted data instead of silently producing wrong dates. Catches
nanosecond/microsecond values that would pass the int64 dtype check
but produce garbage timestamps if interpreted as milliseconds.
Jérôme Benoit [Fri, 1 May 2026 13:46:19 +0000 (15:46 +0200)]
fix: add 30min stop_grace_period to prevent data corruption on shutdown
FreqAI training can take minutes to hours. Docker's default 10s grace
period causes SIGKILL mid-write, corrupting feather/pickle files.
Give freqtrade up to 30 minutes to finish training and flush data
before Docker sends SIGKILL.
Jérôme Benoit [Fri, 1 May 2026 10:32:33 +0000 (12:32 +0200)]
fix: align ensure_datetime_series with freqtrade data handler pattern
Chain .dt.as_unit("ms") to guarantee datetime64[ms, UTC] output
resolution regardless of pandas version, matching the contract
established in freqtrade commit 2c5dc72.
refactor: extract ensure_datetime_series helper for date dtype workaround
Centralizes the int64 epoch-ms vs datetime detection logic into a shared
helper. Handles both formats correctly: unit='ms' for int64, passthrough
for existing datetime columns.
fix: workaround freqtrade 2026.4 date column dtype regression
Freqtrade 2026.4 (commit 2c5dc72) changed feather/parquet handlers to
use .dt.as_unit("ms") instead of to_datetime(col, unit="ms", utc=True).
This breaks when data files store dates as int64 epoch-ms, causing
AttributeError in feature_engineering_standard.
Use pd.to_datetime(col, utc=True) defensively to handle both int64 and
datetime inputs.
repositories / freqai-strategies.git / log
