Raise a contextual DependencyException when an integer test_size is not
smaller than the training rows left after the outer holdout, and when the
feature pipeline (SVM/DBSCAN outlier removal) empties the validation set
after transform, instead of surfacing a low-level sklearn/predict error.
* refactor(quickadapter): normalize None test_size in _add_refit_data
Match _add_validation_split: coerce a None test_size to _TEST_SIZE before
the ==0 short-circuit, so the refit gate stays consistent with the
validation gate regardless of the _TEST_SIZE value.
Record the exact row where each Zigzag label becomes final, include TA-Lib NATR warmup/backfill in label provenance, leave unresolved trailing labels unavailable, compose centered smoothing availability over the complete kernel support, keep the incomplete right smoothing edge unavailable, reject circular wrap smoothing in causal mode, and preserve the public nine-element zigzag() tuple API.
Adds a per-row known_at_lookahead availability channel for labels and weights (weight availability lags the label by one pivot), with an optional local Gaussian fill band; the causal purge folds label and weight availability row-wise. Default path (strategy=none, fill=zero) stays byte-identical; the knn/neighbors>=2 residual is bounded, non-default, and documented.
refactor(quickadapter): extract is_finite_number to Utils and reuse it (#122)
* refactor(quickadapter): extract is_finite_number to Utils and reuse it
- move the numeric/finite/non-bool scalar guard from a QuickAdapterV3
static method to a shared Utils.is_finite_number helper
- reuse it across the strategy label getters/setters and the shared
label_natr_multiplier validation guard
- leave the strict positive-int label_period_candles/label_horizon_candles
paths unchanged (distinct predicate)
* refactor(quickadapter): harden label-param fallbacks and per-row NATR guard
Address the #112 re-review findings (landed in this PR):
- coerce the per-row label_period_candles series with pandas to_numeric
before np.isfinite, matching the guarded scalar path (object/str dtype no
longer raises)
- re-validate the persisted _label_params fallback in the label getters via
is_finite_number before use, otherwise fall back to config
- generalize the is_trade_runmode comment (it gates both persisted-param
reuse and per-candle param setting)
- README: drop the inaccurate 'simulated' wording and the runmode-specific
framing on the label_period_candles/label_natr_multiplier rows
* docs(quickadapter): drop per-candle label-param behavior notes from tunables
The label_period_candles/label_natr_multiplier rows describe labeling
tunables; the per-candle strategy-NATR/exit consumption is an internal
behavior detail (already documented inline in the code), not needed to
configure the tunable. Revert both rows to their base description.
* style(quickadapter): wrap set_label_natr_multiplier guard per ruff format
The committed one-liner exceeded the 88-char line length; apply ruff
format wrapping (behavior unchanged).
* refactor(quickadapter): harden label-param validation and dedupe runmode gate
- add _is_finite_number guard rejecting bool and non-numeric before
np.isfinite (which raised on str/object) across the label getters/setters
- factor the duplicated runmode-in-TRADE_MODES predicate into the
is_trade_runmode cached_property
- document the per-candle NATR construction and drop the redundant fillna
after where() in the per-row NATR path
- note that backtest and hyperopt (not only backtest) retain the per-candle
label period in the README
fix(quickadapter): make HPO state causal in backtests (#111)
* fix(quickadapter): make label HPO causal in backtests
* fix(quickadapter): harden causal label HPO bounds, docs and harmonization
- guard empty prediction history and use NaT/order-safe max()/min() when
bounding label HPO OHLCV to the current FreqAI prediction time
- gate strategy label-param loading on freqtrade TRADE_MODES to mirror the
regressor self.live gate
- document the point-in-time study reset (supersedes explicit continuous=false)
and causal warm-start seeding
- make the optuna_hyperopt.enabled README entry terse and add a dedicated
causal label HPO note; unify terminology on current FreqAI prediction time
* docs(quickadapter): drop causal label HPO note from README
* fix(quickadapter): silence benign Optuna study deletion on fresh storage
- treat a missing study on delete as a debug no-op: non-live runs use a
fresh InMemoryStorage and the first live/dry-run optimization per pair
has no persisted study yet, so optuna.delete_study raises KeyError; keep
warning+traceback for genuine deletion failures
- drop redundant point-in-time frame copies: DataProvider.get_pair_dataframe
already returns a caller-owned frame and it is only read downstream
- lower the non-live 'Label HPO skipped' bounds logs to debug (expected
backtest warmup states, consistent with the throttle debug log)
- refine the point-in-time docstring (dk.full_df is the full feature frame)
and note that self.live is unset at __init__
* docs(quickadapter): tighten HPO comments and de-parenthesize README
- make the __init__ trade-mode, non-live point-in-time HPO, and delete_study
KeyError comments more precise and concise without dropping semantics
- reword the delete_study comment to point at the warning branch instead of
the inaccurate 'real failures raise otherwise'
- rephrase the optuna_hyperopt.continuous README entry without a parenthetical
precision, consistent with the surrounding prose
* chore(serena): migrate project config to the language_servers schema
Serena renamed the deprecated `languages` key to `language_servers` and
refreshed the accompanying comments; regenerate the tracked project config
to match.
chore(quickadapter): raise default fill_sigma_candles to 25.0
Update the canonical DEFAULTS_LABEL_WEIGHTING value, the config template,
and the tunables documentation in lockstep. The default is inert for the
default fill_method ("zero"); it only affects the per-pivot Gaussian
bandwidth (and its knn clip upper bound) when fill_method is "gaussian"
or "epsilon_gaussian".
refactor(quickadapter): disambiguate and centralize final exit stage constants
Rename the _FINAL_EXIT_STAGE tuple to _FINAL_EXIT_STAGE_PARAMS and add the
class-level constant _FINAL_EXIT_STAGE_INDEX (max(partial_exit_stages) + 1),
referenced from the configuration logging, the get_trade_exit_stage clamp,
and the plot config. This removes the triplicated stage-index expression and
distinguishes the final stage parameters from its index. Qualify the plot
config partial_exit_stages and stage-index accesses with QuickAdapterV3 to
match the surrounding constant access.
fix(quickadapter): bound take-profit stage to filled take-profit exits
Derive the take-profit stage from filled take-profit-tagged exit orders
clamped to the final full-exit stage, instead of nr_of_successful_exits
plus in-flight exit orders. This stops the stage index from overshooting
its maximum (e.g. take_profit_*_4 while the final stage is 3) when a
final exit order is in flight or a non-take-profit exit fills.
Guard custom_exit against re-issuing the final take-profit exit while an
order is open by returning None after the model-expiry and reversal
safety exits and before the take-profit stage computation, so safety
exits still fire while in-flight orders no longer trigger a duplicate
take-profit exit.
Jérôme Benoit [Mon, 22 Jun 2026 13:07:40 +0000 (15:07 +0200)]
style(quickadapter): emit label_horizon_candles per pair in HPO=on log block
Two post-implementation review findings (Oracle A + Oracle B) resolved
together by a single design change.
Oracle A flagged a latent value-mismatch in the HPO=on branch: the
global `self._label_horizon_candles()` call (no `pair` argument) reads
from `ft_params` only, so when `label_horizon_candles` is unset and
the helper falls back to `label_period_candles`, the logged value
reflects the global initial seed rather than the per-pair effective
value used at fit time (where HPO-tuned `label_period_candles` per
pair drives the fallback). When the user sets `label_horizon_candles`
explicitly, the global call is correct.
Oracle B flagged a cross-branch ordering inconsistency: HPO=on emitted
`label_horizon_candles` first (before per-pair lines), while HPO=off
emitted it last (after period + multiplier). Maintainers comparing
logs across HPO toggles see the same logical scalar in different
positions.
Move the per-pair effective `label_horizon_candles` into the per-pair
line in the HPO=on branch (third field after `label_period_candles`
and `label_natr_multiplier`). This closes the per-pair accuracy gap
(Oracle A) and yields the same `period → multiplier → horizon` order
in both branches (Oracle B). No behavioral change; logging only.
Jérôme Benoit [Mon, 22 Jun 2026 13:00:18 +0000 (15:00 +0200)]
refactor(quickadapter): disambiguate label_period_candles vs label_horizon_candles
Three distinct issues addressed together for terminology coherence
and operator visibility.
(1) README: replace "NATR horizon" by "NATR period" in the four rows
documenting `label_period_candles` / `min_label_period_candles` /
`max_label_period_candles` / `label_candles_step`. The noun
"horizon" was also the noun in `label_horizon_candles`, which has
the opposite temporal direction (lookahead causal-split guard vs
lookback NATR period). "NATR period" matches both the tunable name
(`label_period_candles`) and the underlying API
(`ta.NATR(..., timeperiod=label_period_candles)`).
(2) `QuickAdapterV3.set_freqai_targets`: rename the local
`label_period` (a `datetime.timedelta` spanning
`len(dataframe) * timeframe_minutes`) to `series_duration`, and
update the two log labels (`label_period: 3 days, 12:00:00` →
`series_duration: 3 days, 12:00:00`). The previous name collided
with `label_period_candles` (an int candle count) at the operator
log level; `series_duration` matches the sibling `series_length`
(an int) declared two lines below.
(3) `QuickAdapterRegressorV3.__init__` startup dump: add
`label_horizon_candles` to the "Label Parameters:" section (both
HPO-enabled and HPO-disabled branches). The parameter is a
training-time data-split lookahead (causal split guards in
`_make_train_test_split_datasets` / `_make_timeseries_split_datasets`),
NOT an HPO hyperparameter — it is neither sampled
(`trial.suggest_int`) nor bounded (`min/max_*`) nor an HPO input to
`label_objective`. Placing it in "Label Parameters:" (runtime
values) rather than "Label Hyperparameters:" (HPO config) matches
its actual semantics. In the HPO-on branch the global value is
emitted before the per-pair loop to visually distinguish it; in the
HPO-off branch it trails the existing global lines so the natural
order is `label_period_candles` → `label_natr_multiplier` →
`label_horizon_candles` (the latter falls back to the former when
unset).
The (c) clause previously named both `ValueError` and
`UnicodeDecodeError` plus their subclass relationship. The exception
classes are visible at the `except ValueError` line below — keep the
behavioral description in the docstring ("not parseable by
json.loads (malformed JSON or invalid UTF-8)") and let the code
document the catch surface.
Jérôme Benoit [Mon, 22 Jun 2026 12:35:27 +0000 (14:35 +0200)]
fix(quickadapter): catch UnicodeDecodeError in journal tail probe
Codex inline review (P2) on PR #102 flagged that
`_optuna_journal_has_corrupt_tail` catches only
`json.JSONDecodeError` from `json.loads(last_line)`, but
`json.loads` raises `UnicodeDecodeError` (subclass of
`ValueError`, NOT of `JSONDecodeError`) when the trailing record
contains invalid UTF-8 bytes — a common crash pattern when
`fsync` is interrupted mid-multibyte. The exception escapes the
helper, propagates out of `optuna_create_storage` (the helper
runs BEFORE the recoverable try/except), reaches
`optuna_create_study`'s broad outer handler, and reproduces the
silent-HPO-skip symptom under a different corruption class.
Empirically reproduced:
>>> json.loads(b'\\xc3\\x28')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in
position 0: invalid continuation byte
Broaden the except clause from `json.JSONDecodeError` to
`ValueError` — the common parent of both `JSONDecodeError` and
`UnicodeDecodeError` — so the helper treats any
`json.loads`-unparseable trailing record as corruption and routes
it through the same quarantine path. Other `ValueError` subclasses
are not plausibly raised by `json.loads` on bytes input.
Reproducer at `/tmp/quickadapter-tests/test_optuna_journal_quarantine.py`
extended from 17 to 19 scenarios (Class C4 detection + end-to-end
quarantine on invalid UTF-8). All pass.
Jérôme Benoit [Mon, 22 Jun 2026 12:31:01 +0000 (14:31 +0200)]
fix(quickadapter): quarantine corrupt optuna journal log and recover (#102)
When `JournalFileBackend`'s replay encounters a corrupt journal record, optuna raises during `JournalStorage.__init__` (Class A/B: immediate `KeyError`/`json.JSONDecodeError`) or defers the raise to the next `_sync_with_backend` (Class C1/C2/C3: truncated tail / malformed last line / bare trailing newline at EOF). Both paths previously caused HPO for the affected pair to be silently skipped on every fit cycle.
Wrap `JournalStorage` construction in a narrow `try / except (KeyError, ValueError, json.JSONDecodeError)`, atomically rename the corrupt log aside as `optuna-<COIN>.log.corrupt-<UTC_µs>`, log a `WARNING`, and retry once on a fresh path. A bounded O(1) tail probe runs BEFORE construction to detect deferred-raise EOF corruption. `OSError` is intentionally excluded so filesystem failures stay operator-actionable.
The recovery follows RocksDB's documented WAL recovery philosophy (quarantine + restart), preserves forensic evidence (rename, no `unlink`), and keeps the live-journal glob `optuna-*.log` from matching quarantined artefacts.
Jérôme Benoit [Mon, 22 Jun 2026 10:09:19 +0000 (12:09 +0200)]
refactor(quickadapter): dispatch optuna_create_sampler via match + assert_never (#103)
Follow-up to #101. Converts the `optuna_create_sampler` if/elif/else dispatch chain to a `match`/`case` statement using value patterns (`QuickAdapterRegressorV3._OPTUNA_SAMPLERS.<name>`) -- the per-field singleton `Literal[...]` typing introduced by #101 unlocks pyright/mypy exhaustiveness narrowing, so the final `case _: assert_never(sampler)` type-checks as `Never` and catches any future extension of `OptunaSampler` that forgets to add a corresponding match branch.
The `case None:` branch is structurally required (not stylistic): without it, after the four `Literal[...]` value patterns, pyright/mypy would narrow `sampler` to `None`, not `Never`, and `assert_never(sampler)` would fail to type-check. Its presence is what makes the "5th sampler addition -> type error at assert_never" claim effective. The error message inside `case None:` is preserved verbatim from the prior `else: raise ValueError(...)` for user-facing wire compatibility.
Behavior delta is confined to non-`Literal` string inputs (e.g. dynamically-injected `"garbage"`): prior code raised `ValueError` with the supported-values list; new code raises `AssertionError` with the standard `typing.assert_never` message. In practice this path is unreachable from every current in-repo call site -- the sole caller `optuna_create_study` already validates `sampler not in samplers` and raises `ValueError` with the supported-values list before dispatching, so misconfigured `_optuna_config["sampler"]` values surface the old error shape from the upstream gate. The new `AssertionError` would only manifest via a hypothetical future caller that bypasses `optuna_create_study`, which does not exist today.
Pattern parity: the same `assert_never` exhaustiveness idiom is already used in this file for `support_policy` dispatch; value-pattern syntax matches the 8 call sites migrated by #101.
Closes the deferred follow-up from PR #101 (issue #88).
Jérôme Benoit [Mon, 22 Jun 2026 09:32:44 +0000 (11:32 +0200)]
refactor(quickadapter): consolidate Optuna sampler tuples to NamedTuple (#101)
PR #81 consolidated `_OPTUNA_NAMESPACES` to `Utils._OptunaNamespaces` (a `NamedTuple` with per-field singleton `Literal` types). Propagate the same pattern to the three sibling sampler tuples in `QuickAdapterRegressorV3.py`:
Each new `_Optuna*Samplers` NamedTuple class lives at module level immediately before `class QuickAdapterRegressorV3`, matching the `_OptunaNamespaces` adjacency in `Utils.py`. Per-field types are singleton `Literal["..."]` (not the `OptunaSampler` union) to unlock pyright/mypy narrowing for a future `assert_never` migration. The class-private `_OPTUNA_*_SAMPLERS` instance constants remain class-attributes of `QuickAdapterRegressorV3` to preserve every consumer's `QuickAdapterRegressorV3._OPTUNA_*` access pattern.
Each `_Optuna*Samplers` redeclares its field defaults (per-class `Literal["tpe"] = "tpe"` etc.), replacing the prior tuple-slice derivation `_OPTUNA_HPO_SAMPLERS = _OPTUNA_SAMPLERS[:2]` and the 6-line custom reordering of `_OPTUNA_LABEL_SAMPLERS`. Trade-off: minor string-literal duplication across 3 classes is accepted for harmonization with the `_OptunaNamespaces` template; the single source of truth for the 4 valid tokens stays the `OptunaSampler = Literal[...]` alias unchanged at module level.
Migrates 8 positional-indexing call sites (`_OPTUNA_*_SAMPLERS[N] # "name"`) to attribute access (`.<name>`); drops the 6 surviving inline annotations at the call sites (2 sites carry no annotation pre-refactor; the multi-line site collapses 3 source lines into 1) plus the 4 analogous annotations inside the deleted 6-line `_OPTUNA_LABEL_SAMPLERS` custom-ordering block. The `_OPTUNA_HPO_SAMPLERS_SET` and `_OPTUNA_LABEL_SAMPLERS_SET` frozenset companions are kept unchanged (still used in O(1) membership testing at `optuna_samplers_by_namespace`); their type annotation `Final[frozenset[OptunaSampler]]` is preserved.
Non-migration sites confirmed unchanged: the frozenset companion construction, the `', '.join` error-message iteration over `_OPTUNA_SAMPLERS`, and the `_SET` membership references. All iterate over the NamedTuple instance and produce byte-identical output.
Add `NamedTuple` to the existing `from typing import (...)` block (alphabetical, between `Literal` and `Optional`). `assert_never` already imported, kept in anticipation of the deferred follow-up.
Per-field singleton `Literal[...]` unlocks `assert_never` exhaustiveness on the `optuna_create_sampler` dispatch chain -- left to a follow-up PR since the migration (`else: raise ValueError(...)` becoming `assert_never(sampler)`) changes the user-facing error contract on the unreachable branch.
The AGENTS.md *Canonical defaults* principle and the README documented enum order are encoded in the field-declaration order of each `_Optuna*Samplers`. NamedTuple remains a `tuple` subclass, so `[0]` indexing, `len(...)`, `frozenset(...)`, `', '.join(...)`, and iteration all keep their existing semantics. No behavior change.
Reviewed by two pre-implementation design passes (3-oracle on v1; Metis + Momus + meta-Oracle on v2) and a 3-oracle review on the live PR, each citing upstream evidence from `freqtrade/freqai/` confirming no external consumer.
Follow-up from PR #81 review (Oracle harmonization dimension).
Prose state-form (`Utils.py`, `QuickAdapterRegressorV3.py`):
- `_normalize_label_column_name` docstring: ``Raises ValueError when
the result contains `&` or `%` after sigil strip``.
- Deprecated-config-key warning aligns with the sibling pattern at
the adjacent branch: ``f"{old_path!r} is deprecated, use
{new_path!r} instead"``.
- `sanitize_and_renormalize` docstring states ``mean(out) == 1`` as the
rescale invariant.
- Optuna-label throttle log reads ``callback throttled,
{N} candles until next emission``.
- Fit-live-predictions warmup log reads ``Fit live predictions not
warmed up: {N} candles until warmup completion``.
Docstrings on validator/composer helpers (3 functions lacking a
docstring at HEAD):
- `_apply_support_policy`: documents the ``policy='raise'`` /
``policy='fallback'`` dispatch contract.
- `_compose_train_weights_with_support`: documents the support-gating
flow (None-label-weights branch routes through
``_apply_support_policy`` when ``strategy != 'none'``; main branch
composes and validates the summary against three thresholds).
- `_validate_optuna_label_best_params`: enumerates the rejection
paths and the optional ``expected_selection_metadata`` drift gate.
Harmonization (post-merge carry-over):
- `LABEL_WEIGHT_SUFFIX` renamed to `_LABEL_WEIGHT_SUFFIX`
(no external consumer; symmetric with
`_LABEL_KNOWN_AT_LOOKAHEAD_SUFFIX`).
- `safe_distribution_fit` call-site contexts harmonized with the
PR #97 / PR #99 ``[<pair>] <event>`` convention:
`f"[{pair}] di_values_weibull_fit"` and
`f"[{pair}] label_norm_fit:{label_col}"`.
Python idioms:
- `_adapt_label_generator` rejects any 3rd positional parameter
whose name is not ``logger``, regardless of whether the parameter
is required or has a default. A defaulted non-``logger`` 3rd
positional raises ``ValueError`` at registration. The 3-arg
pass-through is reached only when ``positional[2].name == "logger"``.
- `_build_sample_weight_inputs` switches the two `logger.debug`
calls to lazy ``%s`` formatting so the f-string body is not
materialized when the debug level is disabled.
Consolidates SAFE + DRY + COSMETIC findings from the 4-axis review of
`add1fb7..d6a718f` (PRs #90, #94, #95, #96, #97 + 2 style + d6a718f).
Migration (PR #94 carry-over to `QuickAdapterV3.py`):
- `_TRADE_DIRECTIONS_SET` and `_ORDER_TYPES_SET` are
`Final[frozenset[T]]` constants adjacent to the canonical
`_TRADE_DIRECTIONS` / `_ORDER_TYPES` tuples; the strategy reads
them directly at all 3 call sites. The
`@staticmethod @lru_cache(maxsize=None)` set-views are absent at
HEAD.
Context-prefix harmonization (PR #97 carry-over):
- The 2 `sanitize_and_renormalize` calls in
`QuickAdapterRegressorV3._apply_pipelines` carry the `[<pair>]`
prefix (`f"[{pair}] post_feature_pipeline:train"` / `:test`),
matching the PR #97 convention at the 4 `compose_sample_weights`
call sites.
Alias removal (PR #96 carry-over):
- `Utils.label_known_at_column_name` is absent at HEAD. The
underlying column has zero external callers in the repo, and
PR #96 shifted semantics (absolute index -> per-row offset);
no back-compat alias is warranted.
DRY (`d6a718f` carry-over):
- The 8-tuple swap blocks in `_make_train_test_split_datasets`
and `_make_timeseries_split_datasets` use pythonic parallel
pair-swap (`a, b = b, a` per slot pair); 4 lines per slot pair
instead of the 18-line 8-tuple parallel assignment.
Dead constant removal:
- `QuickAdapterRegressorV3._AGGREGATE_DISTANCE_METRICS_SET` is
absent at HEAD; the constant has no caller. The
`_CLUSTER_DENSITY_DISTANCE_METRICS_SET` block comment lists the
7 aggregate metrics inline for reference (`harmonic_mean`,
`geometric_mean`, `arithmetic_mean`, `quadratic_mean`,
`cubic_mean`, `power_mean`, `weighted_sum`).
Cosmetic:
- `_LABEL_KNOWN_AT_LOOKAHEAD_SUFFIX` placement is adjacent to
`LABEL_WEIGHT_SUFFIX` (both are label-aux column suffixes).
- `_known_at_lookahead` returns `int64` on both single-series and
multi-series paths (symmetric dtype contract).
- Schema-version reset log: `f"v{existing_schema_version!r}"`
renders unambiguously for non-int corrupt values (booleans,
strings).
Jérôme Benoit [Mon, 22 Jun 2026 01:54:59 +0000 (03:54 +0200)]
fix(quickadapter): route reversed train weights through support_policy (#98)
PR #85 added `_compose_train_weights_with_support` (gates training-set weights through `support_policy`) and `_compose_eval_weights` (eval-side, deliberately bypasses `support_policy`). The `reverse_train_test_order` path in `_make_train_test_split_datasets` and `_make_timeseries_split_datasets` swapped slices AT THE FINAL `build_data_dictionary` call -- AFTER weight composition -- so the actual-train slot received weights composed by `_compose_eval_weights` (silent bypass), and the actual-test slot received weights composed by `_compose_train_weights_with_support` (wrong direction, typically a no-op under `support_policy='fallback'` default).
Reachable only under `causal_mode=false` (deprecated; acausal baselines only) since `causal_mode=true` rejects `reverse_train_test_order=true` upfront.
Fix: perform the train/test slice swap BEFORE weight composition so the `train_*` and `test_*` identifiers map to their actual training roles throughout. Both call sites converge to a single `dk.build_data_dictionary` return; context strings in `support_policy` log/raise messages now reflect the true train/test role.
Add an upfront `ValueError` in `_make_train_test_split_datasets` when `test_size=0` AND `reverse_train_test_order=True`, mirroring the existing `causal_mode`/reverse rejection pattern. The `timeseries_split` path already rejects `test_size < 1` upstream of the swap.
Behavior change in the deprecated path: `support_policy='raise'` now correctly raises on actual-train insufficient support; `support_policy='fallback'` now correctly warns.
Reviewed by three parallel Oracle passes (math + algorithmics + scope/reachability; Python state-of-the-art + harmonization + implementation elegance; documentation + terminology + completeness) at design stage and again post-implementation, each citing upstream evidence from `freqtrade/freqai/`.
Follow-up from PR #80 review, deferred during PR #90.
Jérôme Benoit [Mon, 22 Jun 2026 01:34:16 +0000 (03:34 +0200)]
fix(quickadapter): prefix `compose_sample_weights` and `sanitize_and_renormalize` log entries with caller context (#97)
`compose_sample_weights` and `sanitize_and_renormalize` accept a
required keyword-only `context: str` parameter; every warning, error,
and inner `sanitize_and_renormalize` call uses `context` as its sole
prefix.
- `compose_sample_weights(..., *, logger, context, on_collapse=...)`:
`context: str` keyword-only, required. The 4 internal
warnings/errors (shape-mismatch `ValueError`, all-dropped
`LabelWeightSupportError`, sparse-mass warning, collapse-on-
survivors `LabelWeightSupportError` and fallback warning) prefix
with `{context}:`. Internal `sanitize_and_renormalize` calls
compose sub-contexts: `{context}:base_only`,
`{context}:label_weighted`, `{context}:base_fallback`. The sparse-
mass message reads `sparse weighting mass`, accurate for both
train and eval paths.
- `sanitize_and_renormalize(..., *, logger=None, context: str)`:
`context` keyword-only, required. The 5 warnings/errors
(`drop_mask` shape `ValueError`, `drop_mask` dtype `ValueError`,
rescale-overflow warning, weights-collapsed warning,
mask-covers-all warning) prefix with `{context}:`. The redundant
`(context=%s, ...)` subfield is dropped.
- 5 `compose_sample_weights` call sites in
`QuickAdapterRegressorV3.py` pass `context=context`. The 2
external `sanitize_and_renormalize` call sites
(`post_feature_pipeline:train`, `post_feature_pipeline:test`)
already pass `context=`.
Log format goes from
`compose_sample_weights: sparse training mass (59/2603 rows ...)`
to
`[ETH/USDT] train_test_split:train: sparse weighting mass (59/2603 rows ...)`.
The pair, split method, and train/eval side are traceable in the
log line.
Jérôme Benoit [Mon, 22 Jun 2026 01:21:28 +0000 (03:21 +0200)]
refactor(quickadapter): migrate _*_set() lru_cache family to Final[frozenset] (#94)
Each `@staticmethod @lru_cache(maxsize=None) _X_set()` family member
maps to a class-level `_X_SET: Final[frozenset[...]]` adjacent to its
backing tuple. Derived members express their content through set
algebra over the new `_*_SET` constants.
Call sites at `(cls|QuickAdapterRegressorV3)._X_set()` resolve to the
matching `_X_SET` reference. `optuna_samplers_by_namespace` returns
`tuple[frozenset[OptunaSampler], OptunaSampler]` to match the new
constant types. The surviving `@lru_cache(maxsize=8)` on
`optuna_samplers_by_namespace` falls outside the `_*_set` family and
stays in place.
Jérôme Benoit [Mon, 22 Jun 2026 01:20:11 +0000 (03:20 +0200)]
refactor(quickadapter): rename known_at_index to known_at_lookahead (#96)
PR #95 retained the historical column name `<label>_known_at_index` for what is now a per-row label lookahead in candles, to keep that hotfix strictly minimal. This PR converges the column suffix, the helper, the dataclass field, the static method, and the per-call-site locals onto `_known_at_lookahead`, with a retro-compat alias on the only externally-named public helper (`label_known_at_column_name = label_known_at_lookahead_column_name`).
The auxiliary `<label>_known_at_*` column is regenerated on every training run inside `set_freqai_targets`; FreqAI persists only the fitted model and `extra_returns_per_train`, never auxiliary dataframe columns -- the rename invalidates no on-disk artifact.
Reviewed by three parallel Oracle passes (math + claims-coherence; Python state-of-the-art + harmonization; documentation + terminology + PR-description coherence), each citing upstream evidence from `freqtrade/freqai/freqai_interface.py`, `data_kitchen.py`, and `data_drawer.py`. Consensus fixes were applied: README `causal_mode` formula symbol bound to the column token (`row-wise max(<label>_known_at_lookahead)`) to colocate definition with usage.
The two causal-guard local variable pairs were also harmonized to the local `train_<noun>` family (`train_known_at_lookahead`, `train_known_at_position`) used by the surrounding `_make_*_datasets` methods.
Jérôme Benoit [Mon, 22 Jun 2026 01:02:56 +0000 (03:02 +0200)]
fix(quickadapter): use slice-invariant lookahead for causal guard (#95)
* fix(quickadapter): use slice-invariant lookahead for causal guard
PR #78 stored '<label>_known_at_index' as 'arange(len) + horizon +
kernel_half_width' -- absolute positions in the dataframe passed to
'set_freqai_targets'. freqtrade's 'dk.slice_dataframe' (a '.loc' filter)
runs AFTER 'set_freqai_targets' and drops warmup rows but preserves
column values, so those pre-slice positions survived into the post-slice
'unfiltered_df'. The causal guard then compared them against
'first_test_position' derived from 'np.arange(len(unfiltered_df))' --
local post-slice positions in a different coordinate system. The unit
mismatch wiped out most or all training rows on every pair.
Production crash on 2026-06-22 (XRP/USD): "removed 2621
causal-unsafe train rows" followed by "causal guard removed all
train rows, skipping".
Fix: the column now stores a per-row label lookahead (in candles),
invariant under 'dk.slice_dataframe'. Consumers combine the row's
local position with the lookahead to recover the local known-at
position before comparing to 'first_test_position'. Column name
'<label>_known_at_index' is retained for this hotfix; a rename to
'<label>_known_at_lookahead' (with rétro-compatible alias) is left
to a follow-up PR per AGENTS.md 'small, verifiable changes'.
Touches:
- Utils.py: producer rewritten to store a constant per-row lookahead;
'LabelData' and 'label_known_at_column_name' docstrings document the
new contract; '_LABEL_KNOWN_AT_SUFFIX' carries an inline disambiguation.
- QuickAdapterV3.py: smoothing-lookahead advance comment harmonized to
the canonical 'per-row label lookahead (in candles)' phrasing.
- QuickAdapterRegressorV3.py: '_known_at_index' docstring rewritten;
'train_test_split' and 'timeseries_split' causal-mode branches add
'train_positions + delta' before the '< first_test_position' check;
'timeseries_split' hoists 'train_positions' for symmetry with
'train_test_split'.
- README.md: 'causal_mode' tunable description reflects the new
comparison semantic.
Reviewed by three parallel Oracle passes (math/algo/scope,
Python state-of-the-art / harmonization, documentation /
terminology / concision) with cross-validation; one false alarm
on a missing position-only fallback in 'timeseries_split' was
resolved by confirming 'TimeSeriesSplit.gap' enforces the
chronological purge at the sklearn layer.
* docs(quickadapter): shrink _known_at_index docstring to LabelData pointer
Per multi-oracle PR #95 review (Oracle 3 §8.1): paragraph 1 of
_known_at_index duplicated the slice-invariance rationale already
canonical on LabelData.known_at_index. Replace with a thin pointer per
AGENTS.md *No duplication: maintain single authoritative documentation
source; reference other sources rather than copying.*
Consolidates P1/P2 findings from `chatgpt-codex-connector` review
comments on PRs #78, #79, #80, #81, and PR #90.
Utils.py + label generation:
- `_generate_extrema_label` accepts `logger: Logger | None`; the
`LabelGenerator` type signature, `generate_label_data` dispatcher,
and `QuickAdapterV3.set_freqai_targets` caller propagate the logger.
`_generate_extrema_label` has no `F821 logger` undefined-name path.
- `register_label_generator` routes the input through
`_adapt_label_generator`. The adapter detects the canonical
`(dataframe, params, logger)` shape by a positional parameter named
`logger` at index 2 (with or without a default); other generators
with 2 required positional parameters are wrapped via
`functools.wraps` (preserves `__name__`/`__doc__`/`__wrapped__`) to
drop the logger argument at dispatch, with defaulted positionals
after index 1 left at their defaults. `ValueError` is raised at
registration for `*args`, `**kwargs`, keyword-only `logger`, fewer
than 2 required positionals, more than 3 required positionals, and 3
required positionals whose third name is not `logger`.
- `safe_divide` denominator zero-check uses exact-zero
(`denominator_arr != 0.0`); subnormal and satoshi-scale denominators
pass the gate, and non-finite division outputs coerce to `fallback`
via the post-division finite-mask.
Causal label split lookahead:
- `QuickAdapterV3.get_label_horizon_candles` recomputes the horizon
from the current `label_period_candles` (via
`get_label_period_candles`); init omits `label_horizon_candles`, so
HPO period updates propagate to the horizon. The regressor's
`_optuna_label_params` init likewise omits `label_horizon_candles`.
- `QuickAdapterV3.set_freqai_targets` advances `<label>_known_at_index`
by the smoothing kernel half-width after smoothing. The
`Utils.get_smoothing_kernel_half_width(config, *, series_length)`
helper reuses `get_odd_window`/`get_even_window`/`get_savgol_params`
shared with `smooth()` and dispatches on kernel routing:
- `filtfilt`-routed zero-phase kernels (members of
`SMOOTHING_KERNELS`: `gaussian`, `kaiser`, `kaiser_bessel_derived`,
`triang`): half-width `effective_window - 1` (forward+backward
pass extends dependency to the full filter length on both sides).
- Single-pass centered windows (`smm`, `sma`, `savgol`): half-width
`effective_window // 2`.
- `gaussian_filter1d`: `int(4.0 * sigma + 0.5)` matching
`scipy.ndimage` default truncation.
The helper returns 0 when `smooth()` itself short-circuits:
`series_length < max(window_candles, 3)` (top-level no-op) and, for
the filtfilt/savgol routes, `series_length < effective_window`
(downstream short-series no-op in `zero_phase_filter` /
`savgol_filter`). The `series_length` parameter is keyword-only and
required.
Label-weighting support policy:
- `QuickAdapterRegressorV3._compose_train_weights_with_support` routes
the zero-pivot case (label-weighting strategy configured but no
label weights available) through `_apply_support_policy`; the
support policy governs the contract: `raise` raises, `fallback`
warns.
Label Optuna selection hardening:
- `_OPTUNA_LABEL_SELECTION_SCHEMA_VERSION` is `2`, co-located with
`_OPTUNA_LABEL_BEST_PARAMS_SCHEMA_VERSION` in `Utils`. The two
constants are independent: wire format and selection algorithm
carry separate version axes.
- `_optuna_label_selection_metadata` rejects non-finite `label_weights`
/ `label_p_order` with `ValueError`; downstream dict equality on the
selection_metadata is NaN-safe.
- `_validate_optuna_label_best_params` accepts an
`expected_selection_metadata` keyword. It rejects files that are
not a `{schema_version, params, selection_metadata}` dict, files
with mismatched `schema_version`, files missing or with an invalid
`selection_metadata.schema_version`, and -- when
`expected_selection_metadata` is provided -- files whose stored
`selection_metadata` differs from the caller's current view. Legacy
unversioned best-params files (no `schema_version`) are rejected
outright. `QuickAdapterRegressorV3.optuna_load_best_params` passes
its `_optuna_label_selection_metadata()` view for the label
namespace; `QuickAdapterV3.optuna_load_best_params` omits the
keyword, since the strategy reads only `label_period_candles`,
`label_horizon_candles`, and `label_natr_multiplier` from the
best-params file.
- The study schema-migration branch resets the Optuna label study
whenever `existing_schema_version` is not an `int` (rejects `bool`
and other types) or differs from `target_version`; an unversioned
study (legacy pre-metadata) and a version-mismatched study are
treated identically. Trials selected under a different metric
whitelist or weighting scheme cannot be reused.
- `_optuna_label_selection_metadata` includes `label_weights` and
`label_p_order`; `_calculate_distances` consumes both, so the
idempotent `set_user_attr` write detects drift on those tunables.
- `_calculate_distances` validates `label_weights` length against the
original objective count (raises on mismatch) only on the slicing
path (when `objective_indices is not None` and the count differs).
The sliced vector falls back to uniform weights only when
`np.all(sliced == 0.0)` (the user's positive weights all on
dropped objectives); slices containing negative or non-finite
values flow through to `_validate_label_weights`.
- `_validate_label_selection_metric` accepts an `aggregate_allowed`
parameter; cluster/density category callers pass `False`,
restricting the valid set via the cached
`_cluster_density_distance_metrics_set()` classmethod
(SciPy-compatible non-probability metrics). The aggregate metrics
(`harmonic_mean`, `geometric_mean`, `arithmetic_mean`,
`quadratic_mean`, `cubic_mean`, `power_mean`, `weighted_sum`) live
in the cached `_aggregate_distance_metrics_set()` classmethod
derived by set-algebra from
`_distance_metrics_set() - _scipy_metrics_set() -
_probability_distance_metrics_set()`. README cluster/density metric
rows list the SciPy-compatible set. `compromise_programming`/
`topsis` accepts the full aggregate set via
`_calculate_trial_distance_to_ideal`.
Out of scope:
- Two PR #80 findings: `reverse_train_test_order` support-policy
routing; post-`feature_pipeline.fit_transform` support recheck.
Jérôme Benoit [Sun, 21 Jun 2026 21:45:02 +0000 (23:45 +0200)]
fix(quickadapter): stabilize label optuna selection (#81)
Multi-objective `label` Pareto best-trial selection in the
QuickAdapter regressor.
- Probability-style metrics (`jensenshannon`, `hellinger`,
`shellinger`) rejected for `label_distance_metric` /
`label_cluster_metric` / `label_density_metric` — Pareto objective
matrices are unbounded floats, not probability vectors. Invalid
values fall back to `euclidean` with a warning.
- Constant objective dimensions dropped before trial-distance
computation: a constant objective dimension is non-informative and
would bias the geometry. Tolerance at
`_NON_CONSTANT_OBJECTIVE_ATOL: Final[float] = 1e-8`.
- User-supplied `label_weights` matching the original objective count
slice to align with the non-constant subset; mismatched sizes flow
to `_validate_label_weights(mode="raise")`.
- Deterministic best-trial tie-break by `(distance, trial.number)`,
independent of `study.best_trials` ordering.
- All-constant Pareto front falls back to the lowest `trial.number`.
- Persisted Optuna `label` study user-attr `selection_metadata` nests
`schema_version`
(`_OPTUNA_LABEL_SELECTION_SCHEMA_VERSION: Final[int] = 1`) and
`method_config`. Studies without a recorded `schema_version` are
tagged at the current version on next `optuna_create_study` (trials
preserved); studies recording a different version are reset. The
`selection_metadata` write is idempotent: skipped when unchanged,
warned on diff.
- `OptunaNamespace` Literal and `_OPTUNA_NAMESPACES` (a `NamedTuple`
of `hp` and `label` with per-field literal types) live in `Utils`,
with `.hp` / `.label` accessors at all call sites
(`QuickAdapterRegressorV3`, `QuickAdapterV3`). No per-class tuple
alias, no inline `# "hp"` / `# "label"` annotations.
Jérôme Benoit [Sun, 21 Jun 2026 20:37:10 +0000 (22:37 +0200)]
fix(quickadapter): restore _format_collection ordering before register decorators
Rebase of the label-weight-support-policy PR onto main (atop the style line-wrap commit) reapplied the _format_collection hunk at the wrong location, placing the helper AFTER its @_format_value.register(list|tuple|set|dict|np.ndarray) decorators. Move the definition back to its canonical position above the dispatchers.
Jérôme Benoit [Sun, 21 Jun 2026 20:26:45 +0000 (22:26 +0200)]
feat(quickadapter): add label weight support policy (#85)
Per-row sample-weight composition decoupled from the final per-split
compose. Causal-guard filtering operates on raw base/label weights.
Configurable support thresholds gate the train-weight compose.
- `SampleWeightInputs` dataclass carries `(base, label,
label_weighting_config)` from `_build_sample_weight_inputs`;
`compose_sample_weights(base, label)` runs AFTER
`train_test_split`/`TimeSeriesSplit` AND AFTER causal guards.
`__post_init__` validates 1-D shape, base/label shape parity,
required `label_weighting_config` keys, and `support_policy` enum
membership.
- Thresholds in `freqai.label_weighting`:
`min_pivot_equivalent_count` (default 3),
`min_positive_label_weight_fraction` (default 0.01),
`min_effective_sample_size` (default 3.0; Kish ESS).
- `support_policy: enum {fallback, raise}` (default `fallback`)
drives the failure mode when any threshold trips. Eval (test/val)
weights bypass this policy by design.
- `compose_sample_weights` takes `on_collapse: Literal["raise",
"fallback"]` (default `"raise"`); the train path lets collapse raise
through `LabelWeightSupportError` so `support_policy` catches it,
the eval path passes `"fallback"` to preserve the label-derived
drop mask.
- `_apply_support_policy` (typed `policy: LabelWeightSupportPolicy`)
dispatches `fallback` and `raise` branches via `match`/`case` with
`assert_never` exhaustiveness; `compose_sample_weights` applies the
same pattern on `on_collapse`.
- `_shuffle_split_rows` 4-tuple shuffler covers label weights.
- `_filter_train_by_mask` accepts optional `train_label_weights` for
uniform causal-guard filtering across base+label weight arrays.
- `LabelWeightSupportSummary`, `_effective_sample_size` (Kish ESS),
and `summarize_label_weight_support` documented as operational spec.
README documents the four `label_weighting` rows;
`config-template.json` records the `fallback` default.
Shared finite-sample, guarded distribution-fit, safe divide/log-ratio,
and sigmoid-domain helpers. Log/division feature paths route through
the helpers; distribution fits guard empty, non-finite, and constant
samples.
- `Utils.py` helpers: `FiniteSample` dataclass with `finite_sample`;
`safe_distribution_fit` (documented fallback-length contract);
`safe_divide`; `safe_log_ratio`.
- `nan_average` finite/zero-weight guards; documented divergence from
`np.nanmean` (strips +/-inf as well as NaN; bounded for current
callers).
- `_clip_sigmoid_domain` in `LabelTransformer.py` guards
`sp.special.logit` against values outside the open `(-1, 1)` domain
during `sigmoid` inverse normalization.
- `feature_engineering_expand_basic` and Utils log/divide sites
(`top_log_return`, `bottom_log_return`, `price_retracement_percent`,
`ewo` normalize, `zigzag` log prices, KC/BB/VWAP widths) route
through the safe helpers.
- DI Weibull and label `norm` fits in `fit_live_predictions` use
`safe_distribution_fit`; DI cutoff fallback at
`_DI_CUTOFF_DEFAULT: Final[float] = 2.0`.
Jérôme Benoit [Sun, 21 Jun 2026 18:01:23 +0000 (20:01 +0200)]
feat(quickadapter)!: add causal label split foundation (#78)
Causal split guards on QuickAdapter training. Default causal mode
rejects `data_split_parameters.shuffle=true`,
`feature_parameters.shuffle_after_split=true`, and
`feature_parameters.reverse_train_test_order=true`.
- `feature_parameters.causal_mode` (default `true`): guard toggle.
`false` is deprecated.
- `feature_parameters.label_horizon_candles` (default
`label_period_candles`): candles after a label row before its label
is considered known by causal split guards. Fallback chain
`label_horizon_candles` -> `label_period_candles` -> `1`.
- `<label>_known_at_index` columns expose `LabelData.known_at_index`
per-row; multi-label boundary via element-wise max across present
columns.
- `timeseries_split` `gap` auto-set from `label_horizon_candles` under
causal mode; explicit `gap < label_horizon_candles` rejected.
- Persisted Optuna `label` best-params JSON has shape
`{schema_version, params}`
(`_OPTUNA_LABEL_BEST_PARAMS_SCHEMA_VERSION = 2`). Unversioned files
identified by shape; version-mismatched files emit distinct
"missing" vs "incompatible" warnings.
- `_label_aux_column_name` shared sigil-stripping helper backs
`label_weight_column_name` and `label_known_at_column_name`;
uniform collision guard against `&`/`%` and empty stem.
- `QuickAdapterRegressorV3.version = 3.12.0`.
BREAKING CHANGE: `feature_parameters.causal_mode` defaults to `true`.
Configs with `data_split_parameters.shuffle=true`,
`feature_parameters.shuffle_after_split=true`, or
`feature_parameters.reverse_train_test_order=true` raise at training
time.
Add a fourth off-pivot weighting mode that superposes the epsilon
floor and the gaussian bumps additively, and fix a related defect in
_scatter_weights that allowed pivot rows to sit *below* the off-pivot
field whenever that field at the pivot's index could legitimately
exceed the pivot's raw weight.
with phi = eps * B(W) (B mean or median, eps in [0, 1]; phi = 0 on
empty pivots or non-finite baseline), and per-pivot sigma_p from
_compute_pivot_sigmas (fixed or k-NN). The combined formulation reuses
both existing closed forms verbatim:
Bound: phi <= f(i) <= phi + max_p w_p. The new mode reduces to pure
gaussian when eps = 0 (bit-identical). The reduction is a per-row max
over per-pivot Gaussian bumps; phi is the epsilon floor.
Before this change _scatter_weights wrote out[p] = w_p unconditionally,
so a pivot whose raw weight was below the off-pivot field at its
index appeared as a sharp dip relative to its neighbors. Two
manifestations of the same defect class:
- 'epsilon' / 'epsilon_gaussian' (sub-floor): a pivot with
w_p < phi (e.g. W = (0.001, 1.0, 1.0) with eps = 0.5 and
B = median, phi = 0.5) sat at 0.001 while neighbor rows sat at phi.
- 'gaussian' / 'epsilon_gaussian' (sub-bump): a weak pivot with a
strong neighbor (e.g. W = (0.001, 1.0) at indices (0, 1), sigma = 1)
sat at 0.001 while the off-pivot field at the pivot's own index was
1.0 * exp(-0.5) ~= 0.6065 (the neighbor's gaussian bump).
Both cases are corrected by a single uniform change: _scatter_weights
now writes out[p] = max(w_p, fill[p]) so pivot rows are never written
below the off-pivot field. 'zero' is bit-identical (fill is always 0,
so max(w_p, 0) = w_p when w_p >= 0). 'gaussian' in the sparse-pivot
regime (the typical configuration, especially with k-NN bandwidth) is
also bit-identical because fill[p] equals w_p when no neighbor's bump
at p exceeds w_p.
Implementation
--------------
- _scatter_weights: pivot rows take np.maximum(weights, fill_weights)
unconditionally. Off-pivot rows unchanged.
- _compute_epsilon_floor (renamed from _epsilon_floor): extracted
helper that returns phi (mean / median / fallback). Reused by
'epsilon' and 'epsilon_gaussian'. Parameter baseline narrowed to
the FillEpsilonBaseline Literal type.
- _compute_gaussian_bumps (renamed from _gaussian_bumps): extracted
adapter over _gaussian_fill_weights. Reused by 'gaussian' and
'epsilon_gaussian'. logger is kwarg-only.
- compute_label_weights: dispatcher gains the FILL_METHODS[3] branch.
The combined branch computes bumps once and adds phi in-place via
np.add(out=fill_weights), keeping peak memory at the existing
(chunk, M) buffer; phi is constant in p so the post-reduction add
is algebraically identical to adding inside the chunk loop while
saving O(chunk * M) writes. ValueError messages tightened to
include 'supported values are ...' for parity with
_compute_pivot_sigmas and _aggregate_metrics.
- LabelTransformer.py: extends FillMethod Literal and FILL_METHODS
tuple with 'epsilon_gaussian' at index 3. No new tunables, no new
validators (the existing _EnumValidator(FILL_METHODS) picks up the
new value automatically; existing range / type validators on
fill_epsilon / fill_sigma_* / fill_bandwidth_* apply unchanged).
- QuickAdapterV3.py: logging block refactored from if/elif chain to
parallel if blocks keyed on tuple membership so epsilon and sigma
parameter groups emit independently for each mode that uses them.
Documentation
-------------
README cells updated with set-membership 'Ignored when ...' clauses
matching the new index sets (epsilon | epsilon_gaussian for the
floor parameters, gaussian | epsilon_gaussian for the kernel
parameters). The fill_method description names the additive
composition explicitly and the pivot-row lift invariant
(out[p] = max(w_p, f(p))).
Verified manually on the host via AST extraction harness (no automated
test infrastructure exists in quickadapter/):
- zero mode: bit-exact with prior code (fill is 0, max(w_p, 0) = w_p).
- gaussian mode, sparse pivots: bit-identical to prior code (no
neighbor's bump at p exceeds w_p, so the lift is a no-op).
- gaussian mode, neighbor-dominated regime: pivot rows lifted to the
local field max, fixing the sub-bump dip. Verified with the
counterexample W = (0.001, 1.0) at indices (0, 1), sigma = 1:
legacy out[0] = 0.001, fixed out[0] = 1.0 * exp(-0.5) ~= 0.6065.
- epsilon back-compat (above-floor pivots): phi = eps * mean(W)
reproduced; pivots above phi unchanged.
- epsilon pivot-dip fix: W = (0.001, 1.0, 1.0), eps = 0.5,
baseline = median; legacy out[0] = 0.001, fixed out[0] = phi = 0.5.
- epsilon_gaussian with eps = 0: bit-identical to pure gaussian.
- epsilon_gaussian additive decomposition: out_eg - out_g = phi at
every off-pivot row.
- epsilon_gaussian pivot-row lifted: W = (0.001, 1.0, 1.0) at
well-separated indices (e.g. (0, 100, 200)), eps = 0.5,
baseline = median, sigma = 2.0; out[0] = phi + 0.001 ~= 0.501
(was 0.001 before the scatter fix).
- empty pivots: all four modes return all-zero.
- negative pivot weights still rejected by _gaussian_fill_weights.
- knn bandwidth + epsilon_gaussian: finite, bounded below by phi.
- ValueError messages on invalid fill_method / fill_epsilon_baseline
include 'supported values are ...'.
Jérôme Benoit [Thu, 4 Jun 2026 22:29:58 +0000 (00:29 +0200)]
feat(label_weighting): adaptive k-NN bandwidth for gaussian off-pivot fill (#77)
* feat(label_weighting): adaptive k-NN bandwidth for gaussian off-pivot fill
Address the crushing of weaker pivots by stronger neighbors when pivots
fall within ~sigma_candles of each other in fill_method='gaussian'. The
per-row max aggregator preserves the upper bound Out[i] <= max_p w_p
but a wide constant sigma lets a strong neighbor's Gaussian dominate a
weak pivot's tail.
Add a k-nearest-neighbor bandwidth selector (Loftsgaarden &
Quesenberry 1965; Silverman 1986, paragraph 5.2) that adapts each
pivot's sigma to local pivot density:
where d_k(p) is the index distance to the k-th pivot neighbor. The
upper bound on Out[i] is preserved (no over-amplification) and dense
clusters automatically contract their Gaussians to stop overlapping.
Implementation:
- Pivots are emitted chronologically by zigzag, so the 1D k-NN reduces
to a sliding k-window over sorted indices, O(M) without a spatial
index.
- _gaussian_fill_weights accepts a per-pivot sigma vector via NumPy
broadcasting; the existing chunked exp/multiply/max kernel is
unchanged.
- Default fill_bandwidth='fixed' preserves byte-for-byte the previous
algorithm.
Adds "uniform" to WEIGHT_STRATEGIES (between "none" and the metric
names) which assigns weight=1.0 to every detected pivot. Off-pivot rows
remain governed by the existing fill_method (zero / epsilon / gaussian),
so uniform + gaussian collapses cleanly to a pure proximity kernel
around each pivot. Naming follows sklearn convention
(KNeighbors(weights="uniform"), DummyClassifier(strategy="uniform")).
* chore(quickadapter): bump strategy and regressor version 3.11.11 -> 3.11.12
* refactor(weights): hoist indices_array and valid_mask in compute_label_weights
Compute indices_array and valid_mask once at the top of the function
instead of after the strategy dispatch. The uniform branch can now use
indices_array.size instead of len(indices), and the duplicate
np.asarray / valid_mask construction lower in the function is removed.
Saves one np.asarray and one mask computation per call.
Drop the redundant indices: list[int] parameter now that the only
caller (compute_label_weights) hoists indices_array and valid_mask.
The function takes them positionally and uses indices_array.size for
size checks, removing three len(indices) calls and the optional-kwarg
fallback paths.
* refactor(weights): pipeline API consolidation pass
- compute_label_weights: drop Optional placeholder, accept
Sequence[int] | NDArray[np.integer] for indices
- standardize Optional[X] -> X | None across module (PEP 604)
- _impute_weights: positional call instead of keyword on single arg
- _pivot_equivalent_count: remove unreachable threshold <= 0 branch
(survivors.size > 0 implies survivors.max() > 0 because the input
has been sanitized to non-negative values upstream)
- _scatter_weights: drop dead 'if not np.any(valid_mask)' early
return; vectorized assignment is a no-op when the mask is all-False
- sanitize_and_renormalize: clarify empty-input semantics in docstring
* fix(weights): zero leading and trailing non-finite runs in _impute_weights
The boundary mask only covered the strict tip positions (index 0 and -1),
so multi-element non-finite runs at the boundary were median-imputed
instead of zeroed. With input [NaN, NaN, 1.0, 2.0, NaN, NaN] the function
returned [0.0, 1.5, 1.0, 2.0, 1.5, 0.0] instead of [0, 0, 1.0, 2.0, 0, 0],
silently extending pivot weight to the unconfirmed boundary candles.
Use np.argmax on the finite mask to detect the leading and trailing
non-finite runs and zero the entire run, matching the docstring contract.
* fix(weights): floor stacked metrics in geometric and harmonic aggregation
Power means with p<=0 collapse to 0 on a single zero in the stack:
pmean([1, 0, 3], p=-1) = 0.0 and pmean([1, 0, 3], p=0) = 0.0. Combined
with compose_sample_weights' (arr <= 0) drop_mask predicate, a single
metric returning 0 on a pivot silently drops that row entirely.
Floor stacked_metrics at np.finfo(float).tiny only inside the
geometric_mean and harmonic_mean branches so all-positive pivots
survive aggregation. arithmetic_mean, quadratic_mean, weighted_median
and softmax branches are untouched.
* fix(weights): log when out-of-range pivot indices are dropped
compute_label_weights silently filters out pivot indices outside
[0, n_values) via valid_mask. This made upstream contract violations
invisible: a stale or off-by-one index list would simply produce zero
training weight on those rows with no diagnostic.
Emit logger.warning with the count and dropped fraction whenever
n_dropped > 0 so the upstream caller can spot the issue.
* refactor(weights): collapse 4x label-config validators into a registry
Replace the four near-identical _validate_*_params + get_label_*_config
pairs with a single _LABEL_KIND_REGISTRY mapping each kind name to
(specs, defaults). _label_kind_validator builds the validator on the
fly and get_label_kind_config dispatches to _get_label_config with the
appropriate spec/default pair. The four public get_label_*_config
helpers remain as thin wrappers so existing callers in QuickAdapterV3
and QuickAdapterRegressorV3 are unaffected.
_LabelTransformerConfig.from_dict (LabelTransformer.py) is intentionally
out of scope: it would require propagating a logger through
BaseTransform's freqtrade-side interface, which is upstream-controlled.
* docs(weights): drop verbose empty-input note from sanitize_and_renormalize
The added sentence paraphrased the existing collapse line and restated
obvious facts about zero-length vectors without contractual information.
Revert to the concise pre-W1 docstring.
* docs(weights): drop get_label_kind_config docstring for consistency
The 4 sibling get_label_*_config wrappers have no docstrings; their
parameter names and the registry name self-document the contract. Drop
the redundant docstring on get_label_kind_config to match the family
style.
* fix(weights): drop tiny floor in geometric/harmonic aggregation
The floor at np.finfo(float).tiny added in b1f86a0 preserved pivots
whose metrics included an exact zero, but a zero metric is itself a
'signal absent' marker that downstream compose_sample_weights drops
via the (arr <= 0) mask. Floor was masking the intended drop.
Restore the upstream pmean behavior so a zero in any geometric or
harmonic input produces an exact 0.0 combined weight, allowing
drop_mask to drop the pivot as designed.
Jérôme Benoit [Mon, 25 May 2026 16:05:22 +0000 (18:05 +0200)]
feat(quickadapter): log new label_weighting fill_* tunables
Mirror the existing softmax_temperature conditional-logging pattern:
always log fill_method, then log fill_epsilon and fill_epsilon_baseline
only under fill_method == "epsilon", and fill_sigma_candles only under
fill_method == "gaussian". Keeps the per-column "Weighting:" log block
consistent with the resolved config and avoids printing tunables that
have no effect under the active mode.
Jérôme Benoit [Mon, 25 May 2026 15:52:05 +0000 (17:52 +0200)]
feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting (#74)
* feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting
Adds three off-pivot weighting modes behind a new fill_method tunable in
freqai.label_weighting:
- zero (default): current hard-zero behavior, retained for backward
compatibility.
- epsilon: off-pivot rows receive a flat baseline
fill_epsilon * <baseline>(pivot_weights), where <baseline> is mean or
median, controlled by fill_epsilon_baseline.
- gaussian: off-pivot rows receive a per-row weight from a heatmap-style
decay max_p w_p * exp(-(i-p)^2 / (2 sigma^2)), controlled by
fill_sigma_candles (>= 0.5).
The default is zero so existing configs without the new keys behave
identically. Switching fill_method materially changes per-leaf weight
mass and may require GBM hyperparameter retuning; flagged in the README
description column.
Implementation:
- Adds FillMethod/FILL_METHODS and FillEpsilonBaseline/FILL_EPSILON_BASELINES
Literal types and tuples in LabelTransformer.py.
- Extends DEFAULTS_LABEL_WEIGHTING with the four new keys and their
defaults.
- Extends _WEIGHTING_SPECS in Utils.py with corresponding _EnumValidator
and _NumericValidator entries (epsilon in [0, 1], sigma_candles >= 0.5).
- Refactors _scatter_weights to accept fill_weights as a precomputed
array plus optional indices_array/valid_mask kwargs; preserves
pre-existing length-mismatch ValueError and empty-input early-return
semantics.
- Adds _gaussian_fill_weights helper with in-place pipeline keeping peak
memory at one (chunk, M) buffer; chunk-by-N keyed on
_GAUSSIAN_FILL_CHUNK_BUDGET = 50_000_000 cells (~400 MB peak); emits a
density warning when M / N > 0.1; rejects negative pivot weights.
- Adds *, logger: Logger keyword-only parameter to compute_label_weights
and updates the single call site in QuickAdapterV3.py.
- Replaces the raw nonzero count in compose_sample_weights with a
pivot-equivalent count helper (_pivot_equivalent_count) so the sparse
training mass warning stays meaningful under epsilon / gaussian.
Documentation:
- Four new rows added to the README configuration tunables table under
Label weighting; fill_method flagged as requiring trained-model
deletion when changed.
- Four new keys added to config-template.json under label_weighting.
Verified manually on host via AST extraction harness (no automated test
infrastructure exists in quickadapter/):
- STATIC_OK: defaults + tuples assertions pass.
- SPOTCHECK_4..9: cluster amplification (out[50] ~= 8.0), sigma < 0.5
rejected, negative pivot weights rejected, density warning emitted at
M/N=0.2, empty pivots return zeros, mean/median epsilon ratio = 20.8x.
- SPARSE_4A..C: sparse-mass warning fires under zero mode + sparse
pivots and gaussian sigma=0.5 underflow regime; silent under broad
gaussian fills.
* fix(quickadapter): harden sanitize_and_renormalize against rescale overflow and drop_mask contract violations
Two production-quality safeguards on the load-bearing primitive used by
the new fill_method dispatch in PR #74, plus one cosmetic comment
cleanup.
1. Subnormal-total rescale overflow guard:
When the sum of sanitized weights falls into a subnormal range
(e.g. a single 1e-310 survivor among zeros, n=1000), n/total
overflows to +Inf and safe * Inf propagates Inf to every nonzero
entry, producing mean(out) = NaN and silently violating the
documented mean=1 invariant. The fix computes the rescale factor
into a local c, checks np.isfinite(c), and falls through to the
existing uniform-fallback path with a distinct warning message
('rescale factor non-finite') so operators can distinguish this
regime from the existing 'weights collapsed' case. Bit-identical
on all common paths; c -> 0 underflow is unreachable
(min c = 1/DBL_MAX > 0).
2. drop_mask shape and dtype assertions:
sanitize_and_renormalize is now load-bearing for compose_sample_weights
under all three fill_method modes (zero/epsilon/gaussian). Numpy
broadcasts a (k, n)-shaped mask silently, breaking the (n,) output
contract. Shape and dtype precondition checks raise ValueError
early with prefixed messages matching the function's existing
logger style. Dtype check uses np.issubdtype(..., np.bool_) so
any boolean alias (bool, np.bool_, 'bool') is accepted; integer
masks are rejected.
3. LabelTransformer.py: replace 'current behavior' comment with
'default' on FILL_METHODS[0] since the comparison no longer makes
sense once the PR is merged.
Verified manually:
- REVIEW_FIX_1A..C: bool, np.bool_, 'bool' all accepted; int rejected.
- REVIEW_FIX_2A..B: subnormal-overflow path emits the new distinct
warning; real collapse path emits the original warning.
- REVIEW_FIX_3_OK: docstring contradiction removed.
- REGRESSION_OK: bit-identical common path.
- All original PR #74 verifications still pass (SPARSE_4A..C,
SPOTCHECK_4..9).
* fix(quickadapter): short-circuit compute_label_weights on empty pivot weights
When metrics[strategy] is empty but indices is non-empty, the new
fill_method dispatch in epsilon/gaussian arms slices weights[valid_mask]
before _scatter_weights can short-circuit, raising IndexError on a
size-0 / N-mask shape mismatch. Pre-PR _scatter_weights returned the
default-filled array silently in this case (preserved invariant noted
inline at the empty-input early return).
Add a short-circuit before the dispatch so the contract is consistent
across all three fill methods.
Also trim _gaussian_fill_weights docstring to match the codebase style
(neighboring private helpers carry no docstring or a single short
paragraph) and drop a redundant in-line comment that the in-place
np.multiply(out=buf) pattern already conveys.
Verified on the AST-extraction harness (pre-fix reproduction → fix
verification): 12 contract assertions across 4 edge cases x 3 fill
methods, plus crossmode + non-empty differentiation, all pass; PR #74
SPOTCHECK_4..9, SPARSE_4A..C, REVIEW_FIX_*, REGRESSION_OK still pass.
* chore(quickadapter): bump strategy and regressor version 3.11.10 -> 3.11.11
* refactor(quickadapter): polish label_weighting docs, comments, and sparse-mass diagnostic
Three coordinated polish edits following final-review feedback:
1. _pivot_equivalent_count: replace the 0.5 * median threshold with
_PIVOT_EQUIVALENT_MAX_FRACTION * surviving max (default 0.1). The
median-based heuristic saturated at N under epsilon mode (off-pivot
floor dominates the median once N >> M), silencing the warning the
docstring claimed to provide. The max-relative threshold separates
pivot-class rows from off-pivot fill across the bimodal regimes
fill_method introduces. Constant is module-level and named so the
choice is auditable; warning text now self-describes the threshold
('rows above 10% of surviving max').
2. _scatter_weights: trim the 'Order matters...' comment from 3 lines
to 1 line. The shorter form pins the intentional ordering without
paraphrasing git history; future 'validate inputs first' refactors
are still flagged.
3. README: extend the fill_method row with a concise retuning hint
(per-leaf regularization + Optuna study reset) so the operator
guidance surfaces in user docs, not only in the planning artefact.
Tighten fill_sigma_candles description to match neighboring-row
density.
Verified manually:
- SPARSE_4A..C: original PR cases still pass.
- SPARSE_4D: epsilon+sparse pivots (M=20, N=1000) now correctly fires
the sparse-mass warning (was silenced with median-based threshold).
- SPARSE_4E: zero+skewed pivots ([1,1,...,10]) still fire under the
new threshold (no regression on the skew case).
- SPOTCHECK_4..9, BUG_74_FIX_*, REVIEW_FIX_*, REGRESSION_OK: all
unchanged.
Jérôme Benoit [Mon, 25 May 2026 12:01:55 +0000 (14:01 +0200)]
style(weights): use !r repr for strategy in compute_label_weights ValueError
Replace hardcoded 'none' literal with f"{strategy!r}" to match the
codebase convention of !r-formatting external string identifiers in
error messages and avoid silent drift if WEIGHT_STRATEGIES[0] changes.
Jérôme Benoit [Mon, 25 May 2026 03:18:52 +0000 (05:18 +0200)]
fix(plot): switch raw direction/weight to bar with steelblue color
Raw discrete signals (direction +/-1/0, weight at pivots) are sparse and
read better as bars than lines. Steelblue contrasts with the orange
smoothed lines for clearer visual separation.
Jérôme Benoit [Mon, 25 May 2026 02:38:25 +0000 (04:38 +0200)]
fix(weights): canonical sanitize_and_renormalize and compose_sample_weights
Derived from independent dual-oracle mathematical specification with
proofs (mean=1 invariant, drop preservation, idempotency, collapse
degradation chain).
sanitize_and_renormalize:
- Fix latent bug: fallback path with non-empty drop_mask returned ones
zeroed at drop_mask but did not renormalize, breaking the mean=1
contract. The fallback now renormalizes so mean(out) == 1 holds on
surviving rows.
- Replace .copy()+mutation with np.where for drop_mask application.
compose_sample_weights:
- Replace the post-compose combined.sum() guard (which duplicated the
predicate sanitize_and_renormalize re-evaluates internally) with a
single survivor-aware predicate covering drop_mask | ~isfinite | <=0
in one pass. The check is the explicit branch point for the base-
weights fallback when the label-weighted product collapses on
surviving rows; this preserves the recency signal and the label-
derived drop_mask instead of degrading to uniform.
- Warn when nonzero/n falls below SPARSE_TRAINING_MASS_THRESHOLD (5%,
module-level constant) so operators can spot the sparse-training
regime that pivot-only weights produce on long series with few pivots.
QuickAdapterV3._log_strategy_configuration:
- Warn at startup when label_smoothing.method is 'smm' or 'savgol'
(with polyorder>=2) combined with a non-'none' label_weighting
strategy, since these kernels can collapse a sparse weight signal
and trip the all-rows-dropped guard.
Jérôme Benoit [Mon, 25 May 2026 01:39:28 +0000 (03:39 +0200)]
fix(weights): pivot-only sample weights when label_weighting is active
Replace the full-series median fill with 0.0 in compute_label_weights so
non-pivot rows carry no sample weight when a label_weighting strategy is
configured. The median fill predates PR #72: when weights multiplied the
label, the fill was inert (label=0 × median=0). Once weights became the
sample_weight kwarg of model.fit, the fill silently leaked the median
into the training loss for every non-pivot row, diluting the pivot
detection signal the model is being trained for.
Concretely, training now concentrates on pivots and their smoothed
neighborhoods (via label_smoothing), and the raw/smoothed weight plots
both render clean profiles starting from zero.
strategy='none' (the default) is unaffected: compute_label_weights still
returns a uniform 1.0 vector and every row contributes equally.
Jérôme Benoit [Mon, 25 May 2026 00:45:18 +0000 (02:45 +0200)]
fix(weights): guard DI_values None and read label_frequency_candles from freqai_info
- fit_live_predictions: pred_df.get('DI_values') returns None when
feature_parameters.DI_threshold is 0 or absent (the default), causing
AttributeError on the subsequent .mean()/.std() calls. Fall back to
zeros instead.
- _label_frequency_candles: read from self.freqai_info['feature_parameters']
(matching every other access in the file) instead of self.config, which
is the top-level config dict and never contains feature_parameters
directly. The previous code silently ignored user-provided values and
always fell back to the default 'max(2, 2 * len(self.pairs))'.
Jérôme Benoit [Mon, 25 May 2026 00:31:49 +0000 (02:31 +0200)]
fix(weights): tighten observability and edge-case handling in label pipeline
- sanitize_and_renormalize accepts logger/context kwargs and warns on
uniform-fallback collapse; six call sites in QuickAdapterRegressorV3
thread their stage label (train_test_split / post_feature_pipeline /
timeseries_split, train|test).
- Warn at startup when label_prediction.method='none' for any label, since
populate_entry_trend would silently never trigger.
- Replace .notna() with np.isfinite() in the smoothed-weight clip so +Inf
produced by smoothing kernels is also zeroed instead of relying on the
downstream drop_mask in compose_sample_weights.
- _impute_weights tracks boundary NaN separately so injected zeros do not
bias the interior median; finite endpoints are now preserved.
Jérôme Benoit [Mon, 25 May 2026 00:04:41 +0000 (02:04 +0200)]
refactor(weights): collapse compose_sample_weights to single-target API
LABEL_COLUMNS is single-target by design, so the dict-shaped per-label
map and row-wise aggregation in compose_sample_weights were dead
plumbing. Flatten the signature to a single label_weights vector and
read LABEL_COLUMNS[0] directly in _compose_per_row_weights. Drop the
duplicate-column guard (unreachable under single-target). Align caller
naming on base_weights to match the callee parameter. Add a defensive
check that LABEL_COLUMNS[0] is in dk.label_list to fail loudly if the
project label constant ever diverges from freqtrade's runtime view.
repositories / freqai-strategies.git / log
