feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting (#74)
* feat(quickadapter): add soft off-pivot weighting (epsilon, gaussian) to label_weighting
Adds three off-pivot weighting modes behind a new fill_method tunable in
freqai.label_weighting:
- zero (default): current hard-zero behavior, retained for backward
compatibility.
- epsilon: off-pivot rows receive a flat baseline
fill_epsilon * <baseline>(pivot_weights), where <baseline> is mean or
median, controlled by fill_epsilon_baseline.
- gaussian: off-pivot rows receive a per-row weight from a heatmap-style
decay max_p w_p * exp(-(i-p)^2 / (2 sigma^2)), controlled by
fill_sigma_candles (>= 0.5).
The default is zero so existing configs without the new keys behave
identically. Switching fill_method materially changes per-leaf weight
mass and may require GBM hyperparameter retuning; flagged in the README
description column.
Implementation:
- Adds FillMethod/FILL_METHODS and FillEpsilonBaseline/FILL_EPSILON_BASELINES
Literal types and tuples in LabelTransformer.py.
- Extends DEFAULTS_LABEL_WEIGHTING with the four new keys and their
defaults.
- Extends _WEIGHTING_SPECS in Utils.py with corresponding _EnumValidator
and _NumericValidator entries (epsilon in [0, 1], sigma_candles >= 0.5).
- Refactors _scatter_weights to accept fill_weights as a precomputed
array plus optional indices_array/valid_mask kwargs; preserves
pre-existing length-mismatch ValueError and empty-input early-return
semantics.
- Adds _gaussian_fill_weights helper with in-place pipeline keeping peak
memory at one (chunk, M) buffer; chunk-by-N keyed on
_GAUSSIAN_FILL_CHUNK_BUDGET = 50_000_000 cells (~400 MB peak); emits a
density warning when M / N > 0.1; rejects negative pivot weights.
- Adds *, logger: Logger keyword-only parameter to compute_label_weights
and updates the single call site in QuickAdapterV3.py.
- Replaces the raw nonzero count in compose_sample_weights with a
pivot-equivalent count helper (_pivot_equivalent_count) so the sparse
training mass warning stays meaningful under epsilon / gaussian.
Documentation:
- Four new rows added to the README configuration tunables table under
Label weighting; fill_method flagged as requiring trained-model
deletion when changed.
- Four new keys added to config-template.json under label_weighting.
Verified manually on host via AST extraction harness (no automated test
infrastructure exists in quickadapter/):
- STATIC_OK: defaults + tuples assertions pass.
- SPOTCHECK_4..9: cluster amplification (out[50] ~= 8.0), sigma < 0.5
rejected, negative pivot weights rejected, density warning emitted at
M/N=0.2, empty pivots return zeros, mean/median epsilon ratio = 20.8x.
- SPARSE_4A..C: sparse-mass warning fires under zero mode + sparse
pivots and gaussian sigma=0.5 underflow regime; silent under broad
gaussian fills.
* fix(quickadapter): harden sanitize_and_renormalize against rescale overflow and drop_mask contract violations
Two production-quality safeguards on the load-bearing primitive used by
the new fill_method dispatch in PR #74, plus one cosmetic comment
cleanup.
1. Subnormal-total rescale overflow guard:
When the sum of sanitized weights falls into a subnormal range
(e.g. a single 1e-310 survivor among zeros, n=1000), n/total
overflows to +Inf and safe * Inf propagates Inf to every nonzero
entry, producing mean(out) = NaN and silently violating the
documented mean=1 invariant. The fix computes the rescale factor
into a local c, checks np.isfinite(c), and falls through to the
existing uniform-fallback path with a distinct warning message
('rescale factor non-finite') so operators can distinguish this
regime from the existing 'weights collapsed' case. Bit-identical
on all common paths; c -> 0 underflow is unreachable
(min c = 1/DBL_MAX > 0).
2. drop_mask shape and dtype assertions:
sanitize_and_renormalize is now load-bearing for compose_sample_weights
under all three fill_method modes (zero/epsilon/gaussian). Numpy
broadcasts a (k, n)-shaped mask silently, breaking the (n,) output
contract. Shape and dtype precondition checks raise ValueError
early with prefixed messages matching the function's existing
logger style. Dtype check uses np.issubdtype(..., np.bool_) so
any boolean alias (bool, np.bool_, 'bool') is accepted; integer
masks are rejected.
3. LabelTransformer.py: replace 'current behavior' comment with
'default' on FILL_METHODS[0] since the comparison no longer makes
sense once the PR is merged.
Verified manually:
- REVIEW_FIX_1A..C: bool, np.bool_, 'bool' all accepted; int rejected.
- REVIEW_FIX_2A..B: subnormal-overflow path emits the new distinct
warning; real collapse path emits the original warning.
- REVIEW_FIX_3_OK: docstring contradiction removed.
- REGRESSION_OK: bit-identical common path.
- All original PR #74 verifications still pass (SPARSE_4A..C,
SPOTCHECK_4..9).
* fix(quickadapter): short-circuit compute_label_weights on empty pivot weights
When metrics[strategy] is empty but indices is non-empty, the new
fill_method dispatch in epsilon/gaussian arms slices weights[valid_mask]
before _scatter_weights can short-circuit, raising IndexError on a
size-0 / N-mask shape mismatch. Pre-PR _scatter_weights returned the
default-filled array silently in this case (preserved invariant noted
inline at the empty-input early return).
Add a short-circuit before the dispatch so the contract is consistent
across all three fill methods.
Also trim _gaussian_fill_weights docstring to match the codebase style
(neighboring private helpers carry no docstring or a single short
paragraph) and drop a redundant in-line comment that the in-place
np.multiply(out=buf) pattern already conveys.
Verified on the AST-extraction harness (pre-fix reproduction → fix
verification): 12 contract assertions across 4 edge cases x 3 fill
methods, plus crossmode + non-empty differentiation, all pass; PR #74
SPOTCHECK_4..9, SPARSE_4A..C, REVIEW_FIX_*, REGRESSION_OK still pass.
* chore(quickadapter): bump strategy and regressor version 3.11.10 -> 3.11.11
* refactor(quickadapter): polish label_weighting docs, comments, and sparse-mass diagnostic
Three coordinated polish edits following final-review feedback:
1. _pivot_equivalent_count: replace the 0.5 * median threshold with
_PIVOT_EQUIVALENT_MAX_FRACTION * surviving max (default 0.1). The
median-based heuristic saturated at N under epsilon mode (off-pivot
floor dominates the median once N >> M), silencing the warning the
docstring claimed to provide. The max-relative threshold separates
pivot-class rows from off-pivot fill across the bimodal regimes
fill_method introduces. Constant is module-level and named so the
choice is auditable; warning text now self-describes the threshold
('rows above 10% of surviving max').
2. _scatter_weights: trim the 'Order matters...' comment from 3 lines
to 1 line. The shorter form pins the intentional ordering without
paraphrasing git history; future 'validate inputs first' refactors
are still flagged.
3. README: extend the fill_method row with a concise retuning hint
(per-leaf regularization + Optuna study reset) so the operator
guidance surfaces in user docs, not only in the planning artefact.
Tighten fill_sigma_candles description to match neighboring-row
density.
Verified manually:
- SPARSE_4A..C: original PR cases still pass.
- SPARSE_4D: epsilon+sparse pivots (M=20, N=1000) now correctly fires
the sparse-mass warning (was silenced with median-based threshold).
- SPARSE_4E: zero+skewed pivots ([1,1,...,10]) still fire under the
new threshold (no regression on the skew case).
- SPOTCHECK_4..9, BUG_74_FIX_*, REVIEW_FIX_*, REGRESSION_OK: all
unchanged.