From: Jérôme Benoit Date: Thu, 30 Jul 2026 19:19:16 +0000 (+0200) Subject: perf(quickadapter): harmonize causal availability for provably-stable zero-weight... X-Git-Url: https://git.piment-noir.org/?a=commitdiff_plain;h=30898b98c447ab45dbbc178d3b6a8545fe21b548;p=freqai-strategies.git perf(quickadapter): harmonize causal availability for provably-stable zero-weight imputations (#186) * perf(quickadapter): harmonize causal availability for provably-stable zero-weight imputations Release trailing single-metric and combined leading non-finite runs whose imputation is provably fixed at 0.0 under the causal prefix at their true stabilization candle instead of the frame boundary, matching the existing single-metric leading treatment (#158). - Gap 1: a trailing single-metric non-finite run imputes to 0.0 and stabilizes at the first finite pivot's confirmation; release it there. An all-non-finite metric keeps the nonzero legacy default (1.0) and stays deferred to n, so a dedicated trailing_stable_mask distinguishes it from dependency_mask. - Gap 2: for combined with every component leading with a non-finite run, the aggregate is stably zero over [0, min_c first_finite_c); release it at the max-over-components confirmation candle (unequal run lengths must not leak), guarded by an empirical combined_weights[:S] == 0.0 prefix-stability check. The producer now returns a LabelWeightImputationMasks dataclass carrying both stable masks and a stable_release_index; min bounds the stable run while max sets the release candle, so the release index cannot be derived from the mask alone. The non-causal path and existing single-metric leading behavior remain bit-for-bit unchanged. Closes #169 * refactor(quickadapter): harden causal availability release from review feedback Address self-review findings on the #169 harmonization without changing production behavior (bit-for-bit over a monotone battery vs the prior commit). - Release the combined leading run via the max over the release prefix weight_availability[: stable_release_index + 1] instead of a single lookup. Both are equal when weight_availability is monotone (guaranteed by the zigzag confirmation watermark), but the prefix max stays leak-free even if that invariant is ever violated, since every contributing component confirms at or before stable_release_index (= max_c first_finite_c for combined). - Guard the release on idx.size == raw_idx.size so a dropped (out-of-range) pivot, which would desync the producer-space release index from the filtered weight_availability, conservatively defers to the frame boundary. - Rename all_components_finite -> every_component_has_finite: the flag is true when no component is entirely non-finite, not when every value is finite. - Refine docstrings to state the prefix-max release and that trailing pivots are additionally bounded by their own label availability via the max fold. * docs(quickadapter): align LabelWeightImputationMasks docstring with prefix-max release The dataclass docstring still described the pre-hardening single-lookup release (weight_availability[stable_release_index]). Match the consumer/producer docstrings: the release candle is the max over the release prefix weight_availability[: stable_release_index + 1], where stable_release_index is first_finite for a single metric and the max over components for combined. * fix(quickadapter): stop causal leak of non-terminal trailing pivot weights trailing_stable_mask marked the whole trailing non-finite run (last_finite+1 .. n) and released it at weight_availability[first_finite]. Only the terminal pivot (no closing swing) is provably 0.0 there: it is always the last pivot so its trailing-boundary classification cannot flip. A non-terminal pivot inside a longer trailing run — reachable when an interior swing metric is non-finite (e.g. volume_rate / efficiency_ratio / volume_weighted_efficiency_ ratio on a zero-volume or flat window) — could still become interior (median, != 0.0) until its own later swing confirms, so releasing it at the first finite pivot's confirmation leaked future information (under-purge). Restrict trailing_stable_mask to the terminal pivot; a non-terminal pivot in a length>=2 trailing run stays deferred to the frame boundary n via dependency_mask (conservative, no leak). Behaviour is unchanged for the common length-1 trailing run (terminal only), and for the leading, combined, and non-causal paths. Docstrings aligned (terminal-pivot wording; stable_release_ index == -1 semantics clarified). * docs(quickadapter): correct stable_release_index and prefix-max descriptions Third-round review accuracy fixes (documentation only, no behaviour change): - stable_release_index: the docstrings claimed "-1 when both stable masks are empty", but a single metric with any finite value reports first_finite (>= 0) even when both masks are empty (all-finite or interior-only-NaN). It is -1 only when no pivot is finite. The consumer independently gates the release on a non-empty stable mask, so a non-negative index with empty masks is inert. - Consumer release comment: the prefix max equals weight_availability[stable_release_index] only under the production invariant (zigzag confirmation watermark => monotone weight_availability), not unconditionally; state that explicitly and note the prefix max stays leak-free (at worst defers later) if the invariant is ever violated. * docs(quickadapter): deduplicate imputation-masks docstrings * docs(quickadapter): tighten imputation-masks safety-invariant comments * fix(quickadapter): stop causal leak of terminal pivot-weight imputation The terminal non-finite pivot's 0.0 imputation is not causal-prefix stable: add_pivot backfills its swing metric once a later closing pivot arrives, so the 0.0 becomes a nonzero interior median. Releasing it (and skipping its Gaussian band) before the frame boundary n leaked the fact that no later pivot occurs. Defer the terminal pivot to n via dependency_mask and remove the now-always-empty trailing_stable plumbing (dataclass field, consumer parameter, and caller wiring). The leading-run and combined leading release (both provably prefix-stable) are unchanged. Empirically verified against the prior revision: bit-for-bit availability on all non-terminal cases; terminal cases only ever defer later (leak-free). * refactor(quickadapter): make stable_release_index -1 when no leading run Return stable_release_index = -1 whenever leading_stable_mask is empty (single-metric with first_finite == 0: all-finite or interior-only-NaN), so the field holds the invariant 'index >= 0 iff a leading run is released' instead of relying on the consumer's mask-emptiness gate. The consumer already gates the release on leading_stable_mask.any(), so the availability output is bit-for-bit unchanged (verified across single, combined, and uniform cases). Also align docstrings/terminology (terminal vs trailing pivot) and tighten the stable_release_index docs. * docs(quickadapter): correct all-non-finite combined-component rationale The comment claimed an all-non-finite component (imputed to the nonzero default 1.0) makes the aggregate leading run non-zero, but geometric_mean and harmonic_mean annihilate to 0.0 when any component is 0.0 (verified), so that rationale is false. The real reason the leading release is blocked is that an all-non-finite component never confirms a finite weight in-frame and thus has no first_finite release candle; the pivots defer to n conservatively. Comment-only; no behavior change. * docs(quickadapter): tighten LabelWeightImputationMasks docstring Remove the consumer release-prefix formula duplicated from compute_label_weight_known_at_lookahead (single source of truth), drop the vague 'shared' wording, and name both the producer and consumer of the dataclass. Docstring-only; no behavior change. --- diff --git a/quickadapter/user_data/strategies/QuickAdapterV3.py b/quickadapter/user_data/strategies/QuickAdapterV3.py index 7078f3a..6cfda83 100644 --- a/quickadapter/user_data/strategies/QuickAdapterV3.py +++ b/quickadapter/user_data/strategies/QuickAdapterV3.py @@ -998,17 +998,24 @@ class QuickAdapterV3(IStrategy): ) if label_data.known_at_lookahead is not None: if causal_mode: - ( - imputation_dependency_mask, - imputation_leading_stable_mask, - ) = compute_label_weight_imputation_dependency_mask( - len(label_data.indices), - label_data.metrics, - col_weighting_config, + imputation_masks = ( + compute_label_weight_imputation_dependency_mask( + len(label_data.indices), + label_data.metrics, + col_weighting_config, + ) + ) + imputation_dependency_mask = imputation_masks.dependency_mask + imputation_leading_stable_mask = ( + imputation_masks.leading_stable_mask + ) + imputation_stable_release_index = ( + imputation_masks.stable_release_index ) else: imputation_dependency_mask = None imputation_leading_stable_mask = None + imputation_stable_release_index = -1 dataframe[ label_weight_known_at_lookahead_column_name(label_col) ] = compute_label_weight_known_at_lookahead( @@ -1018,6 +1025,7 @@ class QuickAdapterV3(IStrategy): weighting_config=col_weighting_config, imputation_dependency_mask=imputation_dependency_mask, imputation_leading_stable_mask=imputation_leading_stable_mask, + imputation_stable_release_index=imputation_stable_release_index, ) if label_col == EXTREMA_COLUMN: diff --git a/quickadapter/user_data/strategies/Utils.py b/quickadapter/user_data/strategies/Utils.py index 7e72c19..561bf58 100644 --- a/quickadapter/user_data/strategies/Utils.py +++ b/quickadapter/user_data/strategies/Utils.py @@ -2636,24 +2636,67 @@ def _nonfinite_imputation_dependency_mask( return ~np.isfinite(values) +@dataclass(frozen=True, slots=True) +class LabelWeightImputationMasks: + """Causal-availability masks for non-finite pivot-weight imputations. + + Produced by :func:`compute_label_weight_imputation_dependency_mask` (see it + for the release algorithm) and consumed by + :func:`compute_label_weight_known_at_lookahead`. + + - ``dependency_mask``: pivots deferred to the frame boundary ``n`` + - ``leading_stable_mask``: subset of ``dependency_mask`` provably fixed at + ``0.0`` on the causal prefix, hence released early + - ``stable_release_index``: pivot index bounding that release prefix, or + ``-1`` when ``leading_stable_mask`` is empty + """ + + dependency_mask: NDArray[np.bool_] + leading_stable_mask: NDArray[np.bool_] + stable_release_index: int + + def compute_label_weight_imputation_dependency_mask( n_indices: int, metrics: dict[str, list[float]], weighting_config: dict[str, Any], -) -> tuple[NDArray[np.bool_], NDArray[np.bool_]]: +) -> LabelWeightImputationMasks: """Identify pivot weights whose non-finite imputation can change by prefix. - Returns ``(dependency_mask, leading_stable_mask)``. A ``dependency_mask`` - pivot remains causally unavailable until the frame boundary; for - ``combined``, dependency propagates from every selected component and from - the aggregate before its final imputation. ``leading_stable_mask`` (a subset - of ``dependency_mask``) marks the leading non-finite run of a single-metric - strategy: those pivots impute to ``0.0`` and stabilize once the first finite - pivot's weight is known, so they need not defer to the frame boundary. It is - empty for ``uniform``, ``combined``, and all-non-finite metrics. + Returns a :class:`LabelWeightImputationMasks`. A ``dependency_mask`` pivot + remains causally unavailable until the frame boundary; for ``combined``, + dependency propagates from every selected component and from the aggregate + before its final imputation. + + ``leading_stable_mask`` (a subset of ``dependency_mask``) marks non-finite + runs that impute to ``0.0`` and are provably fixed given only the causal + prefix, so they need not defer to the frame boundary: + + - single-metric: the leading run ``[0, first_finite)`` imputes to ``0.0`` + and stabilizes once the first finite pivot's weight is known; + ``stable_release_index = first_finite``. The terminal pivot and any + non-terminal trailing run stay deferred to ``n`` (their 0.0 is not + causal-prefix stable: a later closing pivot turns the metric finite). + - ``combined`` with every selected component leading with a non-finite run: + the aggregate is stably zero over ``[0, min_c first_finite_c)`` (the + shortest leading run bounds the all-zero prefix), released at the ``max`` + over components ``stable_release_index = max_c first_finite_c`` (the latest + component confirmation; unequal run lengths must not leak). Guarded by an + empirical ``combined_weights[:S] == 0.0`` check. + + ``stable_release_index`` is ``-1`` whenever ``leading_stable_mask`` is empty + (``uniform``, empty or all-non-finite metrics, a single metric with no + leading run such as all-finite or interior-only-NaN, and any ``combined`` + case that fails the checks above); those pivots keep the nonzero default and + defer to ``n``. It is ``>= 0`` if and only if a leading run is released. """ label_weighting = {**DEFAULTS_LABEL_WEIGHTING, **weighting_config} strategy = label_weighting["strategy"] + + def _empty_masks() -> LabelWeightImputationMasks: + zeros = np.zeros(n_indices, dtype=bool) + return LabelWeightImputationMasks(zeros, zeros.copy(), -1) + if strategy == WEIGHT_STRATEGIES[0]: # "none" raise ValueError( "compute_label_weight_imputation_dependency_mask must not be called " @@ -2661,11 +2704,11 @@ def compute_label_weight_imputation_dependency_mask( "weighting is disabled" ) if strategy == WEIGHT_STRATEGIES[1]: # "uniform" - return np.zeros(n_indices, dtype=bool), np.zeros(n_indices, dtype=bool) + return _empty_masks() if strategy in metrics: values = np.asarray(metrics[strategy], dtype=float) if values.size == 0: - return np.zeros(n_indices, dtype=bool), np.zeros(n_indices, dtype=bool) + return _empty_masks() if values.shape != (n_indices,): raise ValueError( f"Invalid metric {strategy!r} shape {values.shape}: " @@ -2674,15 +2717,21 @@ def compute_label_weight_imputation_dependency_mask( dependency = _nonfinite_imputation_dependency_mask(values) leading_stable = np.zeros(n_indices, dtype=bool) finite = ~dependency + release_index = -1 if finite.any(): - leading_stable[: int(np.argmax(finite))] = True - return dependency, leading_stable + first_finite = int(np.argmax(finite)) + if first_finite > 0: + leading_stable[:first_finite] = True + release_index = first_finite + return LabelWeightImputationMasks(dependency, leading_stable, release_index) if strategy != WEIGHT_STRATEGIES[8]: # "combined" raise ValueError(_invalid_weight_strategy_message(strategy, metrics)) dependency_mask = np.zeros(n_indices, dtype=bool) imputed_metrics: list[NDArray[np.floating]] = [] coefficients_list: list[float] = [] + first_finite_indices: list[int] = [] + every_component_has_finite = True for metric_name, values_array, coefficient in _select_combined_metrics( metrics, label_weighting["metric_coefficients"] ): @@ -2691,12 +2740,21 @@ def compute_label_weight_imputation_dependency_mask( f"Invalid metric {metric_name!r} shape {values_array.shape}: " f"must be ({n_indices},)" ) - dependency_mask |= _nonfinite_imputation_dependency_mask(values_array) + component_finite = np.isfinite(values_array) + dependency_mask |= ~component_finite imputed_metrics.append(_impute_weights(values_array)) coefficients_list.append(coefficient) + if component_finite.any(): + first_finite_indices.append(int(np.argmax(component_finite))) + else: + # An all-non-finite component never confirms a finite weight + # in-frame, so it has no first_finite release candle; block the + # leading release and defer these pivots to n conservatively. + every_component_has_finite = False if len(imputed_metrics) == 0: - return dependency_mask, np.zeros(n_indices, dtype=bool) + empty = np.zeros(n_indices, dtype=bool) + return LabelWeightImputationMasks(dependency_mask, empty, -1) combined_weights = _aggregate_imputed_metrics( imputed_metrics, @@ -2710,7 +2768,19 @@ def compute_label_weight_imputation_dependency_mask( f"must be ({n_indices},)" ) dependency_mask |= _nonfinite_imputation_dependency_mask(combined_weights) - return dependency_mask, np.zeros(n_indices, dtype=bool) + + leading_stable = np.zeros(n_indices, dtype=bool) + release_index = -1 + if ( + every_component_has_finite + and first_finite_indices + and min(first_finite_indices) >= 1 + ): + stable_length = min(first_finite_indices) + if bool(np.all(combined_weights[:stable_length] == 0.0)): + leading_stable[:stable_length] = True + release_index = max(first_finite_indices) + return LabelWeightImputationMasks(dependency_mask, leading_stable, release_index) def _compute_epsilon_floor( @@ -3224,13 +3294,14 @@ def compute_label_weight_known_at_lookahead( *, imputation_dependency_mask: Sequence[bool] | NDArray[np.bool_] | None = None, imputation_leading_stable_mask: Sequence[bool] | NDArray[np.bool_] | None = None, + imputation_stable_release_index: int = -1, weighting_config: dict[str, Any] | None = None, ) -> pd.Series: """Per-row causal availability (in candles) of the label WEIGHT column. A metric-based pivot's weight is backfilled from the adjacent closing pivot, so it becomes computable at the next pivot's confirmation - ``i_{k+1} == known_at_positions[indices[k+1]]``; the trailing pivot has no + ``i_{k+1} == known_at_positions[indices[k+1]]``; the terminal pivot has no closing swing (weight 0 via ``_impute_weights``) and never resolves in-frame -> ``n``. A uniform pivot instead has a unit weight at its own label availability. Off-pivot rows keep their label availability, except that a @@ -3251,14 +3322,15 @@ def compute_label_weight_known_at_lookahead( strategies and additive fills keep their existing competing-band dependencies. - ``imputation_dependency_mask`` marks pivot weights whose non-finite - imputation can change as the available prefix grows. Those pivots and their - Gaussian bands are unavailable until the frame boundary. An unresolved - trailing pivot is excluded only when it has no such dependency. - ``imputation_leading_stable_mask`` (a subset) marks a leading non-finite run - that imputes to 0.0 and stabilizes at the first finite pivot's confirmation; - those pivots are released there instead of at the frame boundary and their - zero-weight bands are skipped. + The imputation masks (``imputation_dependency_mask``, + ``imputation_leading_stable_mask``, ``imputation_stable_release_index``) + come from :func:`compute_label_weight_imputation_dependency_mask`. + Dependency pivots and their Gaussian bands wait for the frame boundary; + leading-stable pivots are released over + ``weight_availability[: imputation_stable_release_index + 1]``, folded via + ``max`` with each pivot's own label availability, with their zero-weight + bands skipped. The release applies only in the identity-order case (no + dropped pivot) where the run is a contiguous prefix. """ n = len(known_at_lookahead) positions, known_at_lookahead_values = _sanitize_known_at_lookahead( @@ -3330,16 +3402,19 @@ def compute_label_weight_known_at_lookahead( ) np.maximum(avail_pivot, sigma_availability, out=avail_pivot) avail_pivot[dependency_mask] = n - if leading_stable_mask.any() and np.array_equal(order, np.arange(idx.size)): - # Leading non-finite run imputes to 0.0, stable once the first finite - # pivot's weight is known (backfilled confirmation weight_availability - # [first_finite]), not at the frame boundary. Guarded to the sorted - # (identity-order) case where the run is a contiguous prefix. - first_finite = int(leading_stable_mask.sum()) - if first_finite < weight_availability.size: - avail_pivot[leading_stable_mask] = int( - weight_availability[first_finite] - ) + if ( + leading_stable_mask.any() + and idx.size == raw_idx.size + and np.array_equal(order, np.arange(idx.size)) + and 0 <= imputation_stable_release_index < weight_availability.size + ): + # Prefix max (not weight_availability[stable_release_index]) stays + # leak-free if availability is non-monotone, at worst deferring + # later; guarded to identity order (contiguous prefix run). + release = int( + np.max(weight_availability[: imputation_stable_release_index + 1]) + ) + avail_pivot[leading_stable_mask] = release base[idx] = np.maximum(base[idx], avail_pivot) if fill_radius > 0: for (