From: Jérôme Benoit Date: Sat, 25 Jul 2026 00:08:57 +0000 (+0200) Subject: fix(quickadapter): refit iteration aliases, test_size None docs, live holdout_rmse... X-Git-Url: https://git.piment-noir.org/?a=commitdiff_plain;h=ad3246b8a77e2ac0b594652e205a4f2f9cfdfe86;p=freqai-strategies.git fix(quickadapter): refit iteration aliases, test_size None docs, live holdout_rmse (#124) * fix(quickadapter): clear regressor iteration aliases before refit The refit set the canonical iteration parameter on a deep copy of the user model_training_parameters without removing configured synonyms. CatBoost aborts on duplicate iteration aliases, so a selection fit that used n_estimators, num_boost_round, or num_trees crashed at the post-holdout refit. Purge all known iteration aliases per regressor before setting the canonical count. * docs(quickadapter): correct test_size None applicability None reaches sklearn dynamic sizing only through timeseries_split; the train_test_split path rejects any non-int/float value. Document that None applies to timeseries_split and that train_test_split requires a float or int. * fix(quickadapter): restore holdout_rmse on live model reload A live or dry-run restart that reuses a cached model without invoking fit left _holdout_rmse at the constructor inf placeholder, so fit_live_predictions published holdout_rmse=inf until the next retraining window. Track pairs fitted in the current session and, when none ran, recover the last finite holdout_rmse from historic_predictions. * fix(quickadapter): complete lightgbm iteration aliases and guard alias-map coverage The lightgbm alias set omitted max_iter, num_round, and num_tree. LightGBM does not reject duplicate iteration synonyms; it silently lets a leftover alias win over n_estimators, so a config using one of those names would train the refit at its original capacity instead of the selected count. Complete the set to every num_iterations synonym and add an import-time guard that the alias map covers REGRESSORS. * fix(quickadapter): coerce holdout_rmse to numeric before finite filter Guard the live holdout_rmse restore against a non-float historic column: coerce with pd.to_numeric(errors=coerce) so np.isfinite cannot raise on an object dtype, matching the defensive handling of the non-live replay branch. * refactor(quickadapter): consolidate per-regressor metadata into RegressorSpec Replace the hand-maintained REGRESSORS tuple, the _REFIT_ITERATION_ALIASES map and the scattered magic-index dispatch with a single RegressorSpec source: the _REGRESSOR_SPECS NamedTuple singleton (mirroring _OPTUNA_NAMESPACES) carries each regressor's canonical iteration parameter, its iteration aliases and its RNG seed parameter. REGRESSORS, DEFAULT_REGRESSOR and the by-name lookup derive from it, and import-time guards enforce coverage of the Regressor literal and that each canonical iteration parameter is one of its own aliases. get_refit, fit_regressor and get_optuna_study_model_parameters now dispatch on named specs (regressor == _REGRESSOR_SPECS.xgboost.name) instead of REGRESSORS[i]; fit_regressor's identical per-branch seed setdefault + trial increment is hoisted into one spec-driven prelude. Behavior preserved: same REGRESSORS values/order, same seed handling (verified identical across all five regressors), introspection and library-specific fit logic left in their branches. QuickAdapterRegressorV3 uses DEFAULT_REGRESSOR instead of REGRESSORS[0]. * fix(quickadapter): recover last published holdout_rmse (incl. inf) on live reload Restoring the last FINITE historic holdout_rmse discarded an intentional inf from a model trained with test_size=0 and could resurrect a stale finite score from an earlier holdout-enabled configuration. Take the last published (non-null) value instead, so the recovered metric faithfully reflects the cached model. * docs(quickadapter): restore inner-validation 0.1 fallback note for test_size The test_size row documented only the outer None behavior after the applicability fix; restore that the inner validation split falls back to 0.1, and tighten the refit clause to keep the row within the table column width. --- diff --git a/README.md b/README.md index f1c4cb8..1a2415a 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ docker compose up -d --build | freqai.model_training_parameters.gpu_vram_gb | 80 | enum {8,10,12,16,24,32,40,48,64,80} | Available GPU VRAM (GB) for CatBoost, not total. Constrains `depth`, `border_count`, and `max_ctr_complexity` ranges. | | _Data split parameters_ | | | | | freqai.data_split_parameters.method | `train_test_split` | enum {`train_test_split`,`timeseries_split`} | Data splitting strategy. `train_test_split` for sequential split, `timeseries_split` for chronological split with configurable gap. | -| freqai.data_split_parameters.test_size | 0.1 | float (0,1) \| int >= 1 \| None | Outer holdout size. The same parameter reserves the chronological tail of the remaining training rows as inner validation for HPO and early stopping; a fractional value is relative to those remaining rows, not the original window. The holdout is predicted once and reported as weighted `holdout_rmse` in the original label scale; it therefore measures the pre-refit selection model, not the deployed model refitted afterwards on all causally available window rows. `None` uses sklearn's default outer size and the model's `0.1` fallback for inner validation. | +| freqai.data_split_parameters.test_size | 0.1 | float (0,1) \| int >= 1 \| None | Outer holdout size. The same parameter reserves the chronological tail of the remaining training rows as inner validation for HPO and early stopping; a fractional value is relative to those remaining rows, not the original window. The holdout is predicted once and reported as weighted `holdout_rmse` in the original label scale; it therefore measures the pre-refit selection model, not the refitted deployed model. `None` (sklearn dynamic sizing) applies only to `timeseries_split`; `train_test_split` requires a float or int; inner validation then falls back to `0.1`. | | freqai.data_split_parameters.n_splits | 5 | int >= 2 | Controls train/test proportions for `timeseries_split` (higher = larger train set). | | freqai.data_split_parameters.gap | 0 | int >= 0 | Samples to exclude between train/test for `timeseries_split`. When `0` and `causal_mode=true` (default), auto-set from `label_horizon_candles`; when `0` and `causal_mode=false`, auto-set from `label_period_candles`. Under `causal_mode=true`, an explicit `gap= 1 \| None | Maximum training set size for `timeseries_split`. When set, creates a sliding window instead of expanding train set. None = no limit. | diff --git a/quickadapter/user_data/freqaimodels/QuickAdapterRegressorV3.py b/quickadapter/user_data/freqaimodels/QuickAdapterRegressorV3.py index c5a1e6a..842bfaf 100644 --- a/quickadapter/user_data/freqaimodels/QuickAdapterRegressorV3.py +++ b/quickadapter/user_data/freqaimodels/QuickAdapterRegressorV3.py @@ -68,6 +68,7 @@ from LabelTransformer import ( from Utils import ( DEFAULT_FIT_LIVE_PREDICTIONS_CANDLES, + DEFAULT_REGRESSOR, DEFAULTS_LABEL_PREDICTION, LABEL_COLUMNS, LabelWeightSupportError, @@ -1429,6 +1430,7 @@ class QuickAdapterRegressorV3(BaseRegressionModel): ) self._optuna_hp_value: dict[str, float] = {} self._holdout_rmse: dict[str, float] = {} + self._session_fitted_pairs: set[str] = set() self._optuna_label_values: dict[str, list[float | int]] = {} self._optuna_hp_params: dict[str, dict[str, Any]] = {} self._optuna_label_params: dict[str, dict[str, Any]] = {} @@ -1477,9 +1479,9 @@ class QuickAdapterRegressorV3(BaseRegressionModel): self.set_optuna_label_candle(pair) self._optuna_label_candles[pair] = 0 - self.regressor: Regressor = self.freqai_info.get("regressor", REGRESSORS[0]) + self.regressor: Regressor = self.freqai_info.get("regressor", DEFAULT_REGRESSOR) if self.regressor not in set(REGRESSORS): - self.regressor = REGRESSORS[0] + self.regressor = DEFAULT_REGRESSOR self.freqai_info["regressor"] = self.regressor self._log_model_configuration() @@ -2863,6 +2865,7 @@ class QuickAdapterRegressorV3(BaseRegressionModel): ) else: self._holdout_rmse[dk.pair] = np.inf + self._session_fitted_pairs.add(dk.pair) dk.data["extra_returns_per_train"]["holdout_rmse"] = self._holdout_rmse[dk.pair] if validation_size != 0: refit_model_training_parameters = get_refit_model_training_parameters( @@ -3128,6 +3131,14 @@ class QuickAdapterRegressorV3(BaseRegressionModel): f"[{pair}] replayed holdout_rmse is NaN at " f"{current_dates.index[0]}; defaulting to inf" ) + elif pair not in self._session_fitted_pairs: + historic = self.dd.historic_predictions.get(pair) + if historic is not None and "holdout_rmse" in historic: + holdout_values = pd.to_numeric( + historic["holdout_rmse"], errors="coerce" + ).dropna() + if not holdout_values.empty: + current_holdout_rmse = float(holdout_values.iloc[-1]) holdout_rmse = QuickAdapterRegressorV3.optuna_validate_value( current_holdout_rmse ) diff --git a/quickadapter/user_data/strategies/Utils.py b/quickadapter/user_data/strategies/Utils.py index eea375a..247b803 100644 --- a/quickadapter/user_data/strategies/Utils.py +++ b/quickadapter/user_data/strategies/Utils.py @@ -21,6 +21,7 @@ from typing import ( TypeVar, assert_never, cast, + get_args, ) import numpy as np @@ -3570,13 +3571,80 @@ def zigzag( Regressor = Literal[ "xgboost", "lightgbm", "histgradientboostingregressor", "ngboost", "catboost" ] -REGRESSORS: Final[tuple[Regressor, ...]] = ( - "xgboost", - "lightgbm", - "histgradientboostingregressor", - "ngboost", - "catboost", -) + + +class RegressorSpec(NamedTuple): + name: Regressor + iteration_param: str + iteration_aliases: frozenset[str] + seed_param: str + + +# Per-regressor metadata single source: the canonical boosting-iteration +# parameter, all of its synonyms (the refit must set exactly one, else CatBoost +# aborts on duplicate iteration aliases), and the RNG seed parameter name. +class _RegressorSpecs(NamedTuple): + xgboost: RegressorSpec = RegressorSpec( + "xgboost", + "n_estimators", + frozenset({"n_estimators", "num_boost_round"}), + "random_state", + ) + lightgbm: RegressorSpec = RegressorSpec( + "lightgbm", + "n_estimators", + frozenset( + { + "n_estimators", + "num_iterations", + "num_iteration", + "num_boost_round", + "num_round", + "num_rounds", + "nrounds", + "num_tree", + "num_trees", + "max_iter", + "n_iter", + } + ), + "seed", + ) + histgradientboostingregressor: RegressorSpec = RegressorSpec( + "histgradientboostingregressor", + "max_iter", + frozenset({"max_iter"}), + "random_state", + ) + ngboost: RegressorSpec = RegressorSpec( + "ngboost", + "n_estimators", + frozenset({"n_estimators"}), + "random_state", + ) + catboost: RegressorSpec = RegressorSpec( + "catboost", + "iterations", + frozenset({"iterations", "n_estimators", "num_boost_round", "num_trees"}), + "random_seed", + ) + + +_REGRESSOR_SPECS: Final[_RegressorSpecs] = _RegressorSpecs() +_REGRESSOR_SPEC_BY_NAME: Final[dict[Regressor, RegressorSpec]] = { + spec.name: spec for spec in _REGRESSOR_SPECS +} +REGRESSORS: Final[tuple[Regressor, ...]] = tuple(spec.name for spec in _REGRESSOR_SPECS) +DEFAULT_REGRESSOR: Final[Regressor] = _REGRESSOR_SPECS.xgboost.name + +if set(_REGRESSOR_SPEC_BY_NAME) != set(get_args(Regressor)): + raise RuntimeError( + "_REGRESSOR_SPECS must define a spec for every Regressor literal member" + ) +if any(spec.iteration_param not in spec.iteration_aliases for spec in _REGRESSOR_SPECS): + raise RuntimeError( + "each RegressorSpec.iteration_param must be listed in its iteration_aliases" + ) RegressorCallback = Callable[..., Any] | XGBoostTrainingCallback @@ -3639,15 +3707,14 @@ def get_refit_model_training_parameters( """Return parameters that preserve the selected model capacity for refit.""" refit_parameters = copy.deepcopy(model_training_parameters) - if regressor == REGRESSORS[0]: # "xgboost" + if regressor == _REGRESSOR_SPECS.xgboost.name: fitted_iterations = int(model.get_booster().num_boosted_rounds()) initial_iterations = ( int(init_model.get_booster().num_boosted_rounds()) if init_model is not None else 0 ) - parameter_name = "n_estimators" - elif regressor == REGRESSORS[1]: # "lightgbm" + elif regressor == _REGRESSOR_SPECS.lightgbm.name: best_iteration = getattr(model, "best_iteration_", 0) or 0 fitted_iterations = int( best_iteration if best_iteration > 0 else model.n_estimators_ @@ -3657,32 +3724,31 @@ def get_refit_model_training_parameters( if init_model is not None else 0 ) - parameter_name = "n_estimators" - elif regressor == REGRESSORS[2]: # "histgradientboostingregressor" + elif regressor == _REGRESSOR_SPECS.histgradientboostingregressor.name: fitted_iterations = int(model.n_iter_) initial_iterations = 0 - parameter_name = "max_iter" refit_parameters["early_stopping"] = False - elif regressor == REGRESSORS[3]: # "ngboost" + elif regressor == _REGRESSOR_SPECS.ngboost.name: fitted_iterations = len(model.base_models) initial_iterations = 0 - parameter_name = "n_estimators" - elif regressor == REGRESSORS[4]: # "catboost" + elif regressor == _REGRESSOR_SPECS.catboost.name: fitted_iterations = int(model.tree_count_) initial_iterations = 0 - parameter_name = "iterations" else: raise ValueError( f"Invalid regressor value {regressor!r}: " f"supported values are {', '.join(REGRESSORS)}" ) + spec = _REGRESSOR_SPEC_BY_NAME[regressor] # best_iteration is combined-indexed under current xgboost/lightgbm, so # fitted >= initial + 1 always holds; clamp defensively so a degenerate # non-improving continual-learning refit degrades gracefully instead of # raising and killing the whole training window. refit_iterations = max(fitted_iterations - initial_iterations, 1) - refit_parameters[parameter_name] = refit_iterations + for alias in spec.iteration_aliases: + refit_parameters.pop(alias, None) + refit_parameters[spec.iteration_param] = refit_iterations return refit_parameters @@ -3711,12 +3777,22 @@ def fit_regressor( eval_set = None eval_weights = None - if regressor == REGRESSORS[0]: # "xgboost" + spec = _REGRESSOR_SPEC_BY_NAME.get(regressor) + if spec is None: + raise ValueError( + f"Invalid regressor value {regressor!r}: " + f"supported values are {', '.join(REGRESSORS)}" + ) + model_training_parameters.setdefault(spec.seed_param, 1) + if trial is not None: + model_training_parameters[spec.seed_param] = ( + model_training_parameters[spec.seed_param] + trial.number + ) + + if regressor == _REGRESSOR_SPECS.xgboost.name: from xgboost import XGBRegressor from xgboost.callback import EarlyStopping - model_training_parameters.setdefault("random_state", 1) - early_stopping_rounds = None if has_eval_set: early_stopping_rounds = model_training_parameters.pop( @@ -3735,16 +3811,10 @@ def fit_regressor( ) ) - if trial is not None: - model_training_parameters["random_state"] = ( - model_training_parameters["random_state"] + trial.number + if trial is not None and has_eval_set: + fit_callbacks.append( + optuna.integration.XGBoostPruningCallback(trial, "validation_0-rmse") ) - if has_eval_set: - fit_callbacks.append( - optuna.integration.XGBoostPruningCallback( - trial, "validation_0-rmse" - ) - ) model = XGBRegressor( objective="reg:squarederror", @@ -3760,11 +3830,9 @@ def fit_regressor( sample_weight_eval_set=eval_weights, xgb_model=init_model, ) - elif regressor == REGRESSORS[1]: # "lightgbm" + elif regressor == _REGRESSOR_SPECS.lightgbm.name: from lightgbm import LGBMRegressor, early_stopping - model_training_parameters.setdefault("seed", 1) - early_stopping_rounds = None if has_eval_set: early_stopping_rounds = model_training_parameters.pop( @@ -3782,16 +3850,12 @@ def fit_regressor( ) ) - if trial is not None: - model_training_parameters["seed"] = ( - model_training_parameters["seed"] + trial.number - ) - if has_eval_set: - fit_callbacks.append( - optuna.integration.LightGBMPruningCallback( - trial, "rmse", valid_name="valid_0" - ) + if trial is not None and has_eval_set: + fit_callbacks.append( + optuna.integration.LightGBMPruningCallback( + trial, "rmse", valid_name="valid_0" ) + ) model = LGBMRegressor(objective="regression", **model_training_parameters) model.fit( @@ -3804,10 +3868,9 @@ def fit_regressor( init_model=init_model, callbacks=fit_callbacks if fit_callbacks else None, ) - elif regressor == REGRESSORS[2]: # "histgradientboostingregressor" + elif regressor == _REGRESSOR_SPECS.histgradientboostingregressor.name: from sklearn.ensemble import HistGradientBoostingRegressor - model_training_parameters.setdefault("random_state", 1) model_training_parameters.setdefault("loss", "squared_error") early_stopping = model_training_parameters.pop("early_stopping", True) model_training_parameters.pop("n_jobs", None) @@ -3828,11 +3891,6 @@ def fit_regressor( if "verbose" not in model_training_parameters and verbosity is not None: model_training_parameters["verbose"] = verbosity - if trial is not None: - model_training_parameters["random_state"] = ( - model_training_parameters["random_state"] + trial.number - ) - X_val = None y_val = None if has_eval_set: @@ -3856,12 +3914,10 @@ def fit_regressor( y_val=y_val, sample_weight_val=sample_weight_val, ) - elif regressor == REGRESSORS[3]: # "ngboost" + elif regressor == _REGRESSOR_SPECS.ngboost.name: from ngboost import NGBRegressor from sklearn.tree import DecisionTreeRegressor - model_training_parameters.setdefault("random_state", 1) - verbosity = model_training_parameters.pop("verbosity", None) if "verbose" not in model_training_parameters and verbosity is not None: model_training_parameters["verbose"] = verbosity @@ -3876,11 +3932,6 @@ def fit_regressor( else: model_training_parameters.pop("early_stopping_rounds", None) - if trial is not None: - model_training_parameters["random_state"] = ( - model_training_parameters["random_state"] + trial.number - ) - dist = model_training_parameters.pop("dist", "lognormal") X_val = None @@ -3912,10 +3963,9 @@ def fit_regressor( val_sample_weight=val_sample_weight, early_stopping_rounds=early_stopping_rounds, ) - elif regressor == REGRESSORS[4]: # "catboost" + elif regressor == _REGRESSOR_SPECS.catboost.name: from catboost import CatBoostRegressor, Pool - model_training_parameters.setdefault("random_seed", 1) model_training_parameters.setdefault("loss_function", "RMSE") if model_path is not None and "train_dir" not in model_training_parameters: @@ -3956,11 +4006,6 @@ def fit_regressor( if "verbose" not in model_training_parameters and verbosity is not None: model_training_parameters["verbose"] = verbosity - if trial is not None: - model_training_parameters["random_seed"] = ( - model_training_parameters["random_seed"] + trial.number - ) - pruning_callback = None if trial is not None and has_eval_set and task_type != "GPU": pruning_callback = optuna.integration.CatBoostPruningCallback(trial, "RMSE") @@ -4312,7 +4357,7 @@ def get_optuna_study_model_parameters( ranges[param] = (param_min, param_max) return ranges - if regressor == REGRESSORS[0]: # "xgboost" + if regressor == _REGRESSOR_SPECS.xgboost.name: # Parameter order: boosting -> tree structure -> leaf constraints -> # sampling -> regularization -> binning default_ranges: dict[str, tuple[float, float]] = { @@ -4436,7 +4481,7 @@ def get_optuna_study_model_parameters( return params - elif regressor == REGRESSORS[1]: # "lightgbm" + elif regressor == _REGRESSOR_SPECS.lightgbm.name: # Parameter order: boosting -> tree structure -> leaf constraints -> # sampling -> regularization -> binning default_ranges: dict[str, tuple[float, float]] = { @@ -4543,7 +4588,7 @@ def get_optuna_study_model_parameters( return params - elif regressor == REGRESSORS[2]: # "histgradientboostingregressor" + elif regressor == _REGRESSOR_SPECS.histgradientboostingregressor.name: # Parameter order: boosting -> tree structure -> leaf constraints -> # sampling -> regularization -> binning -> early stopping default_ranges: dict[str, tuple[float, float]] = { @@ -4647,7 +4692,7 @@ def get_optuna_study_model_parameters( ), } - elif regressor == REGRESSORS[3]: # "ngboost" + elif regressor == _REGRESSOR_SPECS.ngboost.name: # Parameter order: boosting -> tree structure -> sampling -> early stopping -> distribution default_ranges: dict[str, tuple[float, float]] = { # Boosting/Training @@ -4714,7 +4759,7 @@ def get_optuna_study_model_parameters( "dist": trial.suggest_categorical("dist", ["normal", "lognormal"]), } - elif regressor == REGRESSORS[4]: # "catboost" + elif regressor == _REGRESSOR_SPECS.catboost.name: # Parameter order: boosting -> tree structure -> regularization -> sampling task_type = model_training_parameters.get("task_type", "CPU") loss_function = model_training_parameters.get("loss_function", "RMSE")