Jérôme Benoit [Sun, 4 Jan 2026 15:54:26 +0000 (16:54 +0100)]
refactor: enhance extrema weighting with sklearn scalers and new methods
Replace manual standardization/normalization calculations with sklearn scalers
for better maintainability and correctness.
Standardization changes:
- Add power_yj (Yeo-Johnson) standardization method
- Replace manual zscore with StandardScaler
- Replace manual robust with RobustScaler
- Add mask size checks for all methods including MMAD
- Store fitted scaler objects instead of manual stats
Normalization changes:
- Add maxabs normalization (new default)
- Replace manual minmax with MinMaxScaler
- Fix sigmoid to output [-1, 1] range (was [0, 1])
- Replace manual calculations with MaxAbsScaler and MinMaxScaler
Other improvements:
- Remove zero-exclusion from mask (zeros are valid values)
- Fit normalization on standardized data (proper pipeline order)
- Add proper RuntimeError for unfitted scalers
Docs:
- Update README to reflect maxabs as new normalization default
- Document power_yj standardization type
- Harmonize mathematical formulas with code notation
Jérôme Benoit [Sat, 3 Jan 2026 22:36:37 +0000 (23:36 +0100)]
refactor: improve string literal replacements and use specific tuple constants
- Replace hardcoded method names in error messages with tuple constants
(lines 2003, 2167: use _DISTANCE_METHODS[0] and [1] instead of literal strings)
- Use _CLUSTER_METHODS instead of _SELECTION_METHODS indices for better
code maintainability (e.g., _CLUSTER_METHODS[0] vs _SELECTION_METHODS[2])
- Fix trial_selection_method comparison order to match tuple constant order
(compromise_programming [0] before topsis [1])
- Remove redundant power_mean None check (already validated by _validate_power_mean)
- Add clarifying comments to tuple constant usages (minkowski, rank_extrema)
Jérôme Benoit [Sat, 3 Jan 2026 20:48:51 +0000 (21:48 +0100)]
fix: eliminate data leakage in extrema weighting normalization (#30)
* fix: eliminate data leakage in extrema weighting normalization
Move dataset-dependent scaling from strategy (pre-split) to model label
pipeline (post-split) to prevent train/test data leakage.
Changes:
- Add ExtremaWeightingTransformer (datasieve BaseTransform) in Utils.py
that fits standardization/normalization stats on training data only
- Add define_label_pipeline() in QuickAdapterRegressorV3 that replaces
FreqAI's default MinMaxScaler with our configurable transformer
- Simplify strategy's set_freqai_targets() to pass raw weighted extrema
without any normalization (normalization now happens post-split)
- Remove pre-split normalization functions from Utils.py (~107 lines)
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor: align ExtremaWeightingTransformer with BaseTransform API
- Call super().__init__() with name parameter
- Match method signatures exactly (npt.ArrayLike, ArrayOrNone, ListOrNone)
- Return tuple from fit() instead of self
- Import types from same namespaces as BaseTransform
* refactor: cleanup type hints
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor: remove unnecessary type casts and annotations
Let numpy types flow naturally without explicit float()/int() casts.
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor: use scipy.special.logit for inverse sigmoid transformation
Replace manual inverse sigmoid calculation (-np.log(1.0 / values - 1.0))
with scipy.special.logit() for better code clarity and consistency.
- Uses official scipy function that is the documented inverse of expit
- Mathematically equivalent to the previous implementation
- Improves code readability and maintainability
- Maintains symmetry: sp.special.expit() <-> sp.special.logit()
Also improve comment clarity for standardization identity function.
The _n_train attribute was being set during fit() but never used
elsewhere in the class or by the BaseTransform interface. Removing
it to reduce code clutter and improve maintainability.
* fix: import paths correction
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* fix: add Bessel correction and ValueError consistency in ExtremaWeightingTransformer
- Use ddof=1 for std computation (sample std instead of population std)
- Add ValueError in _inverse_standardize for unknown methods
- Add ValueError in _inverse_normalize for unknown methods
* chore: refine config-template.json for extrema weighting options
Jérôme Benoit [Fri, 2 Jan 2026 23:10:44 +0000 (00:10 +0100)]
refactor: remove redundant 'Only' prefix from namespace comments
Simplify code comments by removing the word 'Only' from namespace
identifier comments. The context already makes it clear that these
are the supported namespaces.
Jérôme Benoit [Fri, 2 Jan 2026 23:05:31 +0000 (00:05 +0100)]
Remove Optuna "train" namespace as preliminary step to eliminate data leakage
Remove the "train" namespace from Optuna hyperparameter optimization to
address data leakage issues in extrema weighting normalization. This is a
preliminary step before implementing a proper data preparation pipeline
that prevents train/test contamination.
Problem:
Current architecture applies extrema weighting normalization (minmax, softmax,
zscore, etc.) on the full dataset BEFORE train/test split. This causes data
leakage: train set labels are normalized using statistics (min/max, mean/std,
median/IQR) computed from the entire dataset including test set. The "train"
namespace hyperopt optimization exacerbates this by optimizing dataset
truncation with contaminated statistics.
Solution approach:
1. Remove "train" namespace optimization (this commit)
2. Switch to binary extrema labels (strategy: "none") to avoid leakage
3. Future: implement proper data preparation that computes normalization
statistics on train set only and applies them to both train/test sets
This naive train/test splitting hyperopt is incompatible with a correct
data preparation pipeline where normalization must be fit on train and
transformed on test separately.
Changes:
- Remove "train" namespace from OptunaNamespace (3→2 namespaces: hp, label)
- Remove train_objective function and all train optimization logic
- Remove dataset truncation based on optimized train/test periods
- Update namespace indices: label from [2] to [1] throughout codebase
- Remove train_candles_step config parameter and train_rmse metric tracking
- Set extrema_weighting.strategy to "none" (binary labels: -1/0/+1)
- Update documentation to reflect 2-namespace architecture
Jérôme Benoit [Fri, 2 Jan 2026 15:16:18 +0000 (16:16 +0100)]
refactor(quickadapter): consolidate custom distance metrics in Pareto front selection
Extract shared distance metric logic from _compromise_programming_scores and
_topsis_scores into reusable static methods:
- Add _POWER_MEAN_MAP class constant as single source of truth for power values
- Add _power_mean_metrics_set() cached method for metric name lookup
- Add _hellinger_distance() for Hellinger/Shellinger computation
- Add _power_mean_distance() for generalized mean computation with validation
- Add _weighted_sum_distance() for weighted sum computation
Harmonize with existing validation API using ValidationMode and proper contexts.
Jérôme Benoit [Fri, 2 Jan 2026 14:21:16 +0000 (15:21 +0100)]
refactor(quickadapter): unify validation helpers with ValidationMode support
- Add ValidationMode type for "warn", "raise", "none" behavior
- Rename _UNSUPPORTED_CLUSTER_METRICS to _UNSUPPORTED_WEIGHTS_METRICS
- Refactor _validate_minkowski_p, _validate_quantile_q with mode support
- Add _validate_power_mean_p, _validate_metric_weights_support, _validate_label_weights
- Add _validate_enum_value for generic enum validation
- Add _prepare_knn_kwargs for sklearn-specific weight handling
- Remove deprecated _normalize_weights function
- Update all call sites to use new unified API
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* fix: use unbounded cache for constant-returning helper methods
Replace @lru_cache(maxsize=1) with @lru_cache(maxsize=None) for all
static methods that return constant sets. Using maxsize=None is more
idiomatic and efficient for parameterless functions that always return
the same value.
* refactor: add _prepare_distance_kwargs to centralize distance kwargs preparation
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor: cleanup extrema weighting API
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor: cleanup extrema smoothing API
- Remove ClusterSelectionMethod type and related constants
- Unify selection methods to use DistanceMethod for both cluster and trial selection
- Add separate trial_selection_method parameter for within-cluster selection
- Change power_mean default from 2.0 to 1.0 for internal consistency
- Add validation for selection_method and trial_selection_method parameters
* fix: add missing validations for label_distance_metric and label_density_aggregation_param
- Add validation for label_distance_metric parameter at configuration time
- Add early validation for label_density_aggregation_param (quantile and power_mean)
- Ensures invalid configuration values fail fast with clear error messages
- Harmonizes error messages with existing validation patterns in the codebase
* fix: add validation for label_cluster_metric and custom metrics support in topsis
- Add validation that label_cluster_metric is in _distance_metrics_set()
- Implement custom metrics support in _topsis_scores (hellinger, shellinger,
harmonic/geometric/arithmetic/quadratic/cubic/power_mean, weighted_sum)
matching _compromise_programming_scores implementation
* docs: update README.md with refactored label selection methods
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* docs: fix config parameter and bump to v3.9.0
- Fix config-template.json: label_metric -> label_method
- Bump version from 3.8.5 to 3.9.0 in model and strategy
Parameter names now match QuickAdapterRegressorV3.py implementation.
Jérôme Benoit [Sun, 28 Dec 2025 18:51:56 +0000 (19:51 +0100)]
refactor(quickadapter)!: normalize tunables namespace for semantic consistency (#26)
* refactor(quickadapter): normalize tunables namespace for semantic consistency
Rename config keys and internal variables to follow consistent naming conventions:
- `_candles` suffix for time periods in candle units
- `_fraction` suffix for values in [0,1] range
- `_multiplier` suffix for scaling factors
- `_method` suffix for algorithm selectors
Internal variable renames for code consistency:
- threshold_outlier → outlier_threshold_fraction
- thresholds_alpha → soft_extremum_alpha
- extrema_fraction → keep_extrema_fraction (local vars and function params)
- _reversal_lookback_period → _reversal_lookback_period_candles
- natr_ratio → natr_multiplier (zigzag function param)
All deprecated aliases emit warnings and remain functional for backward compatibility.
* chore(quickadapter): remove temporary audit file from codebase
* refactor(quickadapter): align constant names with normalized tunables
Rename class constants to match the normalized config key names:
- PREDICTIONS_EXTREMA_THRESHOLD_OUTLIER_DEFAULT → PREDICTIONS_EXTREMA_OUTLIER_THRESHOLD_FRACTION_DEFAULT
- PREDICTIONS_EXTREMA_THRESHOLDS_ALPHA_DEFAULT → PREDICTIONS_EXTREMA_SOFT_EXTREMUM_ALPHA_DEFAULT
- PREDICTIONS_EXTREMA_EXTREMA_FRACTION_DEFAULT → PREDICTIONS_EXTREMA_KEEP_EXTREMA_FRACTION_DEFAULT
* fix(quickadapter): rename outlier_threshold_fraction to outlier_threshold_quantile
The value (e.g., 0.999) represents the 99.9th percentile, which is
mathematically a quantile, not a fraction. This aligns the naming with
the semantic meaning of the parameter.
* fix(quickadapter): add missing deprecated alias support
* refactor(quickadapter): rename safe configuration value retrieval helper
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor(quickadapter): rename natr_ratio_fraction to natr_multiplier_fraction
- Align naming with label_natr_multiplier for consistency
- Rename get_config_value_with_deprecated_alias to get_config_value
* refactor(quickadapter): centralize label_natr_multiplier migration in get_label_defaults
- Move label_natr_ratio -> label_natr_multiplier migration to get_label_defaults()
- Update get_config_value to migrate in-place (pop old key, store new key)
- Remove redundant get_config_value calls in Strategy and Model __init__
- Simplify cached properties to use .get() since migration is done at init
- Rename _CUSTOM_STOPLOSS_NATR_RATIO_FRACTION to _CUSTOM_STOPLOSS_NATR_MULTIPLIER_FRACTION
* fix(quickadapter): check that df columns exist before using them
Jérôme Benoit [Sun, 28 Dec 2025 01:16:02 +0000 (02:16 +0100)]
feat(quickadapter): add HistGradientBoostingRegressor support (#25)
* feat(quickadapter): add HistGradientBoostingRegressor support
Add sklearn's HistGradientBoostingRegressor as a third regressor option.
- Add 'histgradientboostingregressor' to Regressor type and REGRESSORS
- Implement fit_regressor() with X_val/y_val support and early stopping
- Add native sklearn hyperparameters to get_optuna_study_model_parameters()
- Return empty callbacks list (no Optuna pruning callback support)
- Log warning when init_model is provided (not supported)
* fix(quickadapter): address PR review comments for HistGradientBoostingRegressor
Jérôme Benoit [Tue, 23 Dec 2025 19:13:18 +0000 (20:13 +0100)]
refactor(ReforceXY): PBRS refactoring, bug fix, and documentation harmonization
This commit includes three major improvements to the PBRS implementation:
1. Bug Fix: idle_factor calculation
- Fixed incorrect variable reference in reward_space_analysis.py:625
- Changed 'factor' to 'base_factor' in idle_factor formula
- Formula: idle_factor = base_factor * (profit_aim / risk_reward_ratio) / 4.0
- Also fixed in test_reward_components.py and ReforceXY.py
2. Refactoring: Separation of concerns in PBRS calculation
- Renamed apply_potential_shaping() → compute_pbrs_components()
- Removed base_reward parameter from PBRS functions
- PBRS functions now return only shaping components
- Caller responsible for: total = base_reward + shaping + entry + exit
- Kept deprecated wrapper for backward compatibility
- Updated ReforceXY.py with parallel changes
- Adapted tests to new function signatures
3. Documentation: Complete mathematical notation harmonization
- Achieved 100% consistent notation across both implementations
- Standardized on Greek symbols: Φ(s), γ, Δ(s,a,s')
- Eliminated mixing of word forms (Phi/gamma/Delta) with symbols
- Harmonized docstrings to 156-169 lines with identical theory sections
- Added cross-references between implementations
- Fixed all instances of Δ(s,s') → Δ(s,a,s') to include action parameter
Files modified:
- reward_space_analysis/reward_space_analysis.py: Core refactoring + docs
- user_data/freqaimodels/ReforceXY.py: Parallel refactoring + docs
- tests/components/test_additives.py: Adapted to new signature
- tests/components/test_reward_components.py: Bug fix
- tests/api/test_api_helpers.py: Adapted to new signature
All 50 tests pass. Behavior preserved except for intentional bug fix.
repositories / freqai-strategies.git / log
