Jérôme Benoit [Fri, 1 May 2026 19:03:15 +0000 (21:03 +0200)]
fix: pin pandas>=3.0 in Dockerfile and bump version to 3.11.7
Prevent silent pandas downgrade to 2.x during pip install, which
causes dtype mismatches with freqtrade 2026.4 date handling code.
Includes epoch-ms range validation in ensure_datetime_series.
Jérôme Benoit [Fri, 1 May 2026 14:03:12 +0000 (16:03 +0200)]
fix: validate epoch-ms range before converting int64 date columns
Reject int64 values outside [2010, 2035] epoch-ms range to fail fast
on corrupted data instead of silently producing wrong dates. Catches
nanosecond/microsecond values that would pass the int64 dtype check
but produce garbage timestamps if interpreted as milliseconds.
Jérôme Benoit [Fri, 1 May 2026 13:46:19 +0000 (15:46 +0200)]
fix: add 30min stop_grace_period to prevent data corruption on shutdown
FreqAI training can take minutes to hours. Docker's default 10s grace
period causes SIGKILL mid-write, corrupting feather/pickle files.
Give freqtrade up to 30 minutes to finish training and flush data
before Docker sends SIGKILL.
Jérôme Benoit [Fri, 1 May 2026 10:32:33 +0000 (12:32 +0200)]
fix: align ensure_datetime_series with freqtrade data handler pattern
Chain .dt.as_unit("ms") to guarantee datetime64[ms, UTC] output
resolution regardless of pandas version, matching the contract
established in freqtrade commit 2c5dc72.
refactor: extract ensure_datetime_series helper for date dtype workaround
Centralizes the int64 epoch-ms vs datetime detection logic into a shared
helper. Handles both formats correctly: unit='ms' for int64, passthrough
for existing datetime columns.
fix: workaround freqtrade 2026.4 date column dtype regression
Freqtrade 2026.4 (commit 2c5dc72) changed feather/parquet handlers to
use .dt.as_unit("ms") instead of to_datetime(col, unit="ms", utc=True).
This breaks when data files store dates as int64 epoch-ms, causing
AttributeError in feature_engineering_standard.
Use pd.to_datetime(col, utc=True) defensively to handle both int64 and
datetime inputs.
Jérôme Benoit [Tue, 31 Mar 2026 00:29:28 +0000 (02:29 +0200)]
docs: fix semantic accuracy of README configuration tunables
- polyorder: correct range from int >= 1 to int >= 0 (savgol accepts degree-0)
- robust standardization: replace 'IQR' with '(Q₃-Q₁)' (quantiles are configurable)
- label_weights: broaden scope from 'distance calculations to ideal point' to 'trial selection methods'
- label_p_order: replace 'p-order parameter for distance metrics' with 'Lp exponent for parameterized metrics'
- label_density_aggregation_param: replace 'p-order' with 'Lp exponent' for consistency
Jérôme Benoit [Thu, 12 Feb 2026 23:10:08 +0000 (00:10 +0100)]
fix(ReforceXY): add context-aware guard for efficiency coefficient division
Prevent division explosion in _compute_efficiency_coefficient() when
max_unrealized_profit ≈ min_unrealized_profit by requiring a minimum
meaningful range based on pnl_target. Also adds validation warnings
for potential_gamma=0 and pnl_target<=0 edge cases.
Jérôme Benoit [Thu, 12 Feb 2026 14:17:10 +0000 (15:17 +0100)]
feat(ReforceXY): tune reward sensitivity and extend training period
- Increase pnl_amplification_sensitivity from 0.5 to 2.0 for stronger
reward signal differentiation
- Extend train_period_days from 60 to 120 for more training data
Jérôme Benoit [Mon, 9 Feb 2026 21:04:23 +0000 (22:04 +0100)]
fix(quickadapter): use Optuna params for TimeSeriesSplit gap calculation
Previously gap was calculated from ft_params with a hardcoded default, which could return incorrect values when Optuna optimized parameters. Also standardizes log message format to use [pair] prefix.
- Use test_size parameter in TimeSeriesSplit
- Remove unused dk parameter from _make_timeseries_split_datasets()
- Assign dk.data_dictionary = dd before logging
- Fix typo: train_test_test -> train_test_split in README
* docs: integrate data_split_parameters into tunables table
Remove standalone section and add parameters to existing table
with freqai. prefix for consistency.
* refactor: use FreqAI APIs for weight calculation and data dictionary
- Use dk.set_weights_higher_recent() instead of duplicating weight formula
- Use dk.build_data_dictionary() for consistent data structure
- Respects feature_parameters.weight_factor configuration
- Fix bug: was using data_kitchen_thread_count instead of weight_factor
* refactor: extract _apply_pipelines() to reduce code duplication
- Move pipeline definition and application logic to helper method
- Reduces train() override complexity while keeping same behavior
- Helper can be reused by future custom split implementations
* style: harmonize namespace and remove inline comments
- Rename DATA_SPLIT_METHODS to _DATA_SPLIT_METHODS (private tuple pattern)
- Reference DATA_SPLIT_METHOD_DEFAULT from _DATA_SPLIT_METHODS[0]
- Remove 22 inline comments to match self-documenting codebase style
* fix: align TimeSeriesSplit weight calculation with FreqAI semantics
Calculate weights on combined train+test set before splitting to maintain
temporal weight continuity, matching FreqAI's make_train_test_datasets behavior.
* feat: add gap=0 warning and improve TimeSeriesSplit validation
- Warn when gap=0 about look-ahead bias risk (reference label_period_candles)
- Add _compute_timeseries_min_samples() for accurate minimum sample calculation
- Account for gap and test_size in minimum sample validation
- Improve error message with all relevant parameters
* style: harmonize error messages with codebase conventions
- Use 'Invalid {param} value {value!r}: {constraint}' pattern
- Align with existing validation error format (lines 718, 1145)
* style: add cached set accessor for data split methods
- Add _data_split_methods_set() with @staticmethod @lru_cache
- Use QuickAdapterRegressorV3 prefix for class attribute access
- Use cached set for O(1) membership check in validation
* fix: address PR review comments for TimeSeriesSplit
- Use dd consistently in training logs instead of dk.data_dictionary
- Use self.data_split_parameters consistently in _apply_pipelines
- Add explicit type coercion for n_splits, gap, max_train_size
- Add validation for gap >= 0 and max_train_size >= 1
- Improve test_size validation: float in (0,1) as fraction, int >= 1 as count
- Fix _compute_timeseries_min_samples formula: (n_splits+1)*test_size + n_splits*gap
- Optimize tscv.split() iteration to avoid unnecessary list materialization
* fix: correct min_samples formula to match sklearn validation
sklearn validates: n_samples - gap - (test_size * n_splits) > 0
Correct formula: test_size * n_splits + gap + 1
* feat: auto-calculate TimeSeriesSplit gap from label_period_candles
When gap=0 is configured, automatically set gap to label_period_candles
to prevent look-ahead bias from overlapping label windows. This ensures
temporal separation between train and test sets without requiring manual
configuration.
* fix: remove redundant time import shadowing module
* fix: correct min_samples formula for dynamic test_size and document test_size param
* docs: clarify test_size default per split method
* refactor: move DependencyException import to file header
* style: use class name for class constant access
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* docs: use Python None instead of null in README
* docs: fix train_test_split description (sequential, not random)
* fix: use explicit None check for max_train_size validation
* docs: clarify timeseries_split as chronological split, not cross-validation
* refactor(quickadapter): shorten log prefixes and tailor empty test set error by split method
* refactor(quickadapter): use index pattern for timeseries_split method constant
Replace string literals with index access pattern following existing
codebase convention for _DATA_SPLIT_METHODS.
Also renames variables for semantic clarity:
- test_size_param -> test_size
- feat_dict -> feature_parameters
* refactor(quickadapter): use _TEST_SIZE constant instead of hardcoded 0.1
* chore(quickadapter): bump version to 3.11.2
* fix(quickadapter): restore test_size parameter in TimeSeriesSplit
The test_size variable from data_split_parameters was being
immediately overwritten by a type annotation line, making it
always None regardless of user configuration.
Jérôme Benoit [Mon, 26 Jan 2026 20:27:52 +0000 (21:27 +0100)]
refactor(quickadapter): harmonize parameter naming in threshold computation
- Rename extrema_selection → selection_method to match tunable name
- Rename pred_extrema → pred_label for consistency across methods
- Rename n_extrema → n_values in _build_weights_array (generic function)
- Fix bug: use default_weight param instead of constant in early return
Jérôme Benoit [Mon, 26 Jan 2026 12:23:51 +0000 (13:23 +0100)]
refactor(quickadapter): add format_dict helper and improve numeric formatting
- Add format_dict() with singledispatch for type-safe dict/params formatting
- Refactor format_number() with unified significant digits formula
- Replace raw dict logging with format_dict() across strategy and model
- Remove redundant _format_label_method_config method
- Bump version to 3.11.1
- Refactor label processing into 4 orthogonal phases:
1. Weighting: apply weights to raw label values per column
2. Smoothing: smooth weighted values per column
3. Pipeline: LabelTransformer standardization per column
4. Prediction: threshold calculation per column
- Loop over LABEL_COLUMNS for weighting and smoothing in set_freqai_targets()
- Loop over dk.label_list for thresholds in fit_live_predictions()
- All config helpers return {default, columns} structure with glob pattern support
- Rename ExtremaWeightingTransformer to LabelTransformer
- Harmonize namespace: label_weighting, label_smoothing, label_prediction
- Backward compatible with flat configs and legacy column names
* refactor: remove deprecated internal APIs
Remove unused deprecated functions that were replaced by the orthogonal
label processing architecture:
- get_label_transformer_config() from Utils.py
- get_label_transformer_config import from QuickAdapterV3.py
- extrema_smoothing property from QuickAdapterV3.py
* refactor: use DEFAULTS_LABEL_PREDICTION for outlier_quantile fallback
* refactor: make label weighting generic with metrics dict
- compute_label_weights() takes generic metrics dict instead of hardcoded params
- _compute_combined_weights() takes generic metrics dict
- apply_label_weighting() takes generic metrics dict
- Caller builds metrics dict, making weighting truly transverse to any label
* refactor: centralize deprecation handling with PARAM_DEPRECATIONS table
* refactor: call resolve_deprecated_params once at startup
- Change resolve_deprecated_params to modify dict in-place (returns None)
- Centralize all deprecation calls in bot_start() and __init__()
- Remove calls from properties and utility functions that run multiple times
- This ensures deprecation warnings are logged once, not repeatedly
* fix: resolve deprecations in __init__ before regressor loads
- Move deprecation resolution from bot_start() to Strategy.__init__()
so it runs before FreqaiModel.__init__() (which reads same config)
- Remove label_transformer legacy support (never released)
- Simplify label_weighting/label_pipeline properties
- Keep regressor-specific deprecations in regressor __init__
* fix: address PR review comments
- Fix label_weighting['strategy'] KeyError by using ['default']['strategy']
- Respect label_prediction.method='none' in min_max_pred()
- Use float('inf') specificity for exact matches in get_column_config
- Reuse Utils.get_column_config in LabelTransformer
- Default label_smoothing method to 'gaussian'
* refactor: unify threshold column naming and soft_extremum_alpha
- Remove MINIMA_THRESHOLD_COLUMN/MAXIMA_THRESHOLD_COLUMN constants
- Use uniform {label}_minima_threshold/{label}_maxima_threshold for all labels
- Rename internal soft_alpha to soft_extremum_alpha for consistency with config
- Remove redundant docstrings from LabelTransformer (code is self-documenting)
* refactor: cleanup docstrings and rename internal functions
* refactor: make label_pipeline orthogonal from label_weighting
* refactor: rename get_column_config to get_label_column_config
* fix: add missing method field to label_prediction logging
* refactor: per-column logging and deprecate label_smoothing.window
- Update logging in QuickAdapterV3 and QuickAdapterRegressorV3 to show
resolved per-column configs instead of just defaults with override keys
- Move get_label_column_config() to LabelTransformer.py (re-export from Utils)
- Add deprecation mapping for label_smoothing.window -> window_candles
- Fix extrema_direction undefined variable bug in populate_any_indicators
* fix: correct deprecation mappings for label_prediction params
* refactor: move label_pipeline property and logging to regressor
- Move label_pipeline property from QuickAdapterV3 strategy to QuickAdapterRegressorV3
- Move Pipeline configuration logging from _log_strategy_configuration() to
_log_model_configuration()
- Simplify define_label_pipeline() to use self.label_pipeline property
- Remove unused get_label_pipeline_config import from strategy
- Rename local variable label_weighting to label_weighting_raw for consistency
* fix: import get_label_column_config from LabelTransformer
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* refactor(quickadapter): replace string literals with constant references in LabelTransformer
* refactor(quickadapter): use per-column prediction config in regressor and strategy
* fix: reference correct config paths for label processing
Signed-off-by: Jérôme Benoit <jerome.benoit@piment-noir.org>
* fix(quickadapter): warn when column doesn't match any config pattern
* feat(deprecation): support cross-section parameter moves
Extend PARAM_DEPRECATIONS to handle parameters that moved between
config sections, not just renames within the same section.
- Add tuple[str, str] value type for (old_section, old_key) moves
- Add root_config parameter to resolve_deprecated_params()
- Add deprecation entries for 7 params moved from label_weighting
to label_pipeline: standardization, robust_quantiles,
mmad_scaling_factor, normalization, minmax_range, sigmoid_scale,
gamma
- Add call sites in QuickAdapterV3 and QuickAdapterRegressorV3
* refactor(quickadapter): replace imperative deprecation handling with declarative path-based migrations
- Replace PARAM_DEPRECATIONS dict and resolve_deprecated_params() with
CONFIG_MIGRATIONS tuple and migrate_config()
- Single migrate_config() call in __init__ replaces 6+ resolve_deprecated_params() calls
- Fix bug in set_freqai_targets: move maxima/minima column creation after weighting
- Fix DI_value_param assignment to only occur when Weibull fit succeeds
* refactor(validation): replace imperative validation with declarative system
- Add dataclass-based validators (_EnumValidator, _NumericValidator, etc.)
- Replace ~240 lines of repetitive validation code with _validate_params()
- Consolidate type aliases in LabelTransformer.py (avoid duplicates)
- Fix pyright errors: float() casts, np.asarray() for pmean returns
- Use np.nan as default for optuna .get() (proper 'no value' sentinel)
- Add pyright to requirements-dev.txt
* chore(ReforceXY): add pyright to dev dependencies
- Move DI_value stats computation before label loop
- Unify warmed_up conditional to single if/else block
- Always set threshold values (defaults when not warmed up)
* refactor(quickadapter): add OPTUNA_*_DEFAULT constants and fix static member access
- Add OPTUNA_*_DEFAULT class constants for n_jobs, n_trials, timeout,
n_startup_trials, min_resource, label_candles_step, space_reduction,
space_fraction, and seed
- Update _optuna_config property to use constants instead of hardcoded values
- Update all .get() calls to use constants as defaults for type safety
- Fix static method/property access: use QuickAdapterRegressorV3.method()
instead of self.method() for static members
- Add assertions for narrowing Optional types (weights)
- Fix min_max_pred signature to accept Optional[int] for label_period_candles
Reduces pyright errors from 174 to 158 (-16)
* fix(quickadapter): default label_prediction method to 'thresholding' for backward compatibility
DEFAULTS_LABEL_PREDICTION['method'] was 'none' which broke backward
compatibility - legacy configs without explicit method would skip
threshold computation. Changed to 'thresholding' to preserve historical
behavior where thresholds were always computed by default.
Jérôme Benoit [Mon, 12 Jan 2026 15:58:52 +0000 (16:58 +0100)]
perf(quickadapter): eliminate ~15k np.log() recalculations via pure log space (#41)
* perf(zigzag): eliminate ~15k np.log() recalculations via pure log space
Comprehensive optimization of zigzag() function to operate entirely in
logarithmic space, eliminating redundant np.log() recalculations.
**Performance Impact:**
- ~11,000-15,000 fewer np.log() calls per zigzag() execution
- Pre-computation: ~10,000 calls eliminated
- Pure log space conversion: ~1,050-5,100 calls eliminated
**Implementation Changes:**
Utils.py (zigzag function):
- Pre-compute log arrays once: closes_log, highs_log, lows_log (L1195-1199)
- Convert update_candidate_pivot() to accept log values (L1245)
- Convert add_pivot() to accept log values (L1401)
- Convert initial phase to log space (L1531-1569)
- Convert main loop comparisons to log space (L1583-1615)
- Rename top_change_percent() → top_log_return() (L813)
- Rename bottom_change_percent() → bottom_log_return() (L834)
- Convert efficiency ratio calculations to log space (L1343, L1368)
**API Changes:**
- zigzag() now returns pivots_values_log instead of pivots_values
- calculate_pivot_metrics() accepts log values directly
**Callers Updated:**
- QuickAdapterV3.py: Use renamed functions, add TODO comments (L674, L676, L702)
- QuickAdapterRegressorV3.py: Use len(pivots_indices) instead of len(pivots_values) (L3350, L3396)
**Mathematical Correctness:**
- Maintains semantic equivalence via log monotonicity: a > b ⟺ log(a) > log(b)
- Provides symmetric treatment of returns in log space
- All comparisons and calculations mathematically equivalent
**Breaking Changes (Future):**
- Added TODO comments for feature renaming (requires model retraining)
- %-tcp-period → %-top_log_return-period
- %-bcp-period → %-bottom_log_return-period
- %-close_pct_change → %-close_log_return
* refactor(zigzag): harmonize log variable naming to _log suffix
repositories / freqai-strategies.git / log
