Machine Learning Imputation Strategy¶

Back to README

This document details the statistical reconstruction phase of the pipeline. Distinct from the transformation module, which handles upstream data cleaning and harmonization, this module addresses structural gaps (e.g., long-term sensor outages, historical backfilling) using a physics-informed machine learning strategy.

Objective¶

The objective is to synthesize a complete reference dataset that minimizes structural missingness while strictly avoiding look-ahead bias and preserving physical constraints. The pipeline guarantees that:

Causality is enforced: Exogenous drivers (e.g., weather) are imputed before endogenous targets (e.g., load and generation). This follows a strict Directed Acyclic Graph (DAG)—a one-way dependency model that prevents circular logic.
Physical bounds are respected: Constraints such as “solar power equals zero at night” or “no nuclear production in regions without reactors” are applied deterministically.
Continuity is preserved: Predictions are post-processed with a Brownian bridge (a stochastic method tying a sequence to fixed start and end points) to ensure \(C^0\) continuity—meaning the reconstructed curve seamlessly connects to the real data without abrupt jumps.
Data leakage is prevented: Models are trained strictly on historical data (\(T < T_{cutoff}\)) to infer recent gaps.

Imputation Engines: TS-ICL (Default) vs. XGBoost¶

The imputation phase is pluggable. It runs with either a foundation-model engine (TS-ICL, the default) [Le Naour et al., 2026] or an extreme gradient-boosting engine (XGBoost) [Chen and Guestrin, 2016]:

energy-impute                    # TS-ICL (default)
energy-impute --engine xgboost   # Gradient boosting

Outputs are isolated in engine-specific directories to prevent overwriting:

Outputs/Imputed/tsicl/     dataset_*_imputed.csv     (default)
Outputs/Imputed/xgboost/   dataset_*_imputed.csv

Engine	Description	Covariates	Training
TS-ICL (Default)	Zero-shot probabilistic time-series foundation model.	Same exogenous columns, same step.	None (in-context).
XGBoost	Physics-informed gradient boosting (the “Cascade”).	Exogenous + calendar + J-1/J-7 lags.	Per-column, per-tier.

Dense-Gap Routing (TS-ICL): The engine can classify columns above a missing-data threshold (dense_threshold, default 0.35) as dense gaps and fill them with fast, capped-length time-interpolation instead of the foundation model, reserving TS-ICL for sparse or structural gaps.

Environment: TS-ICL requires Python ≥ 3.12 and the [icl] extra (pip install -e ".[icl]"). If tsicl cannot be imported in the active environment, energy-impute will log a warning and automatically fall back to XGBoost.

Engine Benchmark (Masked-Holdout R²)¶

The energy-impute-benchmark script scores both engines on a fully-present 2023 slice by intentionally masking 15% of the data and measuring the reconstruction error. Two gap structures are tested:

Scatter: Random single-point dropouts (where neighbors remain observed).
Block: Random 1-day (48-step) outages (simulating realistic structural failures).

Both engines receive the exact same exogenous covariates at the same 30-min step. XGBoost additionally receives its engineered calendar and lagged features.

Gap Structure	Target	TS-ICL R²	XGBoost R²	TS-ICL MAE	XGBoost MAE
Scatter (random points)	Load	0.9999	0.9859	75 MW	827 MW
Scatter	Solar	0.9983	0.9926	52 MW	152 MW
Scatter	Pump	0.9866	0.8537	52 MW	212 MW
Block (1-day outages)	Load	0.9930	0.9550	535 MW	1266 MW
Block	Solar	0.9863	0.9878	181 MW	192 MW
Block	Pump	0.7800	0.7726	271 MW	296 MW

Takeaway: TS-ICL outperforms XGBoost in 5 out of 6 test cases, achieving this entirely zero-shot (without per-column training). Its advantage is highly pronounced for scattered gaps, where it efficiently exploits local temporal context (e.g., load MAE of 75 MW vs 827 MW). For long structural block-gaps, XGBoost performs similarly to TS-ICL.

Note on performance metrics: The audit R² found in Outputs/Imputed/<engine>/Logs/ reflects XGBoost’s training fit. Because TS-ICL is zero-shot, it only logs imputation counts in standard runs. Researchers should rely on the Outputs/Logs/imputation_engine_benchmark.csv for objective model comparison.

The “Cascade” Algorithm (XGBoost Engine)¶

The XGBoost imputation process is orchestrated by impute_missing_values.py and strictly follows a sequential dependency order to ensure downstream models have access to complete feature sets.

Phase 1: Deterministic & Structural Repairs¶

Before applying statistical inference, the pipeline prunes the search space using physical constraints:

Solar night constraint: Solar values are forced to \(0.0\) during nighttime (using either national radiation signals or a 23:00-05:00 heuristic).
Structural zeros: Enforces \(0.0\) for generation technologies physically absent in specific regions to prevent the model from hallucinating output (e.g., nuclear generation in Brittany).
Temporal cutoffs: Enforces NaN for datasets unavailable before a specific historical date.

Phase 2: Exogenous Reconstruction (Weather)¶

Weather variables drive the entire energy system and must be reconstructed first.

Algorithm: CascadingImputer utilizing recursive feature relaxation.
Strategy: 1. Spatial kriging: For a regional target, the model prioritizes spatial neighbors alongside the national baseline (e.g., using meteo_wind_bretagne to infer meteo_wind_normandie).

Physical coupling: Leverages thermodynamic relationships (e.g., atmospheric pressure infers wind; radiation infers temperature).

Phase 3: Endogenous Reconstruction (The Grid DAG)¶

To prevent circular dependencies, power grid flows are reconstructed following the concept of “residual load” used in power system economics. Variables are imputed in a strict 7-step causal order:

Exogenous Drivers (Weather): The independent variables that drive the physical system (completed in Phase 2).
Demand Side (Load): The primary system constraint, driven by heating degree days, calendar events, and nebulosity. Must be established first.
Fatal Generation (Wind, Solar): Variable Renewable Energy (VRE) sources. As they have near-zero marginal cost and grid priority, their dispatch is purely weather-driven and independent of the remaining grid state.
Baseload & Modulation (Nuclear, Bioenergy): In France, nuclear performs load-following maneuvers based on the residual load (Load minus VRE).
Interconnections (Cross-Border Exchange): Net exports are driven by international price differentials, effectively proxied by the national residual load.
Flexibility & Storage (Pumped Hydro): Arbitrage assets that consume surplus generation and release it during deficits.
Dispatchable Thermal (Gas, Coal, Oil): The peak-load producers situated at the top of the “merit order” (the economic dispatch sequence). They are imputed last as they respond to the final remaining grid imbalance.

Limitations & Stationarity Assumptions¶

The pipeline uses three temporal fallback tiers to manage autoregressive memory. Users should interpret “Tier 3” (Structural Fallback) imputations with appropriate nuance:

Short-Term (Tiers 1 & 2): Technical gaps reconstructed with high confidence using immediate historical lags (e.g., \(J-1\) to \(J-14\)).
Long-Term (Tier 3): Historical backfilling heavily relies on exogenous variables and assumes stationarity—the statistical assumption that the underlying physical relationship has not fundamentally changed over time (e.g., the relationship between temperature and load in 2022 is assumed identical to 2015).
Drift Risk: The stationarity assumption holds well for raw physical phenomena (e.g., wind speed to power) but may degrade for socio-economic variables (e.g., regional industrial consumption changing due to factory closures over a decade).

Machine Learning Core: Engine Details (`model.py` & `predictive.py`)¶

1. Physics-Informed Enhancements (XGBoost)¶

The standard gradient boosting algorithm is augmented with domain-specific techniques:

Thermal Inertia (Exponential Smoothing): Buildings act as thermal buffers. To capture this memory, temperature features are pre-processed using an Exponentially Weighted Moving Average (EWMA) with a decay factor \(\alpha \approx 0.05\).
Brownian Bridge Correction: To eliminate discontinuities where imputed data meets real historical data, a Brownian bridge post-processor distributes residual errors across the gap:

\[y_{final}(t) = y_{pred}(t) + bias_{start} \cdot (1-w(t)) + bias_{end} \cdot w(t)\]

This ensures \(C^0\) continuity at both boundaries of the prediction window.

2. Recursive Feature Relaxation¶

To handle sparse data availability, the CascadingImputer algorithm recursively relaxes its requirements:

Initial pass: Attempts to fill gaps using the complete feature set (national proxy + spatial neighbors + physics).
Recursive degradation: For timestamps where the full set is unavailable, the algorithm identifies the “limiting feature” (the one most frequently missing), removes it from the training set, and retrains a fallback model.
Loop: This iterates until all gaps are filled or no logical features remain.