# Machine Learning Imputation Strategy

**[Back to README](../README.md)**

This document details the statistical reconstruction phase of the pipeline. Distinct from the transformation module, which handles upstream data cleaning and harmonization, this module addresses structural gaps (e.g., long-term sensor outages, historical backfilling) using a physics-informed machine learning strategy.

## Objective

The objective is to synthesize a complete reference dataset that minimizes structural missingness while strictly avoiding look-ahead bias and preserving physical constraints. The pipeline guarantees that:

1. **Causality is enforced:** Exogenous drivers (e.g., weather) are imputed *before* endogenous targets (e.g., load and generation). This follows a strict Directed Acyclic Graph (DAG)—a one-way dependency model that prevents circular logic.
2. **Physical bounds are respected:** Constraints such as "solar power equals zero at night" or "no nuclear production in regions without reactors" are applied deterministically.
3. **Continuity is preserved:** Predictions are post-processed with a Brownian bridge (a stochastic method tying a sequence to fixed start and end points) to ensure $C^0$ continuity—meaning the reconstructed curve seamlessly connects to the real data without abrupt jumps.
4. **Data leakage is prevented:** Models are trained strictly on historical data ($T < T_{cutoff}$) to infer recent gaps.

---

## Imputation Engines: TS-ICL (Default) vs. XGBoost

The imputation phase is pluggable. It runs with either a foundation-model engine (**TS-ICL**, the default) {cite:p}`lenaour2026tsicl` or an extreme gradient-boosting engine (**XGBoost**) {cite:p}`chen2016xgboost`:

```bash
energy-impute                    # TS-ICL (default)
energy-impute --engine xgboost   # Gradient boosting

```

Outputs are isolated in engine-specific directories to prevent overwriting:

```text
Outputs/Imputed/tsicl/     dataset_*_imputed.csv     (default)
Outputs/Imputed/xgboost/   dataset_*_imputed.csv

```

| Engine | Description | Covariates | Training |
| --- | --- | --- | --- |
| **TS-ICL** (Default) | Zero-shot probabilistic time-series foundation model. | Same exogenous columns, same step. | None (in-context). |
| **XGBoost** | Physics-informed gradient boosting (the "Cascade"). | Exogenous + calendar + J-1/J-7 lags. | Per-column, per-tier. |

**Dense-Gap Routing (TS-ICL):** The engine can classify columns above a missing-data threshold (`dense_threshold`, default `0.35`) as dense gaps and fill them with fast, capped-length time-interpolation instead of the foundation model, reserving TS-ICL for sparse or structural gaps.

**Environment:** TS-ICL requires Python ≥ 3.12 and the `[icl]` extra (`pip install -e ".[icl]"`). If `tsicl` cannot be imported in the active environment, `energy-impute` will log a warning and automatically fall back to XGBoost.

### Engine Benchmark (Masked-Holdout R²)

The `energy-impute-benchmark` script scores both engines on a fully-present 2023 slice by intentionally masking 15% of the data and measuring the reconstruction error. Two gap structures are tested:

* **Scatter:** Random single-point dropouts (where neighbors remain observed).
* **Block:** Random 1-day (48-step) outages (simulating realistic structural failures).

Both engines receive the exact same exogenous covariates at the same 30-min step. XGBoost additionally receives its engineered calendar and lagged features.

| Gap Structure | Target | TS-ICL R² | XGBoost R² | TS-ICL MAE | XGBoost MAE |
| --- | --- | --- | --- | --- | --- |
| Scatter (random points) | Load | **0.9999** | 0.9859 | 75 MW | 827 MW |
| Scatter | Solar | **0.9983** | 0.9926 | 52 MW | 152 MW |
| Scatter | Pump | **0.9866** | 0.8537 | 52 MW | 212 MW |
| Block (1-day outages) | Load | **0.9930** | 0.9550 | 535 MW | 1266 MW |
| Block | Solar | 0.9863 | **0.9878** | 181 MW | 192 MW |
| Block | Pump | **0.7800** | 0.7726 | 271 MW | 296 MW |

**Takeaway:** TS-ICL outperforms XGBoost in 5 out of 6 test cases, achieving this entirely zero-shot (without per-column training). Its advantage is highly pronounced for scattered gaps, where it efficiently exploits local temporal context (e.g., load MAE of 75 MW vs 827 MW). For long structural block-gaps, XGBoost performs similarly to TS-ICL.

*Note on performance metrics:* The audit R² found in `Outputs/Imputed/<engine>/Logs/` reflects XGBoost's *training fit*. Because TS-ICL is zero-shot, it only logs imputation counts in standard runs. Researchers should rely on the `Outputs/Logs/imputation_engine_benchmark.csv` for objective model comparison.

---

## The "Cascade" Algorithm (XGBoost Engine)

The XGBoost imputation process is orchestrated by `impute_missing_values.py` and strictly follows a sequential dependency order to ensure downstream models have access to complete feature sets.

### Phase 1: Deterministic & Structural Repairs

Before applying statistical inference, the pipeline prunes the search space using physical constraints:

* **Solar night constraint:** Solar values are forced to $0.0$ during nighttime (using either national radiation signals or a 23:00-05:00 heuristic).
* **Structural zeros:** Enforces $0.0$ for generation technologies physically absent in specific regions to prevent the model from hallucinating output (e.g., nuclear generation in Brittany).
* **Temporal cutoffs:** Enforces `NaN` for datasets unavailable before a specific historical date.

### Phase 2: Exogenous Reconstruction (Weather)

Weather variables drive the entire energy system and must be reconstructed first.

* **Algorithm:** `CascadingImputer` utilizing recursive feature relaxation.
* **Strategy:** 1. **Spatial kriging:** For a regional target, the model prioritizes spatial neighbors alongside the national baseline (e.g., using `meteo_wind_bretagne` to infer `meteo_wind_normandie`).
2. **Physical coupling:** Leverages thermodynamic relationships (e.g., atmospheric pressure infers wind; radiation infers temperature).

### Phase 3: Endogenous Reconstruction (The Grid DAG)

To prevent circular dependencies, power grid flows are reconstructed following the concept of "residual load" used in power system economics. Variables are imputed in a strict 7-step causal order:

1. **Exogenous Drivers (Weather):** The independent variables that drive the physical system (completed in Phase 2).
2. **Demand Side (Load):** The primary system constraint, driven by heating degree days, calendar events, and nebulosity. Must be established first.
3. **Fatal Generation (Wind, Solar):** Variable Renewable Energy (VRE) sources. As they have near-zero marginal cost and grid priority, their dispatch is purely weather-driven and independent of the remaining grid state.
4. **Baseload & Modulation (Nuclear, Bioenergy):** In France, nuclear performs load-following maneuvers based on the residual load (Load minus VRE).
5. **Interconnections (Cross-Border Exchange):** Net exports are driven by international price differentials, effectively proxied by the national residual load.
6. **Flexibility & Storage (Pumped Hydro):** Arbitrage assets that consume surplus generation and release it during deficits.
7. **Dispatchable Thermal (Gas, Coal, Oil):** The peak-load producers situated at the top of the "merit order" (the economic dispatch sequence). They are imputed last as they respond to the final remaining grid imbalance.

## Limitations & Stationarity Assumptions

The pipeline uses three temporal fallback tiers to manage autoregressive memory. Users should interpret "Tier 3" (Structural Fallback) imputations with appropriate nuance:

* **Short-Term (Tiers 1 & 2):** Technical gaps reconstructed with high confidence using immediate historical lags (e.g., $J-1$ to $J-14$).
* **Long-Term (Tier 3):** Historical backfilling heavily relies on exogenous variables and assumes **stationarity**—the statistical assumption that the underlying physical relationship has not fundamentally changed over time (e.g., the relationship between temperature and load in 2022 is assumed identical to 2015).
* **Drift Risk:** The stationarity assumption holds well for raw physical phenomena (e.g., wind speed to power) but may degrade for socio-economic variables (e.g., regional industrial consumption changing due to factory closures over a decade).

---

## Machine Learning Core: Engine Details (`model.py` & `predictive.py`)

### 1. Physics-Informed Enhancements (XGBoost)

The standard gradient boosting algorithm is augmented with domain-specific techniques:

* **Thermal Inertia (Exponential Smoothing):** Buildings act as thermal buffers. To capture this memory, temperature features are pre-processed using an Exponentially Weighted Moving Average (EWMA) with a decay factor $\alpha \approx 0.05$.
* **Brownian Bridge Correction:** To eliminate discontinuities where imputed data meets real historical data, a Brownian bridge post-processor distributes residual errors across the gap:

$$y_{final}(t) = y_{pred}(t) + bias_{start} \cdot (1-w(t)) + bias_{end} \cdot w(t)$$



This ensures $C^0$ continuity at both boundaries of the prediction window.

### 2. Recursive Feature Relaxation

To handle sparse data availability, the `CascadingImputer` algorithm recursively relaxes its requirements:

1. **Initial pass:** Attempts to fill gaps using the complete feature set (national proxy + spatial neighbors + physics).
2. **Recursive degradation:** For timestamps where the full set is unavailable, the algorithm identifies the "limiting feature" (the one most frequently missing), removes it from the training set, and retrains a fallback model.
3. **Loop:** This iterates until all gaps are filled or no logical features remain.

---

**[Back to README](../README.md)**