# TAM Feature Extraction

**[Back to README](../README.md)**

This document describes how the canonical 30-minute dataset produced by the assembly and imputation phases is converted into the feature matrices consumed by the TAM (Time series Additive Model) family of forecasting benchmarks {cite:p}`tam2026package`. This is a downstream delivery step, not part of the ETL pipeline itself: it reads an already-assembled dataset and re-shapes it to match a specific modeling package's expected schema.

## Objective

`scripts/modeling/prep_tam_features.py` engineers the model-specific variables (autoregressive lags, thermal-inertia smoothing, legacy unit conventions) that the TAM formulas require, and exports them as flat, comma-separated matrices independent of the internal pipeline schema. This isolates the ETL/imputation pipeline's column-naming and unit conventions from the modeling package's own conventions, so the two can evolve independently.

## Running the Extraction

```bash
python scripts/modeling/prep_tam_features.py                  # TS-ICL-imputed input (default)
python scripts/modeling/prep_tam_features.py --engine xgboost  # XGBoost-imputed input
python scripts/modeling/prep_tam_features.py --fast-test       # Also emit a non-imputed matrix

```

The `--engine` flag selects which imputation engine's output to source from (see [IMPUTATION.md](IMPUTATION.md)); it has no effect on the feature engineering itself. `--fast-test` additionally builds a matrix directly from the raw assembled dataset, bypassing imputation entirely, for rapid iteration when the imputation phase has not been run or is still in progress.

## Outputs

All matrices are written to `Outputs/TAM_Delivery/`:

| File | Source | Purpose |
| --- | --- | --- |
| `dataset_tam_new_pipeline.csv` | Imputed canonical matrix | Primary delivery: reflects the pipeline's active data mix. Replaces `dataset_national.csv` (same filename) in `tam-ml` from version 1.2.7 onward. |
| `dataset_tam_legacy_aligned.csv` | Imputed canonical matrix + historical temperature overlay | Matches the `dataset_national.csv` conventions used by `tam-ml` versions up to and including 1.2.6, for direct benchmark comparability across the version transition. |
| `dataset_tam_not_imputed_fast_test.csv` | Raw assembled matrix | `--fast-test` only. Autoregressive lags and thermal smoothing propagate any NaN blocks present in the source. |
| `dataset_tam_not_imputed_fast_test_aligned.csv` | Raw assembled matrix + historical temperature overlay | `--fast-test` only, legacy-aligned variant of the above. |

## Feature Engineering

Starting from the assembled dataset's `rte_load_france` and `meteo_temperature_celsius_france_load` columns (renamed to `Load` and `temperature`):

* **Autoregressive lags:** `Load_d1` and `Load_d7` are the load 48 and 336 steps prior (1 and 7 days at the 30-minute step).
* **Unit conversion:** `temperature` is converted from Celsius to Kelvin, matching the TAM formulas' expected input scale.
* **Thermal inertia:** `temperature_smooth_990`/`_950` apply exponentially weighted moving averages ($\alpha = 0.010$ and $\alpha = 0.050$) to approximate a building's thermal lag, with `_max_smooth_*`/`_min_smooth_*` variants tracking the 48-step (24-hour) rolling extremes of each smoothed series. This mirrors the thermal-inertia treatment used in the XGBoost imputation engine (see [IMPUTATION.md](IMPUTATION.md)).

### Legacy Alignment

The legacy-aligned outputs additionally apply three overlays that reproduce conventions specific to `dataset_national.csv` as used by `tam-ml` versions up to and including 1.2.6, mirrored in this repository as `tests/baselines/dataset_national_old.csv`:

* **`fill_temperature_from_old`:** Overrides `temperature` with the historical dataset's own values where available, so the aligned output uses the same temperature source as the reference dataset rather than the current pipeline's.
* **`replicate_old_timezone_leakage`:** Shifts `temperature` forward by 1 step (winter) or 2 steps (summer), reproducing a timezone-handling convention present in the legacy dataset's temperature series.
* **`replicate_old_load_shift`:** Shifts `Load` back by one 30-minute step before computing the autoregressive lags, reproducing the legacy dataset's end-of-period timestamp convention for `Load`.

These three flags exist solely to reproduce known legacy conventions; the primary delivery matrix (`dataset_tam_new_pipeline.csv`) does not use any of them.

## Verification

All three matrices below were scored with the same model configuration: a `TAM-Spline` formula, fit on years before 2022 with test on 2024. RMSE is reported on the test period (year 2024), for both the static fit (`StaticTAM`) and its online-adapted counterpart (`AdaptiveTAM`):

| Dataset | StaticTAM (Test RMSE) | AdaptiveTAM (Test RMSE) |
| --- | --- | --- |
| `dataset_national_old.csv` (`tam-ml` <= 1.2.6 reference) | 927.54 | 836.47 |
| `dataset_tam_legacy_aligned.csv` (<= 1.2.6 conventions) | 923.18 | 833.89 |
| `dataset_tam_new_pipeline.csv` (-> `tam-ml` >= 1.2.7) | 920.02 | 823.43 |

`dataset_tam_legacy_aligned.csv` lands close to the reference, as expected from reproducing its conventions; `dataset_tam_new_pipeline.csv` outperforms both on both models, confirming the accuracy gain reflects genuine pipeline improvements rather than an artifact of the legacy-alignment overlays.

---

**[Back to README](../README.md)**