TAM Feature Extraction¶

Back to README

This document describes how the canonical 30-minute dataset produced by the assembly and imputation phases is converted into the feature matrices consumed by the TAM (Time series Additive Model) family of forecasting benchmarks [Allioux and Goude, 2026]. This is a downstream delivery step, not part of the ETL pipeline itself: it reads an already-assembled dataset and re-shapes it to match a specific modeling package’s expected schema.

Objective¶

scripts/modeling/prep_tam_features.py engineers the model-specific variables (autoregressive lags, thermal-inertia smoothing, legacy unit conventions) that the TAM formulas require, and exports them as flat, comma-separated matrices independent of the internal pipeline schema. This isolates the ETL/imputation pipeline’s column-naming and unit conventions from the modeling package’s own conventions, so the two can evolve independently.

Running the Extraction¶

python scripts/modeling/prep_tam_features.py                  # TS-ICL-imputed input (default)
python scripts/modeling/prep_tam_features.py --engine xgboost  # XGBoost-imputed input
python scripts/modeling/prep_tam_features.py --fast-test       # Also emit a non-imputed matrix

The --engine flag selects which imputation engine’s output to source from (see IMPUTATION.md); it has no effect on the feature engineering itself. --fast-test additionally builds a matrix directly from the raw assembled dataset, bypassing imputation entirely, for rapid iteration when the imputation phase has not been run or is still in progress.

Outputs¶

All matrices are written to Outputs/TAM_Delivery/:

File	Source	Purpose
`dataset_tam_new_pipeline.csv`	Imputed canonical matrix	Primary delivery: reflects the pipeline’s active data mix. Replaces `dataset_national.csv` (same filename) in `tam-ml` from version 1.2.7 onward.
`dataset_tam_legacy_aligned.csv`	Imputed canonical matrix + historical temperature overlay	Matches the `dataset_national.csv` conventions used by `tam-ml` versions up to and including 1.2.6, for direct benchmark comparability across the version transition.
`dataset_tam_not_imputed_fast_test.csv`	Raw assembled matrix	`--fast-test` only. Autoregressive lags and thermal smoothing propagate any NaN blocks present in the source.
`dataset_tam_not_imputed_fast_test_aligned.csv`	Raw assembled matrix + historical temperature overlay	`--fast-test` only, legacy-aligned variant of the above.

Feature Engineering¶

Starting from the assembled dataset’s rte_load_france and meteo_temperature_celsius_france_load columns (renamed to Load and temperature):

Autoregressive lags: Load_d1 and Load_d7 are the load 48 and 336 steps prior (1 and 7 days at the 30-minute step).
Unit conversion: temperature is converted from Celsius to Kelvin, matching the TAM formulas’ expected input scale.
Thermal inertia: temperature_smooth_990/_950 apply exponentially weighted moving averages (\(\alpha = 0.010\) and \(\alpha = 0.050\)) to approximate a building’s thermal lag, with _max_smooth_*/_min_smooth_* variants tracking the 48-step (24-hour) rolling extremes of each smoothed series. This mirrors the thermal-inertia treatment used in the XGBoost imputation engine (see IMPUTATION.md).

Legacy Alignment¶

The legacy-aligned outputs additionally apply three overlays that reproduce conventions specific to dataset_national.csv as used by tam-ml versions up to and including 1.2.6, mirrored in this repository as tests/baselines/dataset_national_old.csv:

fill_temperature_from_old: Overrides temperature with the historical dataset’s own values where available, so the aligned output uses the same temperature source as the reference dataset rather than the current pipeline’s.
replicate_old_timezone_leakage: Shifts temperature forward by 1 step (winter) or 2 steps (summer), reproducing a timezone-handling convention present in the legacy dataset’s temperature series.
replicate_old_load_shift: Shifts Load back by one 30-minute step before computing the autoregressive lags, reproducing the legacy dataset’s end-of-period timestamp convention for Load.

These three flags exist solely to reproduce known legacy conventions; the primary delivery matrix (dataset_tam_new_pipeline.csv) does not use any of them.

Verification¶

All three matrices below were scored with the same model configuration: a TAM-Spline formula, fit on years before 2022 with test on 2024. RMSE is reported on the test period (year 2024), for both the static fit (StaticTAM) and its online-adapted counterpart (AdaptiveTAM):

Dataset	StaticTAM (Test RMSE)	AdaptiveTAM (Test RMSE)
`dataset_national_old.csv` (`tam-ml` <= 1.2.6 reference)	927.54	836.47
`dataset_tam_legacy_aligned.csv` (<= 1.2.6 conventions)	923.18	833.89
`dataset_tam_new_pipeline.csv` (-> `tam-ml` >= 1.2.7)	920.02	823.43

dataset_tam_legacy_aligned.csv lands close to the reference, as expected from reproducing its conventions; dataset_tam_new_pipeline.csv outperforms both on both models, confirming the accuracy gain reflects genuine pipeline improvements rather than an artifact of the legacy-alignment overlays.

Back to README