FORCE: French Open Research Catalogue of Energy - A High-Performance Python Pipeline

Version: 1.1.1
License: LGPL-3.0-or-later.
Copyright (c): 2020-2026 EDF (Electricité De France).
Copyright (c): 2023 Sorbonne Université.

Active contributor: Yann Allioux (2020-2026)
Past contributor: Nathan Doumèche (2023-2025)
Scientific advisor: Yannig Goude

PyPI Version CI Docs License: LGPL v3 Python 3.10+ GitHub Repo stars DOI: Software DOI: Dataset

Abstract

FORCE provides an extract, load, and transform (ETL) pipeline to aggregate, harmonize, and validate multi-source energy data for the French power system. It addresses the challenge of heterogeneous data by unifying divergent temporal resolutions (e.g., 15-minute vs. 30-minute), spatial granularities, and physical units into a continuous time series matrix.

The package is built for researchers working on time series forecasting, grid economics, and climate-energy modeling. Relying on DuckDB as its analytical backend [Raasveldt and Mühleisen, 2019], FORCE consolidates data from four primary sources:

  1. RTE (Transmission): A multi-layer architecture tracking data maturity (real-time, consolidated, definitive) to synthesize a single reference record [RTE, 2026].

  2. Enedis (Distribution): Granular consumption profiles and decentralized generation metrics [Enedis, 2026].

  3. Météo-France (Weather): Synoptic observations transformed via asset-weighted spatial aggregation [Météo-France, 2026].

  4. ODRÉ (Assets): A georeferenced registry of power plants used to generate spatial weights [ODRÉ, 2026].

Documentation & Scientific Methodology

Each component of the pipeline is documented with full algorithmic transparency:

Component

Documentation Link

Key Algorithms

Calendar

CALENDAR.md

Skeleton generation, UTC-DST bridge, topological day typing [Etalab, 2026, Ministère de l'Éducation Nationale, 2026, Augusti, 2018].

RTE

RTE.md

Multi-version pivoting, conservative upsampling, maturity-aware deduplication.

Enedis

ENEDIS.md

Patch and fill strategy, profile reconstruction.

Météo

METEO.md

Multi-barycenter aggregation, vector decomposition (for circular wind variables).

Radiation

RADIATION.md

PVGIS POA (Plane of Array) & ERA5 GHI (Global Horizontal Irradiance) collection.

Assets

ODRE.md

Two-stage geocoding [OpenDataSoft, 2026], spatial centroids.

Imputation

IMPUTATION.md

Pluggable engines, recursive feature relaxation, spatial kriging (interpolation), 7-step causal DAG (Directed Acyclic Graph).

Hierarchies

HIERARCHIES.md

Verified column sum-trees, energy balance, calendar encodings.

Audit

audit/README.md

Per-dataset visualization suite, notebook-engine reproduction.

TAM Extraction

TAM.md

Autoregressive lag/thermal-inertia feature engineering, legacy dataset alignment.

Installation & Usage

1. Installation

FORCE requires Python 3.10 or higher (Python 3.12+ recommended for the full TS-ICL imputation engine).

As a package (for end users):

pip install force-dataset

From source (for development or running the pipeline). All dependencies are managed via pyproject.toml.

python -m venv .venv
# activate: PowerShell: .\.venv\Scripts\Activate.ps1 | bash/zsh: source .venv/bin/activate
pip install -e ".[dev]"

Optional extras include dev (tests/lint), docs (Sphinx), dashboard (Jupyter), climate (CDS/CMIP6), and icl (TS-ICL imputation engine, requiring Python >= 3.12).

2. Execution Pipeline

Execute the pipeline steps sequentially to build the dataset:

Phase A: Collection Fetches raw data from APIs. The state-aware engine detects data maturation (e.g., when real-time data is retroactively validated).

energy-collect

Phase B: Transformation Applies cleaning, upsampling, and technical repair. Structural gaps are strictly preserved as NaN.

energy-transform

Phase C: Database Audit Scans DuckDB files for temporal coverage gaps, empty tables, and schema anomalies. For diagnostic visualizations, use energy-viz-transform and energy-viz-assemble.

energy-check

Phase D: Assembly & Imputation Joins sources and applies physics-informed imputation.

energy-assemble
energy-impute  # Uses TS-ICL by default; pass '--engine xgboost' for gradient boosting

Scientific Method: Data Harmonization

The pipeline addresses specific challenges in energy data engineering:

1. Temporal Resolution Harmonization (30min to 15min)

The pipeline standardizes all data streams to a 15-minute resolution:

  • Energy Flows (Grid Data): We apply a conservative smoothing kernel that mathematically preserves the energy integral (MWh) over the block. Note: Minor numerical deviations may occur when enforcing physical bounds (e.g., clipping negative generation). Formula: \(y_{t+15} = \frac{3Y_t + Y_{t+30}}{4}\).

  • State Variables (Meteo Data): Instantaneous observations (e.g., temperature) use strict linear upsampling (\(limit=1\)) to calculate the intermediate point (\(t+15\)) without modifying the original anchors.

2. Spatio-Temporal Heterogeneity

A simple geographic average of weather stations introduces bias because energy assets are unevenly distributed.

  • Spatial Solution: We calculate usage-specific centroids (\(G_{load}\), \(G_{wind}\), \(G_{solar}\)) for every region. Station data is aggregated using a modified inverse distance weighting (IDW) algorithm. To ensure numerical stability, this IDW applies a 1 km softening factor and assumes an oblate spheroid Earth (\(R=6371\) km).

  • Temporal Solution: The pipeline computes time-varying weights (\(W_{r,y}\)). The influence of a region \(r\) on the national signal evolves annually \(y\) based on its actual market share. Formula: \(I_{national}(t) = \sum_{r} W_{r, y(t)} \cdot I_{r}(t)\)

3. Data Maturity & Versioning

Energy data evolves from real-time estimates to validated definitive records. For critical metrics like national load, the pipeline produces multiple time series versions:

  • _realtime: For simulating production forecasting.

  • _definitive: For training on ground truth.

  • _best: A synthesized golden record for general use.

4. Two-Stage Gap Handling Strategy

We separate technical repair from statistical reconstruction to guarantee data integrity:

Stage 1: Technical Repair (ETL Phase)

Targets micro-gaps (< 1h) caused by sensor dropouts.

  • Grid Data: Cyclical variables are repaired via a profile-guided bridge (\(D-7\)) to preserve intra-hour peaks. Stochastic variables (wind/solar) use linear interpolation.

  • Meteo Data: Utilizes lapse-rate aware k-Nearest Neighbors (k-NN) to prioritize vertical similarity, and Time-of-Day (TOD) fallbacks to bridge gaps when spatial neighbors are unavailable.

Stage 2: Statistical Reconstruction (Imputation Phase)

Targets structural outages and historical backfilling for gaps > 1h.

  • TS-ICL Engine (Default): A zero-shot time-series foundation model. It handles raw temporal context directly and ignores the training_cutoff_years parameter.

  • XGBoost Engine (Fallback): An extreme gradient boosting regressor that uses engineered lag features, thermal inertia smoothing, and a Brownian bridge to ensure \(C_0\) continuity at gap boundaries.

Dataset Output Structure

The Outputs/Imputed/ directory contains the fully reconstructed datasets in two frequencies (15min, 30min).

Primary Files

  • dataset_15min_france_load.csv: Drivers for consumption forecasting (temperature, nebulosity, calendar).

  • dataset_15min_france_wind.csv: Drivers for wind power (wind speed, air density, gusts).

  • dataset_15min_france_solar.csv: Drivers for photovoltaics (irradiance proxies, temperature).

  • dataset_15min_full.csv: The complete high-dimensional matrix (~400 columns).

Key Feature Glossary

Feature

Unit

Description

rte_load_france

MW

Reference record for national load (best of definitive/consolidated/real-time).

rte_load_france_definitive

MW

Strictly validated historical load (isolated for deviation/error analysis).

enedis_load_residential_total_france

MW

Reconstructed residential load combining telemetered (Linky) and profiled data.

meteo_temperature_celsius_france_load

°C

National temperature proxy, dynamically weighted by regional population density.

day_type_week_period_hour_changed

Cat

Composite calendar key encoding day of the week, holidays, and seasonality.

Project Structure

force/
├── src/force/
│   ├── collectors/       # API Connectors (State-Aware & Robust)
│   ├── transformation/   # Physics-Aware Smoothing & Multi-Version Pivoting
│   ├── check_*.py        # Quality Control & Auditing Modules
│   └── imputation/       # ML Imputation Engine (Predictive, Spatial, Audit)
├── documentation/        # Detailed Methodological Papers
├── data_warehouse/       # Local DuckDB Storage (Normalized Schemas)
└── Outputs/              # Final CSV Datasets

Dataset Access

Recognizing that reproducing the full ETL pipeline requires time, computational resources and reliable API access, we provide a static, versioned snapshot of the finalized dataset hosted on Zenodo.

Citation

If you use the FORCE package or the pre-compiled dataset in your research, please cite them using their permanent archives:

To cite the FORCE Software Pipeline:

@misc{force2026package,
  title={FORCE: French Open Research Catalogue of Energy (v1.1.1)},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  doi={10.5281/zenodo.21108937},
  note={With a contribution of Nathan Doumèche during his thesis}
}

To cite the FORCE Dataset Snapshot:

@dataset{force2026dataset,
  title={FORCE Dataset: French Open Research Catalogue of Energy - 2026 Snapshot},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.21109134},
  url={https://doi.org/10.5281/zenodo.21109134}
}