# FORCE: French Open Research Catalogue of Energy - A High-Performance Python Pipeline
  
**Version:** 1.1.1  
**License:** LGPL-3.0-or-later.  
**Copyright (c):** 2020-2026 EDF (Electricité De France).   
**Copyright (c):** 2023 Sorbonne Université.  
  
**Active contributor:** Yann Allioux (2020-2026)  
**Past contributor:**  Nathan Doumèche (2023-2025)  
**Scientific advisor:** Yannig Goude  

[![PyPI Version](https://img.shields.io/pypi/v/force-dataset.svg)](https://pypi.org/project/force-dataset/)
[![CI](https://github.com/EDF-Lab/force/actions/workflows/ci.yml/badge.svg)](https://github.com/EDF-Lab/force/actions)
[![Docs](https://github.com/EDF-Lab/force/actions/workflows/docs.yml/badge.svg)](https://edf-lab.github.io/force/)
[![License: LGPL v3](https://img.shields.io/badge/License-LGPL_v3-blue.svg)](https://opensource.org/licenses/LGPL-3.0)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![GitHub Repo stars](https://img.shields.io/github/stars/EDF-Lab/force?style=social)](https://github.com/EDF-Lab/force)
[![DOI: Software](https://zenodo.org/badge/DOI/10.5281/zenodo.21108937.svg)](https://doi.org/10.5281/zenodo.21108937)
[![DOI: Dataset](https://zenodo.org/badge/DOI/10.5281/zenodo.21109134.svg)](https://doi.org/10.5281/zenodo.21109134)

## Abstract

FORCE provides an extract, load, and transform (ETL) pipeline to aggregate, harmonize, and validate multi-source energy data for the French power system. It addresses the challenge of heterogeneous data by unifying divergent temporal resolutions (e.g., 15-minute vs. 30-minute), spatial granularities, and physical units into a continuous time series matrix.

The package is built for researchers working on time series forecasting, grid economics, and climate-energy modeling. Relying on DuckDB as its analytical backend {cite:p}`raasveldt2019duckdb`, FORCE consolidates data from four primary sources:

1. **RTE (Transmission):** A multi-layer architecture tracking data maturity (real-time, consolidated, definitive) to synthesize a single reference record {cite:p}`RTE:Eco2Mix`.
2. **Enedis (Distribution):** Granular consumption profiles and decentralized generation metrics {cite:p}`Enedis:Bilan`.
3. **Météo-France (Weather):** Synoptic observations transformed via asset-weighted spatial aggregation {cite:p}`MeteoFrance:Synop`.
4. **ODRÉ (Assets):** A georeferenced registry of power plants used to generate spatial weights {cite:p}`ODRE:Registre`.

---

## Documentation & Scientific Methodology

Each component of the pipeline is documented with full algorithmic transparency:

| Component | Documentation Link | Key Algorithms |
| --- | --- | --- |
| **Calendar** | [CALENDAR.md](documentation/CALENDAR.md) | Skeleton generation, UTC-DST bridge, topological day typing {cite:p}`Etalab:Feries,EducationNationale:Calendrier,Augusti:Vacances`. |
| **RTE** | [RTE.md](documentation/RTE.md) | Multi-version pivoting, conservative upsampling, maturity-aware deduplication. |
| **Enedis** | [ENEDIS.md](documentation/ENEDIS.md) | Patch and fill strategy, profile reconstruction. |
| **Météo** | [METEO.md](documentation/METEO.md) | Multi-barycenter aggregation, vector decomposition (for circular wind variables). |
| **Radiation** | [RADIATION.md](documentation/RADIATION.md) | PVGIS POA (Plane of Array) & ERA5 GHI (Global Horizontal Irradiance) collection. |
| **Assets** | [ODRE.md](documentation/ODRE.md) | Two-stage geocoding {cite:p}`OpenDataSoft:Georef`, spatial centroids. |
| **Imputation** | [IMPUTATION.md](documentation/IMPUTATION.md) | Pluggable engines, recursive feature relaxation, spatial kriging (interpolation), 7-step causal DAG (Directed Acyclic Graph). |
| **Hierarchies** | [HIERARCHIES.md](documentation/HIERARCHIES.md) | Verified column sum-trees, energy balance, calendar encodings. |
| **Audit** | [audit/README.md](scripts/audit/README.md) | Per-dataset visualization suite, notebook-engine reproduction. |
| **TAM Extraction** | [TAM.md](documentation/TAM.md) | Autoregressive lag/thermal-inertia feature engineering, legacy dataset alignment. |

---

## Installation & Usage

### 1. Installation

FORCE requires Python 3.10 or higher (Python 3.12+ recommended for the full TS-ICL imputation engine).

**As a package** (for end users):

```bash
pip install force-dataset

```

**From source** (for development or running the pipeline). All dependencies are managed via `pyproject.toml`.

```bash
python -m venv .venv
# activate: PowerShell: .\.venv\Scripts\Activate.ps1 | bash/zsh: source .venv/bin/activate
pip install -e ".[dev]"

```

Optional extras include `dev` (tests/lint), `docs` (Sphinx), `dashboard` (Jupyter), `climate` (CDS/CMIP6), and `icl` (TS-ICL imputation engine, requiring Python >= 3.12).

### 2. Execution Pipeline

Execute the pipeline steps sequentially to build the dataset:

**Phase A: Collection** Fetches raw data from APIs. The state-aware engine detects data maturation (e.g., when real-time data is retroactively validated).

```bash
energy-collect

```

**Phase B: Transformation** Applies cleaning, upsampling, and technical repair. Structural gaps are strictly preserved as `NaN`.

```bash
energy-transform

```

**Phase C: Database Audit** Scans DuckDB files for temporal coverage gaps, empty tables, and schema anomalies. For diagnostic visualizations, use `energy-viz-transform` and `energy-viz-assemble`.

```bash
energy-check

```

**Phase D: Assembly & Imputation** Joins sources and applies physics-informed imputation.

```bash
energy-assemble
energy-impute  # Uses TS-ICL by default; pass '--engine xgboost' for gradient boosting

```

---

## Scientific Method: Data Harmonization

The pipeline addresses specific challenges in energy data engineering:

### 1. Temporal Resolution Harmonization (30min to 15min)

The pipeline standardizes all data streams to a 15-minute resolution:

* **Energy Flows (Grid Data):** We apply a conservative smoothing kernel that mathematically preserves the energy integral (MWh) over the block. *Note: Minor numerical deviations may occur when enforcing physical bounds (e.g., clipping negative generation).* Formula: $y_{t+15} = \frac{3Y_t + Y_{t+30}}{4}$.
* **State Variables (Meteo Data):** Instantaneous observations (e.g., temperature) use strict linear upsampling ($limit=1$) to calculate the intermediate point ($t+15$) without modifying the original anchors.

### 2. Spatio-Temporal Heterogeneity

A simple geographic average of weather stations introduces bias because energy assets are unevenly distributed.

* **Spatial Solution:** We calculate usage-specific centroids ($G_{load}$, $G_{wind}$, $G_{solar}$) for every region. Station data is aggregated using a modified inverse distance weighting (IDW) algorithm. To ensure numerical stability, this IDW applies a 1 km softening factor and assumes an oblate spheroid Earth ($R=6371$ km).
* **Temporal Solution:** The pipeline computes time-varying weights ($W_{r,y}$). The influence of a region $r$ on the national signal evolves annually $y$ based on its actual market share. Formula: $I_{national}(t) = \sum_{r} W_{r, y(t)} \cdot I_{r}(t)$

### 3. Data Maturity & Versioning

Energy data evolves from real-time estimates to validated definitive records. For critical metrics like national load, the pipeline produces multiple time series versions:

* `_realtime`: For simulating production forecasting.
* `_definitive`: For training on ground truth.
* `_best`: A synthesized golden record for general use.

### 4. Two-Stage Gap Handling Strategy

We separate technical repair from statistical reconstruction to guarantee data integrity:

#### Stage 1: Technical Repair (ETL Phase)

Targets micro-gaps (< 1h) caused by sensor dropouts.

* **Grid Data:** Cyclical variables are repaired via a profile-guided bridge ($D-7$) to preserve intra-hour peaks. Stochastic variables (wind/solar) use linear interpolation.
* **Meteo Data:** Utilizes lapse-rate aware k-Nearest Neighbors (k-NN) to prioritize vertical similarity, and Time-of-Day (TOD) fallbacks to bridge gaps when spatial neighbors are unavailable.

#### Stage 2: Statistical Reconstruction (Imputation Phase)

Targets structural outages and historical backfilling for gaps > 1h.

* **TS-ICL Engine (Default):** A zero-shot time-series foundation model. It handles raw temporal context directly and ignores the `training_cutoff_years` parameter.
* **XGBoost Engine (Fallback):** An extreme gradient boosting regressor that uses engineered lag features, thermal inertia smoothing, and a Brownian bridge to ensure $C_0$ continuity at gap boundaries.

---

## Dataset Output Structure

The `Outputs/Imputed/` directory contains the fully reconstructed datasets in two frequencies (**15min**, **30min**).

### Primary Files

* `dataset_15min_france_load.csv`: Drivers for consumption forecasting (temperature, nebulosity, calendar).
* `dataset_15min_france_wind.csv`: Drivers for wind power (wind speed, air density, gusts).
* `dataset_15min_france_solar.csv`: Drivers for photovoltaics (irradiance proxies, temperature).
* `dataset_15min_full.csv`: The complete high-dimensional matrix (~400 columns).

### Key Feature Glossary

| Feature | Unit | Description |
| --- | --- | --- |
| `rte_load_france` | MW | Reference record for national load (best of definitive/consolidated/real-time). |
| `rte_load_france_definitive` | MW | Strictly validated historical load (isolated for deviation/error analysis). |
| `enedis_load_residential_total_france` | MW | Reconstructed residential load combining telemetered (Linky) and profiled data. |
| `meteo_temperature_celsius_france_load` | °C | National temperature proxy, dynamically weighted by regional population density. |
| `day_type_week_period_hour_changed` | Cat | Composite calendar key encoding day of the week, holidays, and seasonality. |

---

## Project Structure

```text
force/
├── src/force/
│   ├── collectors/       # API Connectors (State-Aware & Robust)
│   ├── transformation/   # Physics-Aware Smoothing & Multi-Version Pivoting
│   ├── check_*.py        # Quality Control & Auditing Modules
│   └── imputation/       # ML Imputation Engine (Predictive, Spatial, Audit)
├── documentation/        # Detailed Methodological Papers
├── data_warehouse/       # Local DuckDB Storage (Normalized Schemas)
└── Outputs/              # Final CSV Datasets

```

---

## Dataset Access

Recognizing that reproducing the full ETL pipeline requires time, computational resources and reliable API access, we provide a static, versioned snapshot of the finalized dataset hosted on Zenodo. 
* **Download the 2026 Dataset Snapshot:** [https://doi.org/10.5281/zenodo.21109134](https://doi.org/10.5281/zenodo.21109134)

---

## Citation

If you use the FORCE package or the pre-compiled dataset in your research, please cite them using their permanent archives:

**To cite the FORCE Software Pipeline:**
```bibtex
@misc{force2026package,
  title={FORCE: French Open Research Catalogue of Energy (v1.1.1)},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  doi={10.5281/zenodo.21108937},
  note={With a contribution of Nathan Doumèche during his thesis}
}
```

**To cite the FORCE Dataset Snapshot:**

```bibtex
@dataset{force2026dataset,
  title={FORCE Dataset: French Open Research Catalogue of Energy - 2026 Snapshot},
  author={Allioux, Yann and Goude, Yannig},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.21109134},
  url={https://doi.org/10.5281/zenodo.21109134}
}
```

---

## License & Copyright

**FORCE: French Open Research Catalogue of Energy** is released under the **LGPL-3.0-or-later** license. 

* **2025-2026 Advanced imputation engine, pipeline architecture, Enedis & radiation features, DuckDB Ecosystem:** Copyright © EDF, author Yann Allioux.
* **2023 Imputation engine & scientific publication:** Copyright © EDF & Sorbonne Université, authors Yann Allioux, Nathan Doumèche.
* **2020-2022 Core framework & baselines:** Copyright © EDF, author Yann Allioux.

*Data Attribution & Licenses:*
* *Grid and synoptic weather data provided by **RTE**, **Enedis**, **Météo-France**, and **ODRÉ** under the Etalab Open License 2.0.*
* *Solar radiation baseline data provided by the European Commission **PVGIS** (SARAH-3).*
* *Reanalysis and climate projection data generated using Copernicus Climate Change Service information (**ERA5**, **CMIP6**) and subject to the Copernicus License. Users of this pipeline must register for a Copernicus account and accept their Terms of Use.*