# Enedis Distribution Grid Pipeline

**[Back to README](../README.md)**

This document details the extraction and harmonization algorithms applied to data from Enedis, the French Distribution System Operator (DSO). The pipeline uses a flexible architecture designed to unify the heterogeneous metering standards of the French distribution grid. Specifically, it handles the historical transition from legacy 30-minute estimated telemetry to modern 15-minute exact telemetry driven by the national "Linky" smart meter rollout.

---

## Part 1: Collection & Ingestion Modules

This stage is split into two specialized components to handle the dual "push/pull" nature of Enedis data acquisition: an API collector for real-time incremental updates, and a bulk ingestion engine for high-throughput historical patching.

### Data Source

* **Provider:** Enedis (via OpenDataSoft).
* **Dataset:** *Bilan électrique au pas demi-heure* (Electrical balance sheet, half-hourly).
* **Protocol:**
    1.  **REST API v1:** For recent, ongoing data (JSON/CSV stream).
    2.  **Bulk CSV:** For historical dumps (flat files).
* **Granularity:** 30-minute intervals.
* **License:** Etalab Open License {cite:p}`Enedis:Bilan`.

### Software Architecture

#### A. API Collector (`collect_enedis.py`)
Designed for incremental updates, this module fetches the "delta" (new data) since the last execution.
* **Parallel extraction:** Uses multi-threading (`ThreadPoolExecutor`) to slice the required time horizon into smaller intervals, bypassing API timeout limits.
* **Idempotent persistence:** Uses duplicate-safe database transactions to ensure that if the pipeline is interrupted, re-running it will not create duplicate rows.

#### B. Bulk Ingestion Engine (`enedis_ingestion.py`)
Designed for deep historical backfilling from cold storage CSVs.
* **Patch and fill strategy:** Implements an update/insert (upsert) logic that:
    1.  **Patches** existing records where critical data (e.g., total load) is missing (0 or `NaN`) but is present in the CSV.
    2.  **Inserts** entirely new records to restore missing historical periods without overwriting newer, higher-quality API data.

---

## Part 2: Pre-Processing Module (`enedis_utils.py`)

Raw Enedis data is highly heterogeneous, with inconsistent nomenclature and localized formatting. This module standardizes the data before it enters the warehouse.

### 1. Schema Mapping
Maps French API fields to an English internal schema (e.g., `soutirage_rte` $\rightarrow$ `net_export_to_rte`). This standardizes naming conventions across the pipeline.

### 2. Numeric Formatting
Handles European decimal separators. It strips internal whitespace and converts commas to dots (e.g., `"1 000,50"` $\rightarrow$ `1000.50`) before casting to numeric types.

### 3. Unit Conversion
Converts raw physical flows from Watts (W) to Megawatts (MW) by applying a $10^{-6}$ scaling factor. State variables (like temperature or radiation proxies included in the Enedis feed) are explicitly excluded from this scaling.

### 4. Timezone Standardization
Forces all parsed timestamps into UTC-aware `datetime64` objects to prevent Dayight Saving Time (DST) collisions during later assembly.

---

## Part 3: Transformation Module (`transform_enedis.py`)

This module aligns the distribution data with the 15-minute pipeline standard and reconstructs total load profiles.

### 1. Temporal Unification
Historically, Enedis data was sampled at 30-minute intervals. 
* **Conservative upsampling:** The pipeline standardizes the legacy timeline to a high-frequency 15-minute resolution using a conservative smoothing kernel. This mathematical kernel ensures that the total energy integral (MWh) over the 30-minute block is strictly preserved. Native historical 30-minute values are kept as-is rather than round-tripped through this kernel.
* **Gap filling:** Meteo columns use linear interpolation capped at 8 steps (2 hours). Load and production columns use a profile-guided fill referencing the same weekday one week earlier (D-7), also capped at 8 steps.
* **Quality gate:** Data before `2020-01-01` is nulled out for columns judged unreliable prior to that date.

### 2. Profile Reconstruction
The total power consumption for a given sector (residential, professional, enterprise) is split in the raw data into two streams:
1.  **Telemetered data (Linky):** Actual measurements from smart meters.
2.  **Profiled data:** Statistically estimated load for customers using legacy meters.

As the Linky rollout progressed between 2016 and 2021, the "Telemetered" volume grew while the "Profiled" volume shrank. To provide a stationary feature for machine learning models, the transformer reconstructs the true sector load by summing both streams:
> $Load_{Residential\_Total} = Load_{Residential\_Telemetered} + Load_{Residential\_Profiled}$

---

## Final Feature Schema

The transformed database (`transformed_enedis_15min` and `_30min`) produces the following structure:

| Category | Column Name | Description |
| :--- | :--- | :--- |
| **Index** | `date` | **Primary Key** (UTC timestamp). |
| **Network Balance** | `enedis_net_export_to_rte_france` | Surplus injected from distribution to the transmission grid (MW). |
| | `enedis_net_import_from_rte_france` | Power drawn from transmission to distribution (MW). |
| | `enedis_net_export_to_dso_france` | Surplus injected between distribution operators (MW). |
| | `enedis_grid_losses_france` | Technical grid losses (MW). |
| **Total Flows** | `enedis_load_total_france` | Total consumption on the distribution grid (MW). |
| | `enedis_production_total_france` | Total decentralized generation (MW). |
| **Decentralized Generation** | `enedis_production_solar_total_france` | Total PV production (MW). |
| | `enedis_production_solar_profiled_france` | **[Raw]** Solar estimated data (MW). |
| | `enedis_production_wind_total_france` | Total wind production (MW). |
| | `enedis_production_hydro_total_france` | Total hydro production (MW). |
| | `enedis_production_bioenergy_total_france` | Biomass and cogeneration (MW). |
| | `enedis_production_bioenergy_profiled_france` | **[Raw]** Bioenergy estimated data (MW). |
| | `enedis_production_other_total_france` | Waste and thermal generation (MW). |
| | `enedis_production_other_profiled_france` | **[Raw]** Other production estimated data (MW). |
| **Detailed Load** | `enedis_load_industrial_total_france` | High Voltage A (industrial) load (MW). |
| | `enedis_load_industrial_telemetered_france` | **[Raw]** Industrial Linky data (MW). |
| | `enedis_load_industrial_profiled_france` | **[Raw]** Industrial estimated data (MW). |
| | `enedis_load_residential_total_france` | **[Computed]** Res. telemetered + profiled (MW). |
| | `enedis_load_professional_total_france` | **[Computed]** Pro. telemetered + profiled (MW). |
| | `enedis_load_entreprise_total_france` | **[Computed]** Ent. telemetered + profiled (MW). |
| | `enedis_load_residential_telemetered_france` | **[Raw]** Residential Linky data (MW). |
| | `enedis_load_residential_profiled_france` | **[Raw]** Residential estimated data (MW). |
| | `enedis_load_professional_telemetered_france` | **[Raw]** Professional Linky data (MW). |
| | `enedis_load_professional_profiled_france` | **[Raw]** Professional estimated data (MW). |
| | `enedis_load_entreprise_telemetered_france` | **[Raw]** Enterprise Linky data (MW). |
| | `enedis_load_entreprise_profiled_france` | **[Raw]** Enterprise estimated data (MW). |
| **Meteo Proxies** | `enedis_meteo_temperature_real_c_france` | Enedis' internal population-weighted temperature proxy (°C). |
| | `enedis_meteo_temperature_ref_c_france` | Enedis' reference temperature proxy (°C). |
| | `enedis_meteo_radiation_pseudo_france` | Enedis' internal solar irradiance proxy (Index). |
| **Calendar** | `enedis_month_france` | Month of year, as provided by Enedis (1-12). |