Enedis Distribution Grid Pipeline¶

Back to README

This document details the extraction and harmonization algorithms applied to data from Enedis, the French Distribution System Operator (DSO). The pipeline uses a flexible architecture designed to unify the heterogeneous metering standards of the French distribution grid. Specifically, it handles the historical transition from legacy 30-minute estimated telemetry to modern 15-minute exact telemetry driven by the national “Linky” smart meter rollout.

Part 1: Collection & Ingestion Modules¶

This stage is split into two specialized components to handle the dual “push/pull” nature of Enedis data acquisition: an API collector for real-time incremental updates, and a bulk ingestion engine for high-throughput historical patching.

Data Source¶

Provider: Enedis (via OpenDataSoft).
Dataset: Bilan électrique au pas demi-heure (Electrical balance sheet, half-hourly).
Protocol:
1. REST API v1: For recent, ongoing data (JSON/CSV stream).
2. Bulk CSV: For historical dumps (flat files).
Granularity: 30-minute intervals.
License: Etalab Open License [Enedis, 2026].

Software Architecture¶

A. API Collector (`collect_enedis.py`)¶

Designed for incremental updates, this module fetches the “delta” (new data) since the last execution.

Parallel extraction: Uses multi-threading (ThreadPoolExecutor) to slice the required time horizon into smaller intervals, bypassing API timeout limits.
Idempotent persistence: Uses duplicate-safe database transactions to ensure that if the pipeline is interrupted, re-running it will not create duplicate rows.

B. Bulk Ingestion Engine (`enedis_ingestion.py`)¶

Designed for deep historical backfilling from cold storage CSVs.

Patch and fill strategy: Implements an update/insert (upsert) logic that:
1. Patches existing records where critical data (e.g., total load) is missing (0 or NaN) but is present in the CSV.
2. Inserts entirely new records to restore missing historical periods without overwriting newer, higher-quality API data.

Part 2: Pre-Processing Module (`enedis_utils.py`)¶

Raw Enedis data is highly heterogeneous, with inconsistent nomenclature and localized formatting. This module standardizes the data before it enters the warehouse.

1. Schema Mapping¶

Maps French API fields to an English internal schema (e.g., soutirage_rte \(\rightarrow\) net_export_to_rte). This standardizes naming conventions across the pipeline.

2. Numeric Formatting¶

Handles European decimal separators. It strips internal whitespace and converts commas to dots (e.g., "1 000,50" \(\rightarrow\) 1000.50) before casting to numeric types.

3. Unit Conversion¶

Converts raw physical flows from Watts (W) to Megawatts (MW) by applying a \(10^{-6}\) scaling factor. State variables (like temperature or radiation proxies included in the Enedis feed) are explicitly excluded from this scaling.

4. Timezone Standardization¶

Forces all parsed timestamps into UTC-aware datetime64 objects to prevent Dayight Saving Time (DST) collisions during later assembly.

Part 3: Transformation Module (`transform_enedis.py`)¶

This module aligns the distribution data with the 15-minute pipeline standard and reconstructs total load profiles.

1. Temporal Unification¶

Historically, Enedis data was sampled at 30-minute intervals.

Conservative upsampling: The pipeline standardizes the legacy timeline to a high-frequency 15-minute resolution using a conservative smoothing kernel. This mathematical kernel ensures that the total energy integral (MWh) over the 30-minute block is strictly preserved. Native historical 30-minute values are kept as-is rather than round-tripped through this kernel.
Gap filling: Meteo columns use linear interpolation capped at 8 steps (2 hours). Load and production columns use a profile-guided fill referencing the same weekday one week earlier (D-7), also capped at 8 steps.
Quality gate: Data before 2020-01-01 is nulled out for columns judged unreliable prior to that date.

2. Profile Reconstruction¶

The total power consumption for a given sector (residential, professional, enterprise) is split in the raw data into two streams:

Telemetered data (Linky): Actual measurements from smart meters.
Profiled data: Statistically estimated load for customers using legacy meters.

As the Linky rollout progressed between 2016 and 2021, the “Telemetered” volume grew while the “Profiled” volume shrank. To provide a stationary feature for machine learning models, the transformer reconstructs the true sector load by summing both streams:

\(Load_{Residential\_Total} = Load_{Residential\_Telemetered} + Load_{Residential\_Profiled}\)

Final Feature Schema¶

The transformed database (transformed_enedis_15min and _30min) produces the following structure:

Category	Column Name	Description
Index	`date`	Primary Key (UTC timestamp).
Network Balance	`enedis_net_export_to_rte_france`	Surplus injected from distribution to the transmission grid (MW).
	`enedis_net_import_from_rte_france`	Power drawn from transmission to distribution (MW).
	`enedis_net_export_to_dso_france`	Surplus injected between distribution operators (MW).
	`enedis_grid_losses_france`	Technical grid losses (MW).
Total Flows	`enedis_load_total_france`	Total consumption on the distribution grid (MW).
	`enedis_production_total_france`	Total decentralized generation (MW).
Decentralized Generation	`enedis_production_solar_total_france`	Total PV production (MW).
	`enedis_production_solar_profiled_france`	[Raw] Solar estimated data (MW).
	`enedis_production_wind_total_france`	Total wind production (MW).
	`enedis_production_hydro_total_france`	Total hydro production (MW).
	`enedis_production_bioenergy_total_france`	Biomass and cogeneration (MW).
	`enedis_production_bioenergy_profiled_france`	[Raw] Bioenergy estimated data (MW).
	`enedis_production_other_total_france`	Waste and thermal generation (MW).
	`enedis_production_other_profiled_france`	[Raw] Other production estimated data (MW).
Detailed Load	`enedis_load_industrial_total_france`	High Voltage A (industrial) load (MW).
	`enedis_load_industrial_telemetered_france`	[Raw] Industrial Linky data (MW).
	`enedis_load_industrial_profiled_france`	[Raw] Industrial estimated data (MW).
	`enedis_load_residential_total_france`	[Computed] Res. telemetered + profiled (MW).
	`enedis_load_professional_total_france`	[Computed] Pro. telemetered + profiled (MW).
	`enedis_load_entreprise_total_france`	[Computed] Ent. telemetered + profiled (MW).
	`enedis_load_residential_telemetered_france`	[Raw] Residential Linky data (MW).
	`enedis_load_residential_profiled_france`	[Raw] Residential estimated data (MW).
	`enedis_load_professional_telemetered_france`	[Raw] Professional Linky data (MW).
	`enedis_load_professional_profiled_france`	[Raw] Professional estimated data (MW).
	`enedis_load_entreprise_telemetered_france`	[Raw] Enterprise Linky data (MW).
	`enedis_load_entreprise_profiled_france`	[Raw] Enterprise estimated data (MW).
Meteo Proxies	`enedis_meteo_temperature_real_c_france`	Enedis’ internal population-weighted temperature proxy (°C).
	`enedis_meteo_temperature_ref_c_france`	Enedis’ reference temperature proxy (°C).
	`enedis_meteo_radiation_pseudo_france`	Enedis’ internal solar irradiance proxy (Index).
Calendar	`enedis_month_france`	Month of year, as provided by Enedis (1-12).