Data Hierarchies & Algebraic Relationships¶

Back to README

This document catalogues the structural relationships between columns within the assembled datasets. It verifies the physical energy balances on each grid, outlines which spatial and categorical aggregations are mathematically exact versus approximate, details family totals that contain unlisted sub-streams, and explains the synthesis of the RTE golden record.

These relationships were validated against the complete dataset via a streaming pass over the non-imputed dataset_30min_full.csv (253,630 rows spanning 2012-01-01 to 2026). For each relationship, we report the number of rows where all operands are present (\(n\)), the mean absolute error, and the relative error.

Note on Diagnostics: The script scripts/audit/gen_hierarchies.py renders these physical checks into diagnostic plots located in Outputs/audit/comparison/: hierarchy_balances.png (grid balances), hierarchy_regional.png (national vs. regional sums), and hierarchy_aggregations.png (load/production decompositions). See scripts/audit/README.md §9 for generation details.

1. Energy Balances: Transport vs. Distribution¶

Each grid is a physically conserved sub-system. Validating that inflows match outflows provides the strongest test of data integrity and sign conventions.

1a. Distribution Grid (Enedis): Closes Exactly¶

All six terms in this equation are raw published fields (consommation_totale, production_totale, pertes, injection_rte, soutirage_rte, soutirage_vers_autres_grd) [Enedis, 2026]. Because no terms are synthesized by the pipeline, this serves as a pure validation of physical integrity.

\[\underbrace{\text{net\_import\_from\_rte} + \text{production\_total}}_{\text{inflow}} = \underbrace{\text{load\_total} + \text{grid\_losses} + \text{net\_export\_to\_rte} + \text{net\_export\_to\_dso}}_{\text{outflow}}\]

Balance	\(n\)	Within 1 MW	Mean Abs. Err.	Max Abs. Err.
Inflow = Outflow	87,647	100.00%	0.0017 MW	131 MW¹

¹ The only departures >1 MW occur precisely on the two Daylight Saving Time switch days per year due to standard timestamp folding/skipping.

This exact closure validates the pipeline’s Enedis sign convention: it balances perfectly when net_import_from_rte is treated as the bulk supply drawn from the transmission grid (~35 GW inflow) and net_export_to_rte is treated as local surplus injected back (~3 GW outflow).

1b. Transmission Grid (RTE): Closes Approximately¶

The balance on the RTE transmission grid, utilizing the eco2mix platform data [RTE, 2026], is defined as:

\[\text{load} = \underbrace{(\text{nuclear} + \text{gas} + \text{coal} + \text{fuel} + \text{hydro} + \text{wind} + \text{solar} + \text{bioenergy})}_{\text{generation}} + \text{exchange\_net} + \text{pump}\]

Here, exchange_net > 0 represents net cross-border imports, and pump < 0 represents pumped-storage consumption (averaging ≈ −800 MW). This equation closes to within ~27 MW (≈ 0.05% of the total load). The minor residual is accounted for by grid-scale batteries, rounding differences at the source, and data-maturity lag on recently ingested real-time rows.

Balance	\(n\)	Mean Abs. Err.	Rel. Error
Load = Generation + Exchange + Pump	253,525	28.5 MW	0.054%

2. Load & Production Decompositions¶

2a. Enedis Load: National = Σ 4 Segment Totals (Exact)¶

The published national load matches the sum of its four customer segment totals exactly. This confirms internal consistency, as load_total and load_industrial_total are raw published fields, while the residential, professional, and enterprise segment totals are reconstructed by the pipeline (combining telemetered and profiled data).

\[\text{enedis\_load\_total\_france} = \sum_{\text{seg}} \text{enedis\_load\_[seg]\_total\_france}\]

(Where seg \(\in\) {residential, professional, entreprise, industrial})

Relationship	\(n\)	Exact Match %	Mean Abs. Err.	Rel. Error
National load = Σ 4 segment totals	112,703	100.00%	0.0000 MW	0.00000%

2b. Enedis Production: Total ≈ Σ 5 Streams (Approximate)¶

The total distribution generation tracks the sum of specific energy vectors to within ~0.05% (mean residual of 3.7 MW). This is an approximation rather than an identity because Enedis publishes the total aggregate independently from the technological breakdown.

\[\text{enedis\_production\_total\_france} \approx \text{solar} + \text{wind} + \text{hydro} + \text{bioenergy} + \text{other}\]

Relationship	\(n\)	Mean Abs. Err.	Rel. Error
Production total ≈ Σ 5 streams	112,703	3.73 MW	0.0457%

3. RTE National vs. the 12 Metropolitan Regions (Approximate)¶

National time series are published independently of regional streams. The national aggregates track the sum of the 12 continental metropolitan regions very closely (within 0.01% to 0.4%) but do not close exactly. This deviation occurs because:

Non-interconnected zones (like Corsica and overseas territories) are included in national totals but not the 12 regions.
Cross-border commercial exchanges (exchange_net) are primarily booked at the national level.
Minor data-maturity lag exists on recent real-time rows.

\[\text{rte\_[fam]\_france} \approx \sum_{r \in 12\ \text{regions}} \text{rte\_[fam]\_}r\]

Family	\(n\)	Mean Abs. Err.	Rel. Error
Load	235,957	4.61 MW	0.0087%
Nuclear²	95,706	4.97 MW	0.0127%
Wind	235,907	2.23 MW	0.0586%
Solar	236,007	1.89 MW	0.1132%
Hydro	235,960	2.74 MW	0.0382%
Bioenergy	235,960	3.79 MW	0.3566%
Pump²	95,706	1.90 MW	n/a³
Exchange_net	236,007	28.18 MW	0.3966%

² nuclear and pump regional columns exist for fewer historical rows (≈ 95.7k), capping the comparison count. ³ pump naturally hovers near 0 MW, making relative error mathematically undefined; the ~1.9 MW absolute gap is the relevant stability metric.

4. RTE Family Totals with Unlisted Sub-streams (Partial)¶

For four specific generation families, the dataset provides columns for major sub-streams, but these sub-streams do not equal the family total. The difference consists of unlisted sub-streams that are aggregated at the source but not explicitly exported as separate columns in the pipeline.

Family	Exposed Sub-streams	Exposed Coverage	Unlisted Remainder
`rte_hydro_france`	`hydro_river` + `hydro_lake`	~90%	STEP turbining + small hydro (~690 MW)
`rte_gas_france`	`gas_ccg` + `gas_tac`	~61%	Cogeneration + other (~1260 MW)
`rte_bioenergy_france`	`bio_biogas`	~26%	Biomass + waste + other (~790 MW)
`rte_wind_france`	`wind_onshore`	~87%	`wind_offshore` (~740 MW)

Notes on Specific Vectors:

Wind: While wind = onshore + offshore is an exact identity at the raw source, wind_offshore is intentionally omitted from the final CSV exports to reduce dimensionality. Consequently, wind_onshore covers only ~87% of the total wind generation.
STEP (Station de Transfert d’Énergie par Pompage): This refers to pumped-storage hydroelectricity. The energy generated when water is released (turbining) lives inside rte_hydro_france. Conversely, the energy consumed to pump water upstream is tracked separately in rte_pump_france as a negative load.

5. RTE Golden Record (Maturity Coalesce)¶

The synthesized reference variable rte_load_france (and other base RTE series) is not an arithmetic average of its three maturity versions. Instead, it utilizes a priority coalesce operation (a sequential fallback) that selects the highest maturity value available for any given timestamp:

\[\text{rte\_load\_france} = \mathrm{COALESCE}(\text{\_definitive}, \text{\_consolidated}, \text{\_realtime})\]

Source Nature	Share of Rows
Definitive	89.91%
Consolidated	7.48%
Real-Time	2.61%
Missing (No Match)	0.00% (2 rows)

Summary of Relationships¶

Category	Description
Exact Identity	Enedis distribution balance (§1a); Enedis load decomposition (§2a).
Closes (Approximate)	RTE transmission balance (§1b); RTE national ≈ Σ 12 regions (§3).
Tracks (Approximate)	Enedis production ≈ sum of 5 technological streams (§2b).
Partial	RTE hydro, gas, bioenergy, and wind (contain unlisted remainders) (§4).
Coalesce	RTE golden record prioritization by data maturity (§5).

Back to README