FORCE: Audit & Visualization Scripts

Copyright (c) 2025-2026 EDF (Electricité De France). Author: Yann Allioux

This folder contains standalone, reproducible scripts that render the project’s scientific audit figures from the assembled dataset. Each script is independent and can be run on its own; together they form a visual QA suite for the pipeline output.

How to run

All scripts read the assembled dataset Outputs/dataset_30min_full.csv (produced by energy-assemble) and require a headless matplotlib backend on Windows:

# from the repository root, with the project environment active
set MPLBACKEND=Agg # Windows (PowerShell: $env:MPLBACKEND="Agg")
python scripts/audit/gen_availability_heatmap.py

Outputs land either in Outputs/audit/ (diagnostic figures) or paper/images/ (figures referenced by the JOSS paper), as noted per script below.

1. Data completeness & coverage

gen_availability_heatmap.py to Outputs/audit/availability_heatmap.png

Weekly fill-rate (fraction of non-null timestamps) for every numeric column, grouped by source (Enedis / Météo / RTE / Calendar) and rendered as a red to green heatmap. Also exports Outputs/audit/availability_stats.csv (per-column global and in-window availability). Look for: the Enedis block turning green only from ~2020 (smart-meter rollout); the post-2023 gap in the radiation rows (PVGIS coverage ends 2023); any unexpected white (missing) stripes in RTE/Météo.

gen_scope_timeline.py to paper/images/scope_timeline.png

Horizontal Gantt of each data stream’s active period, annotated with its fill rate and grouped by provider (TSO / DSO / Météo-France / ODRÉ / Etalab). Look for: the 2012 start of RTE vs the 2020 onset of Enedis DSO observability.

2. Cross-source validation

gen_validation_stacked.py to paper/images/validation_stacked.png

Two panels: (A) the multi-barycenter effect, solar-weighted vs population-weighted national temperature, whose daytime divergence confirms PV assets sit in climatically distinct areas; (B) TSO vs DSO load harmonization, RTE national load vs Enedis distribution load, the gap representing transmission-connected industry + grid losses. Look for: tight temporal alignment of the two load curves; a consistent (not erratic) TSO-DSO gap.

3. Spatial decomposition

gen_spatial_breakdown.py to paper/images/spatial_breakdown_{wind,solar}.png

Stacked area of regional generation (wind and solar) summing to the national signal (dashed line), illustrating the portfolio effect. Look for: colored bands summing to the national line (bottom-up consistency); the north/west concentration for wind vs the southern dominance for solar.

gen_radiation_regional.py to Outputs/audit/radiation_regional.png (new)

Monthly PVGIS radiation climatology for each region (thin lines) against the national solar-weighted reference (bold). Applies the notebook’s “find national reference” idea to the new radiation feature. Look for: southern regions (Occitanie, PACA) sitting above the national curve; all regions sharing the summer-peaked seasonal shape.

4. Data maturity & observability

gen_maturity_forensics.py to paper/images/maturity_forensics.png

The Golden Record lifecycle for national load: (A) the full timeline coloured by quality level (Definitive to Consolidated to Real-Time); (B) a zoom on the Definitive to Consolidated handoff showing the inter-stream delta. Look for: no discontinuity in the synthesized signal at the maturity cutoffs.

gen_enedis_observability.py to paper/images/enedis_observability_*.png

Smart-meter rollout dashboards (composite + per sector: residential / professional / entreprise): (A) telemetered vs profiled load with a rising telemetry-rate line; (B) the average diurnal cycle. Look for: the green (“telemetered”) area expanding over time; the residential nocturnal water-heating / EV peak absent from the generalized profile.

5. Physical & driver relationships (new)

gen_generation_mix.py to Outputs/audit/generation_mix.png (new)

Stacked area of the full RTE generation mix (nuclear, hydro, gas, coal, bioenergy, wind, solar) against the national load (dashed) for a representative winter week. Look for: nuclear baseload, the daily solar bumps, and dispatchable gas tracking the load peaks.

gen_thermosensitivity.py to Outputs/audit/thermosensitivity.png (new)

Scatter of national load vs population-weighted temperature, coloured by month, the classic thermosensitivity relationship. Look for: the “hockey-stick”, load rising steeply below ~15 °C (electric heating) with a milder summer cooling uptick; winter months (cold colours) on the high-load arm.

gen_solar_clock.py to Outputs/audit/solar_clock.png (new)

Heatmap of mean solar generation across hour-of-day × month, the solar production envelope. Look for: the bright midday core widening from winter to summer (longer days, higher sun); strict zeros at night confirming the physical night constraint.

6. Per-dataset sweep: gen_all_audits.py (new)

Runs a generic, column-gated audit over every CSV dataset, both the assembled ones in Outputs/ and the imputed ones in Outputs/Imputed/, and writes one folder of figures per dataset:

Outputs/audit/<dataset_name>/
    missingness.png            # % missing per column (bar)
    availability_heatmap.png   # weekly fill rate, all numeric columns
    national_overview.png      # daily-mean time series of key national variables
    correlation.png            # correlation matrix of key national variables
    distributions.png          # sampled value histograms
    generation_mix.png         # only if RTE generation columns are present
    thermosensitivity.png      # only if load + national temperature are present
    solar_clock.png            # only if rte_solar_france is present

Each figure is emitted only when the dataset actually contains the required columns, so the rich *_full datasets get the complete set while the thematic shards (*_france_load, *_france_solar, *_france_wind) get the applicable subset.

Because it runs over both Outputs/ and Outputs/Imputed/, you get non-imputed vs imputed versions side by side, most informative on missingness.png and availability_heatmap.png, where the imputed folders should show the structural gaps filled (except the by-design Enedis pre-2020 and post-2023 PVGIS-radiation windows).

Run (generates all per-dataset folders in one pass):

python scripts/audit/gen_all_audits.py

Reads are column-selective via DuckDB, so even the multi-GB *_full datasets are handled by streaming aggregation rather than full in-memory loads.

7. Faithful notebook reproduction: gen_notebook_audit.py (new)

This is a direct port of audit.ipynb’s interactive engine (plot_analysis + generate_hybrid_config), turned into a batch generator that saves every figure the notebook produces for every CSV dataset (assembled + imputed). Unlike gen_all_audits.py (a small fixed set), this reproduces the full auto-generated config:

  • Calendar line-stacks, normalized step plots per calendar theme (vacation zones, day types, seasons, technical).

  • Physical balance checks, Enedis conservation (In = Out) with a delta panel (global balance, load-by-sector, production-by-technology).

  • Regional decompositions, for every regional variable group (temperature, nebulosity, wind speed, air density, humidity, wind direction/gusts, dew point, precip, pressure, radiation, RTE load/solar/wind/nuclear/hydro/bioenergy/pump/exchange): two figures each, (a) spatial consistency (sum/mean of regions vs the national reference + delta), (b) regional breakdown (every region overlaid with the national line).

  • Expert pairwise comparisons, a comprehensive, themed list (every defined pair is plotted when both columns exist, no dedup-skip):

Theme

Pairs (driver to output, and cross-validations)

Solar & Radiation

PVGIS POA vs RTE solar gen; PVGIS POA vs Enedis solar gen; PVGIS POA vs Enedis pseudo-radiation (real vs proxy); PVGIS POA vs ERA5 GHI (when available); PVGIS POA vs nebulosity; PVGIS POA vs site temperature; pseudo-radiation vs solar gen; nebulosity vs solar gen; RTE vs Enedis solar; Enedis solar profiled vs total

Wind

speed vs RTE wind gen; speed vs Enedis wind gen; RTE vs Enedis wind; total vs onshore; speed vs air density; mean speed vs gusts; speed vs direction; air density vs gusts; load- vs wind-weighted speed; pressure vs wind speed

Load & Demand

load vs temperature (thermosensitivity); RTE vs Enedis load; load vs nebulosity; load vs solar (net load); actual vs day-ahead / intraday forecast; intraday vs day-ahead forecast; residential vs professional vs enterprise; industrial vs total; losses vs total; telemetered vs profiled (×4 sectors)

Data maturity

best vs definitive / consolidated / real-time; definitive vs real-time (deviation)

Generation mix & system

solar vs wind (VRE complementarity); nuclear vs load; nuclear vs coal; pumping vs hydro; hydro lake vs run-of-river; gas vs bioenergy; gas CCGT vs TAC; gas vs CO₂ rate; fuel vs coal; hydro vs precipitation; exchange vs Enedis flow; exchange vs load; Enedis production vs TSO export

Temperature & meteo

load- vs solar-weighted temp; Meteo-France vs Enedis temp; Enedis real vs normal temp; temp vs dew point; humidity vs dew point; nebulosity load vs solar weight; precip 1h vs humidity; precip 1h vs 3h; pressure MSL vs station; pressure vs temp

Calendar

TOY vs TOD; bank holiday vs bridge day; summer vs Christmas

  • Orphan auto-pairing, any remaining columns are paired so nothing goes unplotted.

Radiation note: the “radiation vs radiation” check available today is PVGIS POA vs the Enedis nebulosity-pseudo proxy (enedis_meteo_radiation_pseudo_france). The PVGIS vs ERA5 pair is defined but only renders once ERA5 is collected and wired into the assembled dataset (currently ERA5 lives in a separate synop_radiation_era5 table, see METEO.md Table D).

Output per dataset: Outputs/audit/<dataset_name>/NNN_<type>_<title>.png (NNN = config index; <type> ∈ calendar / balance / spatial / regional / pair). A rich *_full dataset yields ~80+ figures; thematic shards yield their applicable subset.

python scripts/audit/gen_notebook_audit.py

Each dataset is loaded for a representative window (2024, long_term rolling = 7 days) via DuckDB so even the multi-GB *_full files load only the needed slice.

gen_all_audits.py (§6) and gen_notebook_audit.py (§7) both write into the same per-dataset folders and are complementary: §6 gives a compact fixed overview, §7 gives the exhaustive notebook reproduction.

8. PVGIS vs ERA5 radiation comparison: gen_pvgis_vs_era5.py (new)

A dedicated comparison that reads both raw radiation tables directly from raw_meteo.duckdb, synop_radiation (PVGIS, plane-of-array 35°) and synop_radiation_era5 (ERA5, GHI), joined on (date, station_id), independent of the assembled CSVs. Output: Outputs/audit/comparison/pvgis_vs_era5.png, a 4-panel figure: (A) monthly climatology, (B) monthly-mean time series over the overlap, (C) POA-vs-GHI scatter with correlation / regression / ratio, (D) per-station mean bias.

It runs safely today and prints a clear message if ERA5 isn’t collected yet, so it is “ready and waiting”:

python scripts/audit/gen_pvgis_vs_era5.py

PVGIS POA (35° tilt) and ERA5 GHI (horizontal) are different physical quantities, so the comparison reports correlation, bias and the POA/GHI ratio, not equality.

9. Hierarchy witnesses: gen_hierarchies.py (new)

The visual companion to documentation/HIERARCHIES.md: it renders the verified structural relationships between columns from a single DuckDB scan of dataset_30min_full.csv. Three figures. (Definitional relationships, segment total = telemetered + profiled, calendar encodings, are excluded; only non-trivial ones are shown.)

hierarchy_balances.png to Outputs/audit/comparison/hierarchy_balances.png

The energy inflow/outflow balance on both grids, side by side. (A) distribution (Enedis) over a representative week, import + production (inflow) vs load + losses + export_rte + export_dso (outflow), the two lines coinciding; (B) transmission (RTE) over the same week, Σ generation + net imports + pump (supply) vs load, with the shaded gap; (C) the Enedis balance scatter pinned to y = x (mean residual ≈ 0.002 MW); (D) the RTE balance scatter, also on y = x (≈ 28 MW / 0.05% residual). Look for: both balances closing, the distribution one to machine precision, the transmission one to ~0.05% (Corsica + overseas + rounding/maturity-lag). pump carries the real pumped-storage consumption (≈ −800 MW) after the clip-sign fix.

hierarchy_regional.png to Outputs/audit/comparison/hierarchy_regional.png

RTE national vs Σ 12 metropolitan regions for all eight families (load, nuclear, wind, solar, hydro, bioenergy, pump, exchange_net) as a 2×4 scatter grid, each annotated with its rel|err|. Look for: all families hugging the diagonal (national ≈ Σ regions to <0.4%) without closing exactly (Corsica + overseas + rounding/lag); pump collapsing to a near-zero blob.

hierarchy_aggregations.png to Outputs/audit/comparison/hierarchy_aggregations.png

(A) Enedis load decomposition, national load (published) vs Σ 4 segment totals (closes exactly); (B) Enedis production total vs Σ 5 streams (small systematic gap); (C) RTE family totals as horizontal bars of the exposed sub-streams plus the hatched unlisted remainder (STEP turbining, cogeneration, biomass, offshore wind); (D) the RTE golden record, monthly composition of rte_load_france by source nature. Look for: the hatched remainder dominating the bioenergy bar (biogas is only ~26%); the golden record shifting from Definitive to Consolidated/Real-Time at the recent edge.

python scripts/audit/gen_hierarchies.py

10. Imputation engine comparison: gen_imputation_compare.py (new)

gen_imputation_compare.py to Outputs/audit/comparison/imputation_tsicl_vs_xgboost.png

Overlays, on real gaps, the original series (with the gap shaded), the TS-ICL fill and the XGBoost fill, with observed datapoints before and after each gap for context. It auto-locates interior gaps in the non-imputed dataset_30min_full.csv and reads both engine outputs (Outputs/Imputed/tsicl/ and Outputs/Imputed/xgboost/), so it requires both engines to have been run. Look for: TS-ICL tracking the local rhythm/amplitude smoothly vs XGBoost’s jaggier over/undershoot; e.g. the negative pumped-storage spikes recovered on rte_pump_grand_est.

python scripts/audit/gen_imputation_compare.py

Notes

  • Scripts are read-only on the data and safe to run while other pipeline stages execute (they only read the CSV / warehouse).

  • Radiation columns follow the convention radiation_{plane}_{source}_wm2 (e.g. meteo_radiation_poa_pvgis_wm2_france_solar).

  • For a quick driver check, gen_radiation_regional.py and gen_solar_clock.py read only the columns they need (via DuckDB) and run in seconds even on the full dataset.