---
orphan: true
---

# FORCE: Audit & Visualization Scripts

Copyright (c) 2025-2026 EDF (Electricité De France).
Author: Yann Allioux

This folder contains standalone, reproducible scripts that render the project's
**scientific audit figures** from the assembled dataset. Each script is independent and
can be run on its own; together they form a visual QA suite for the pipeline output.

## How to run

All scripts read the assembled dataset `Outputs/dataset_30min_full.csv` (produced by
`energy-assemble`) and require a headless matplotlib backend on Windows:

```bash
# from the repository root, with the project environment active
set MPLBACKEND=Agg # Windows (PowerShell: $env:MPLBACKEND="Agg")
python scripts/audit/gen_availability_heatmap.py
```

Outputs land either in `Outputs/audit/` (diagnostic figures) or `paper/images/`
(figures referenced by the JOSS paper), as noted per script below.

---

## 1. Data completeness & coverage

### `gen_availability_heatmap.py` to `Outputs/audit/availability_heatmap.png`
Weekly fill-rate (fraction of non-null timestamps) for **every numeric column**, grouped
by source (Enedis / Météo / RTE / Calendar) and rendered as a red to green heatmap. Also
exports `Outputs/audit/availability_stats.csv` (per-column global and in-window
availability).
**Look for:** the Enedis block turning green only from ~2020 (smart-meter rollout); the
post-2023 gap in the radiation rows (PVGIS coverage ends 2023); any unexpected white
(missing) stripes in RTE/Météo.

### `gen_scope_timeline.py` to `paper/images/scope_timeline.png`
Horizontal Gantt of each data stream's active period, annotated with its fill rate and
grouped by provider (TSO / DSO / Météo-France / ODRÉ / Etalab).
**Look for:** the 2012 start of RTE vs the 2020 onset of Enedis DSO observability.

---

## 2. Cross-source validation

### `gen_validation_stacked.py` to `paper/images/validation_stacked.png`
Two panels: (A) the **multi-barycenter effect**, solar-weighted vs population-weighted
national temperature, whose daytime divergence confirms PV assets sit in climatically
distinct areas; (B) **TSO vs DSO load harmonization**, RTE national load vs Enedis
distribution load, the gap representing transmission-connected industry + grid losses.
**Look for:** tight temporal alignment of the two load curves; a consistent (not erratic)
TSO-DSO gap.

---

## 3. Spatial decomposition

### `gen_spatial_breakdown.py` to `paper/images/spatial_breakdown_{wind,solar}.png`
Stacked area of regional generation (wind and solar) summing to the national signal
(dashed line), illustrating the **portfolio effect**.
**Look for:** colored bands summing to the national line (bottom-up consistency); the
north/west concentration for wind vs the southern dominance for solar.

### `gen_radiation_regional.py` to `Outputs/audit/radiation_regional.png` *(new)*
Monthly **PVGIS radiation** climatology for each region (thin lines) against the
national solar-weighted reference (bold). Applies the notebook's "find national
reference" idea to the new radiation feature.
**Look for:** southern regions (Occitanie, PACA) sitting above the national curve;
all regions sharing the summer-peaked seasonal shape.

---

## 4. Data maturity & observability

### `gen_maturity_forensics.py` to `paper/images/maturity_forensics.png`
The Golden Record lifecycle for national load: (A) the full timeline coloured by quality
level (Definitive to Consolidated to Real-Time); (B) a zoom on the Definitive to Consolidated
handoff showing the inter-stream delta.
**Look for:** no discontinuity in the synthesized signal at the maturity cutoffs.

### `gen_enedis_observability.py` to `paper/images/enedis_observability_*.png`
Smart-meter rollout dashboards (composite + per sector: residential / professional /
entreprise): (A) telemetered vs profiled load with a rising telemetry-rate line;
(B) the average diurnal cycle.
**Look for:** the green ("telemetered") area expanding over time; the residential
nocturnal water-heating / EV peak absent from the generalized profile.

---

## 5. Physical & driver relationships *(new)*

### `gen_generation_mix.py` to `Outputs/audit/generation_mix.png` *(new)*
Stacked area of the full RTE generation mix (nuclear, hydro, gas, coal, bioenergy, wind,
solar) against the national load (dashed) for a representative winter week.
**Look for:** nuclear baseload, the daily solar bumps, and dispatchable gas tracking the
load peaks.

### `gen_thermosensitivity.py` to `Outputs/audit/thermosensitivity.png` *(new)*
Scatter of national load vs population-weighted temperature, coloured by month, the
classic thermosensitivity relationship.
**Look for:** the "hockey-stick", load rising steeply below ~15 °C (electric heating)
with a milder summer cooling uptick; winter months (cold colours) on the high-load arm.

### `gen_solar_clock.py` to `Outputs/audit/solar_clock.png` *(new)*
Heatmap of mean solar generation across **hour-of-day × month**, the solar production
envelope.
**Look for:** the bright midday core widening from winter to summer (longer days, higher
sun); strict zeros at night confirming the physical night constraint.

---

## 6. Per-dataset sweep: `gen_all_audits.py` *(new)*

Runs a **generic, column-gated** audit over *every* CSV dataset, both the assembled ones
in `Outputs/` and the imputed ones in `Outputs/Imputed/`, and writes **one folder of
figures per dataset**:

```
Outputs/audit/<dataset_name>/
    missingness.png            # % missing per column (bar)
    availability_heatmap.png   # weekly fill rate, all numeric columns
    national_overview.png      # daily-mean time series of key national variables
    correlation.png            # correlation matrix of key national variables
    distributions.png          # sampled value histograms
    generation_mix.png         # only if RTE generation columns are present
    thermosensitivity.png      # only if load + national temperature are present
    solar_clock.png            # only if rte_solar_france is present
```

Each figure is emitted **only when the dataset actually contains the required columns**, so
the rich `*_full` datasets get the complete set while the thematic shards
(`*_france_load`, `*_france_solar`, `*_france_wind`) get the applicable subset.

Because it runs over both `Outputs/` and `Outputs/Imputed/`, you get **non-imputed vs
imputed** versions side by side, most informative on `missingness.png` and
`availability_heatmap.png`, where the imputed folders should show the structural gaps
filled (except the by-design Enedis pre-2020 and post-2023 PVGIS-radiation windows).

Run (generates all per-dataset folders in one pass):
```bash
python scripts/audit/gen_all_audits.py
```
Reads are column-selective via DuckDB, so even the multi-GB `*_full` datasets are handled by
streaming aggregation rather than full in-memory loads.

## 7. Faithful notebook reproduction: `gen_notebook_audit.py` *(new)*

This is a direct port of `audit.ipynb`'s interactive engine (`plot_analysis` +
`generate_hybrid_config`), turned into a batch generator that **saves every figure the
notebook produces** for **every** CSV dataset (assembled + imputed). Unlike
`gen_all_audits.py` (a small fixed set), this reproduces the *full* auto-generated config:

- **Calendar line-stacks**, normalized step plots per calendar theme (vacation zones, day
  types, seasons, technical).
- **Physical balance checks**, Enedis conservation (In = Out) with a delta panel
  (global balance, load-by-sector, production-by-technology).
- **Regional decompositions**, for every regional variable group (temperature, nebulosity,
  wind speed, air density, humidity, wind direction/gusts, dew point, precip, pressure,
  radiation, RTE load/solar/wind/nuclear/hydro/bioenergy/pump/exchange): **two** figures
  each, (a) *spatial consistency* (sum/mean of regions vs the national reference + delta),
  (b) *regional breakdown* (every region overlaid with the national line).
- **Expert pairwise comparisons**, a comprehensive, themed list (every defined pair is
  plotted when both columns exist, no dedup-skip):

| Theme | Pairs (driver to output, and cross-validations) |
| --- | --- |
| **Solar & Radiation** | PVGIS POA vs RTE solar gen; PVGIS POA vs Enedis solar gen; **PVGIS POA vs Enedis pseudo-radiation** (real vs proxy); **PVGIS POA vs ERA5 GHI** (when available); PVGIS POA vs nebulosity; PVGIS POA vs site temperature; pseudo-radiation vs solar gen; nebulosity vs solar gen; RTE vs Enedis solar; Enedis solar profiled vs total |
| **Wind** | speed vs RTE wind gen; speed vs Enedis wind gen; RTE vs Enedis wind; total vs onshore; speed vs air density; mean speed vs gusts; speed vs direction; air density vs gusts; load- vs wind-weighted speed; pressure vs wind speed |
| **Load & Demand** | load vs temperature (thermosensitivity); RTE vs Enedis load; load vs nebulosity; load vs solar (net load); **actual vs day-ahead / intraday forecast**; intraday vs day-ahead forecast; residential vs professional vs enterprise; industrial vs total; losses vs total; telemetered vs profiled (×4 sectors) |
| **Data maturity** | best vs definitive / consolidated / real-time; definitive vs real-time (deviation) |
| **Generation mix & system** | solar vs wind (VRE complementarity); nuclear vs load; nuclear vs coal; pumping vs hydro; hydro lake vs run-of-river; gas vs bioenergy; gas CCGT vs TAC; gas vs CO₂ rate; fuel vs coal; hydro vs precipitation; exchange vs Enedis flow; exchange vs load; Enedis production vs TSO export |
| **Temperature & meteo** | load- vs solar-weighted temp; Meteo-France vs Enedis temp; Enedis real vs normal temp; temp vs dew point; humidity vs dew point; nebulosity load vs solar weight; precip 1h vs humidity; precip 1h vs 3h; pressure MSL vs station; pressure vs temp |
| **Calendar** | TOY vs TOD; bank holiday vs bridge day; summer vs Christmas |

- **Orphan auto-pairing**, any remaining columns are paired so nothing goes unplotted.

> **Radiation note:** the "radiation vs radiation" check available today is **PVGIS POA vs the
> Enedis nebulosity-pseudo proxy** (`enedis_meteo_radiation_pseudo_france`). The **PVGIS vs
> ERA5** pair is defined but only renders once ERA5 is collected *and* wired into the assembled
> dataset (currently ERA5 lives in a separate `synop_radiation_era5` table, see METEO.md
> Table D).

Output per dataset: `Outputs/audit/<dataset_name>/NNN_<type>_<title>.png`
(`NNN` = config index; `<type>` ∈ `calendar` / `balance` / `spatial` / `regional` / `pair`).
A rich `*_full` dataset yields ~80+ figures; thematic shards yield their applicable subset.

```bash
python scripts/audit/gen_notebook_audit.py
```
Each dataset is loaded for a representative window (2024, `long_term` rolling = 7 days) via
DuckDB so even the multi-GB `*_full` files load only the needed slice.

> `gen_all_audits.py` (§6) and `gen_notebook_audit.py` (§7) both write into the same
> per-dataset folders and are complementary: §6 gives a compact fixed overview, §7 gives the
> exhaustive notebook reproduction.

## 8. PVGIS vs ERA5 radiation comparison: `gen_pvgis_vs_era5.py` *(new)*

A dedicated comparison that reads **both raw radiation tables directly** from
`raw_meteo.duckdb`, `synop_radiation` (PVGIS, plane-of-array 35°) and
`synop_radiation_era5` (ERA5, GHI), joined on `(date, station_id)`, independent of the
assembled CSVs. Output: `Outputs/audit/comparison/pvgis_vs_era5.png`, a 4-panel figure:
(A) monthly climatology, (B) monthly-mean time series over the overlap, (C) POA-vs-GHI
scatter with correlation / regression / ratio, (D) per-station mean bias.

It runs safely **today** and prints a clear message if ERA5 isn't collected yet, so it is
"ready and waiting":
```bash
python scripts/audit/gen_pvgis_vs_era5.py
```
PVGIS POA (35° tilt) and ERA5 GHI (horizontal) are **different physical quantities**, so the
comparison reports correlation, bias and the POA/GHI ratio, not equality.

## 9. Hierarchy witnesses: `gen_hierarchies.py` *(new)*

The visual companion to [documentation/HIERARCHIES.md](../../documentation/HIERARCHIES.md):
it renders the verified structural relationships between columns from a single DuckDB scan of
`dataset_30min_full.csv`. Three figures. (Definitional relationships, segment
`total = telemetered + profiled`, calendar encodings, are excluded; only non-trivial ones are
shown.)

### `hierarchy_balances.png` to `Outputs/audit/comparison/hierarchy_balances.png`
The energy **inflow/outflow balance on both grids**, side by side. (A) **distribution (Enedis)**
over a representative week, `import + production` (inflow) vs `load + losses + export_rte +
export_dso` (outflow), the two lines coinciding; (B) **transmission (RTE)** over the same week, 
`Σ generation + net imports + pump` (supply) vs `load`, with the shaded gap; (C) the Enedis
balance scatter pinned to `y = x` (mean residual ≈ 0.002 MW); (D) the RTE balance scatter, also
on `y = x` (≈ 28 MW / 0.05% residual).
**Look for:** both balances closing, the distribution one to machine precision, the
transmission one to ~0.05% (Corsica + overseas + rounding/maturity-lag). `pump` carries the
real pumped-storage consumption (≈ −800 MW) after the clip-sign fix.

### `hierarchy_regional.png` to `Outputs/audit/comparison/hierarchy_regional.png`
RTE **national vs Σ 12 metropolitan regions** for all eight families (load, nuclear, wind, solar,
hydro, bioenergy, pump, exchange_net) as a 2×4 scatter grid, each annotated with its `rel|err|`.
**Look for:** all families hugging the diagonal (national ≈ Σ regions to <0.4%) without closing
exactly (Corsica + overseas + rounding/lag); `pump` collapsing to a near-zero blob.

### `hierarchy_aggregations.png` to `Outputs/audit/comparison/hierarchy_aggregations.png`
(A) Enedis **load decomposition**, national load (published) vs Σ 4 segment totals (closes
exactly); (B) Enedis **production** total vs Σ 5 streams (small systematic gap); (C) RTE family
totals as horizontal bars of the **exposed sub-streams plus the hatched unlisted remainder**
(STEP turbining, cogeneration, biomass, offshore wind); (D) the RTE **golden record**, monthly
composition of `rte_load_france` by source nature.
**Look for:** the hatched remainder dominating the bioenergy bar (biogas is only ~26%); the
golden record shifting from Definitive to Consolidated/Real-Time at the recent edge.

```bash
python scripts/audit/gen_hierarchies.py
```

## 10. Imputation engine comparison: `gen_imputation_compare.py` *(new)*

### `gen_imputation_compare.py` to `Outputs/audit/comparison/imputation_tsicl_vs_xgboost.png`
Overlays, on real gaps, the **original** series (with the gap shaded), the **TS-ICL** fill and
the **XGBoost** fill, with observed datapoints **before and after** each gap for context. It
auto-locates interior gaps in the non-imputed `dataset_30min_full.csv` and reads both engine
outputs (`Outputs/Imputed/tsicl/` and `Outputs/Imputed/xgboost/`), so it requires both engines
to have been run.
**Look for:** TS-ICL tracking the local rhythm/amplitude smoothly vs XGBoost's jaggier
over/undershoot; e.g. the negative pumped-storage spikes recovered on `rte_pump_grand_est`.

```bash
python scripts/audit/gen_imputation_compare.py
```

## Notes
- Scripts are read-only on the data and safe to run while other pipeline stages execute
  (they only read the CSV / warehouse).
- Radiation columns follow the convention `radiation_{plane}_{source}_wm2`
  (e.g. `meteo_radiation_poa_pvgis_wm2_france_solar`).
- For a quick driver check, `gen_radiation_regional.py` and `gen_solar_clock.py` read only
  the columns they need (via DuckDB) and run in seconds even on the full dataset.