Calendar and Temporal Feature Pipeline¶

Back to README

This document details the extraction and feature engineering algorithms applied to the pipeline’s temporal dimension. The pipeline constructs a deterministic “skeleton” of time (spanning from 2012 to two years into the future) and enriches it with exogenous context, such as public holidays, school vacations, Daylight Saving Time (DST) transitions, and special periods like COVID-19 lockdowns.

This enriched calendar serves as the baseline grid for all time-series alignment in the broader extract, transform, and load (ETL) pipeline.

Part 1: Collection Module (`collect_calendar.py`)¶

This module generates the fundamental time index and retrieves official administrative calendars from French government APIs. It creates a temporal baseline that resolves timezone complexities—specifically bridging Coordinated Universal Time (UTC) and local European time (Europe/Paris) containing DST shifts.

Data Sources¶

Primary Source: Generated continuous time skeleton (via Python’s pandas).
Holidays: data.gouv.fr dataset for national public holidays [Etalab, 2026].
School Vacations: A merged dataset aggregating two sources to ensure historical continuity:
1. Post-2018: The modern interval-format dataset from the Ministry of Education [Ministère de l'Éducation Nationale, 2026].
2. Pre-2018: An archived daily-boolean dataset to backfill historical data [Augusti, 2018].
Scope: Metropolitan France (broken down into school zones A, B, and C).

Key Processing Steps¶

Merging Historical and Modern Datasets: The French administration changed how it formats school vacation data in 2018. The collector automatically detects the schema, applies heuristic separator detection, and standardizes both into a single continuous timeline.
Temporal Skeleton Generation: Creates standard high-frequency time grids (15-minute and 30-minute intervals) spanning the entire project horizon.

Part 2: Transformation Module (`transform_calendar.py`)¶

The raw calendar data is transformed into a rich set of categorical and continuous features designed specifically for machine learning models (e.g., Gradient Boosting Machines, Neural Networks).

1. Categorical Day Classification¶

The model explicitly flags days that exhibit abnormal energy consumption patterns:

Weekends and Holidays: Boolean flags for Saturdays/Sundays and official bank holidays.
Bridge Days (Ponts): Automatically detects if a Tuesday or Thursday is a holiday, subsequently flagging the adjacent Monday or Friday as a “bridge day” (where a significant portion of the workforce takes leave).
School Vacations: Boolean flags tracking whether any zone (A, B, or C) is currently on vacation, which directly impacts residential versus commercial load distribution.

2. Time Standardization: Continuous Local Time (CLT)¶

Energy consumption is driven by human activity, which follows local clock time, not UTC. However, standard local time includes non-linear jumps (skipping an hour in spring, repeating an hour in autumn) due to DST.

To solve this, the pipeline maps all data to Continuous Local Time (CLT). CLT is a monotonic index that aligns with the local human clock but mathematically removes the DST discontinuities, preventing models from interpreting DST shifts as sudden, massive drops or spikes in energy demand.

3. Cyclical Encoding (Time of Day / Time of Year)¶

To help machine learning models understand the circular nature of time (e.g., that 23:45 is only 15 minutes away from 00:00), we normalize the time steps into continuous cyclical variables:

Time of Day (tod): Normalized from 0 to 1 based on the daily frequency limit.
- Formula: \(tod = \frac{\text{minute\_of\_day}}{\text{steps\_per\_day}}\)
Time of Year (toy): Normalized from 0 to 1 based on the exact progress through the current year (accounting for leap years).
- Formula: \(toy = \frac{\text{day\_of\_year} - 1}{\text{days\_in\_year}} + \frac{tod}{\text{days\_in\_year}}\)

(Note: In downstream imputation models, these tod and toy features are converted into sine and cosine pairs to provide a strictly continuous, circular input.)

4. Composite Categorical Keys¶

Instead of feeding the model multiple separate dummy variables, the pipeline creates composite features that capture cross-interactions. For example, day_type_week_period_hour_changed merges the day of the week with the current DST season (Summer/Winter), allowing tree-based models to split on seasonal weekly routines in a single operation.

5. Validity Masking (Outlier Exclusion)¶

Certain periods contain highly irregular data that can corrupt historical training sets. The module defines a DayValidity flag, setting it to 0 (invalid) for:

DST Switch Days: The exact 24-hour periods during the spring and autumn clock changes.
COVID-19 Lockdowns: The three major French lockdown periods (Spring 2020, Autumn 2020, and Spring 2021). These periods are masked out so forecasting models do not learn pandemic-era demand patterns as standard behavior.

Final Feature Schema¶

The transformed database (transformed_calendar_15min and _30min) produces the following structure:

Column Name	Type	Description
`date`	`TIMESTAMPTZ`	Standard UTC timestamp (Primary Index).
`Date`	`INTEGER`	Local date reference (Format: YYYYMMDD).
`month`	`INTEGER`	Month of the year (1-12).
`year`	`INTEGER`	4-digit year.
`tod`	`DOUBLE`	Time of day (normalized 0.0 to 1.0).
`toy`	`DOUBLE`	Time of year (normalized 0.0 to 1.0).
`week_number`	`INTEGER`	ISO week number (Format: `yyyyww`).
`DayValidity`	`INTEGER`	Validity Flag. 0 if DST switch or COVID-19 period, else 1.
Day Types
`day_type_week`	`INTEGER`	Day of week (0=Monday … 6=Sunday).
`day_type_jf`	`INTEGER`	Is a bank holiday (1/0).
`day_type_vjf`	`INTEGER`	Is the day before a holiday (1/0).
`day_type_ljf`	`INTEGER`	Is the day after a holiday (1/0).
`day_type_week_jf`	`INTEGER`	Is a weekend OR a holiday (1/0).
`day_type_hc`	`INTEGER`	Is a day containing a UTC DST switch (1/0).
`dst_repair_target`	`INTEGER`	Is a day containing a Local CLT DST switch (1/0). Used for gap imputation.
Periods
`period_hour_changed`	`INTEGER`	DST Status (1=Summer Time, 0=Winter Time).
`period_holiday`	`INTEGER`	Is any school vacation active (1/0).
`period_holiday_zone_a`	`INTEGER`	School vacation active in Zone A (1/0).
`period_holiday_zone_b`	`INTEGER`	School vacation active in Zone B (1/0).
`period_holiday_zone_c`	`INTEGER`	School vacation active in Zone C (1/0).
`period_christmas`	`INTEGER`	Is the Christmas holiday period (1/0).
`period_summer`	`INTEGER`	Is the Summer holiday period (1/0).
Composite Keys
`day_type_week_period_hour_changed`	`INTEGER`	Composite: Day of Week + Summer/Winter Season.
`day_type_week_jf_period_holiday`	`INTEGER`	Composite: Weekend/Holiday + Active School Vacation.

Output Integration¶

The outputs of this module are saved to data_warehouse/transform_calendar.duckdb. They are never used in isolation; rather, the calendar_15min and calendar_30min tables are utilized as the Left Table in all downstream SQL joins during the dataset assembly phase. This ensures zero missing timestamps in the final outputs.