Preparing a Backcast

A Backcast is a Balmorel scenario run for one historical Backcast Year, so its results can be compared with what actually happened (see CONTEXT.md). pybalmorel.backcasting covers the whole workflow:

Prepare the Backcast’s inputs: download ENTSO-E data, derive the XKFX transmission override from ENTSO-E’s Observed Max Flow, and write season- and hour-resolved historical fuel prices and a CO2 tax for the Backcast Year. This page.
Run the Balmorel scenario as usual.
Validate the Backcast’s results against ENTSO-E observations. See Validating a Backcast.

The transmission override follows ADRs 0005, 0006 and 0007. pybalmorel.backcasting depends on pybalmorel.entsoe for raw data access and on pybalmorel.MainResults for Balmorel results — never the other way round.

Installation

Backcasting needs the entso-e extra:

pip install pybalmorel[entso-e]

A GAMS installation is also needed for reading Balmorel results, i.e. for transmission-max-flow (which reads the scenario’s XKFX borders) and for validating a Backcast.

CLI

# Download ENTSO-E load, prices, generation and cross-border flows
python -m pybalmorel.backcasting download \
    <entsoe-data-path> \
    <year> \
    [--query load --query crossborder_flows]

# Write XKFX_ENTSOE.inc: transmission overrides from ENTSO-E Observed Max Flow
python -m pybalmorel.backcasting transmission-max-flow \
    <balmorel-scenario-path> \
    <entsoe-data-path> \
    <year>

# Split a saved Yahoo Finance TTF=F page into natural_gas_TTF_<year>.csv
python -m pybalmorel.backcasting ttf-csv <html-path> <historicalprices-path>

# Write FUELPRICE.inc + FUELPRICE_CONSTANT.inc: historical NATGAS/COAL prices
python -m pybalmorel.backcasting fuelprices \
    <balmorel-scenario-path> \
    <historicalprices-path> \
    <year>

# Write TAX_CO2_ST.inc: historical EUA price as CO2 tax
python -m pybalmorel.backcasting co2-tax \
    <balmorel-scenario-path> \
    <historicalprices-path> \
    <year> \
    [--co2-override UNITED_KINGDOM=85 --co2-override SERBIA=0]

Run python -m pybalmorel.backcasting <command> --help for details on each command.

Downloading ENTSO-E data

download fetches the Backcast Year’s ENTSO-E data into <entsoe-data-path>, for every zone in pybalmorel.entsoe.bidding_zones:

`--query`	Files	Used by
`load`	`<year>_<zone>_load.csv`	`format`
`day_ahead_prices`	`<year>_<zone>_day_ahead_prices.csv`	`format`
`generation`	`<year>_<zone>_generation.csv`	`format`
`crossborder_flows`	`<year>_<from>-<to>_crossborder_flows.csv`	`transmission-max-flow`, `format`

Without --query, all four are downloaded. --query is repeatable.

Flows are fetched for every directed pair of neighbouring zones.

The download can be stopped and rerun at any time. On a rerun:

Files that already exist are skipped.
Zones and pairs ENTSO-E answered “no matching data” for on an earlier run are skipped. They are listed in <year>_<query>_nodata.txt in <entsoe-data-path>, one zone per line (or, for flows, two EIC codes separated by a tab). Delete a line to query that zone or pair again, or the whole file to query them all again, e.g. if ENTSO-E has since published the data.
Failed requests are retried. Timeouts, dropped connections and server errors are never written to the _nodata.txt files, and don’t stop the download. Requests time out after 120 seconds, so a connection lost mid-request, e.g. when switching networks, doesn’t hang the download.

Key features

Backcast Calendar: every timestamp ↔ (Season, Term) conversion goes through pybalmorel.backcasting.calendar. S01 T001 is 00:00 UTC on the first Monday of the year, and every day has 24 terms (see ADR 0008). Balmorel results are matched by label, never by position. Inputs not in UTC, such as a demand series in local time, must be converted before they are mapped onto it.
Bidding Zone Mapping: uses bidding_zone_codes/bidding_zone_translation from pybalmorel.entsoe to bridge ENTSO-E bidding zones and Balmorel regions.
Transmission overrides: flows.py computes the Observed Max Flow per border and writes it to XKFX_ENTSOE.inc (see ADRs 0005 and 0006) — a non-destructive override file, never a rewrite of the base XKFX.inc.

Price Year

All generated prices are in €2016. Nominal prices are converted from USD at the Eurostat annual-average EUR/USD rate of the Backcast Year and deflated with the annual euro-area HICP. Both tables are bundled in pybalmorel.backcasting.pricelevel (see ADR 0009), with the Eurostat dataset codes next to them; extend them when a new year is needed.

Fuel prices

The fuelprices command converts historical daily fuel prices into Balmorel’s season- and hour-resolved FUELPRICE(YYY,AAA,FFF,SSS,TTT) input, so a Backcast sees the fuel prices of the actual year. It reads two CSV files from <historicalprices-path>:

Fuel	File	Unit
`NATGAS`	`natural_gas_TTF_<year>.csv` (TTF futures, see below)	€/MWh, converted to €/GJ
`COAL`	`Rotterdam Coal Futures Historical Data.csv` (must cover the year)	USD/t, converted to €/GJ with 30.2 GJ/t

Getting the TTF price files

Yahoo Finance only lets paying users download its history as CSV. Instead, open https://finance.yahoo.com/quote/TTF%3DF/history/, choose a date range (it can span several years), and save the page from the browser as HTML. Then run ttf-csv on it:

python -m pybalmorel.backcasting ttf-csv \
    "2020-2025 Dutch TTF Natural Gas Calendar (TTF=F) Stock Historical Prices & Data - Yahoo Finance.html" \
    <historicalprices-path>

This writes one natural_gas_TTF_<year>.csv per calendar year on the page, overwriting any that already exist. Each file has the columns Date,Open,High,Low,Close,Adj Close,Volume. Date has no year (Dec 30), and Volume has no thousands separators, so the file is valid CSV. The fuelprices command uses Adj Close.

Daily prices apply to the UTC calendar day on the Backcast Calendar, carrying the last price forward over weekends and holidays. Two files are written to <scenario-path>/data/:

FUELPRICE.inc — the (S,T)-resolved NATGAS and COAL prices.
FUELPRICE_CONSTANT.inc — flags every other fuel as constant over the year.

Only natural gas and coal are covered; oil is not covered yet. For annual averages instead, see the average fuel prices CLI.

CO2 tax

The co2-tax command converts the EEX EUA primary-auction results into Balmorel’s TAX_CO2_ST(YYY,CCC,SSS,TTT) input (see ADR 0002 in the Balmorel repo) and writes <scenario-path>/data/TAX_CO2_ST.inc. It reads emission-spot-primary-market-auction-report-<year>-data.xlsx, found anywhere under <historicalprices-path>, downloaded from EEX.

Only successful auctions are used. Days with several auctions (EU, DE, PL) get the mean clearing price, and days without one carry the last price forward.

Every country gets the EUA price. Use --co2-override COUNTRY=VALUE (repeatable, Balmorel CCC names, nominal €/tCO2 of the Backcast Year) for countries with a different carbon price, such as the UK, or 0 for countries without one.

Note

fuelprices and co2-tax never edit the scenario’s options, but they warn when Balmorel would ignore what they wrote:

balopt.opt (the scenario’s own, else base/model/balopt.opt) must set FUELPRICE_DOL=YYY_AAA_FFF_SSS_TTT and TAX_CO2_DOL=YYY_CCC_SSS_TTT.
HYDROGEN_FUELPRICE.inc must not assign COAL or NATGAS prices, since FUELPRICE_HYDROGEN overrides IFUELPRICE wherever it is nonzero.

Configuration

API Key: set ENTSOE_API_KEY in .env, or provide it when prompted.
GAMS Path: set GAMS_SYSTEM_DIR in .env for Balmorel result processing.
Output: raw ENTSO-E data in <entsoe-data-path>; XKFX, fuel price and CO2 tax overrides written to <scenario-path>/data/; validation data cached in <scenario-path>/backcastoutput/.

Data flow

Raw ENTSO-E data fetched via pybalmorel.entsoe (download) and saved as CSV.
The XKFX override is computed from the ENTSO-E flows (transmission-max-flow).
Historical prices are mapped onto the UTC Backcast Calendar and written (fuelprices, co2-tax).

The Backcast is then run in Balmorel and validated.