fix(prices): take Nord Pool when the Sourceful harvest is stale - #1044
fix(prices): take Nord Pool when the Sourceful harvest is stale#1044frahlg wants to merge 1 commit into
Conversation
Sourceful's ENTSO-E cache for SE3 can sit unhealthy for a day after Nord Pool has published tomorrow. Predicted from midnight was that gap, not a chart bug. Fall back to Nord Pool's day-ahead dataportal and fetch again at 13:05 Europe/Stockholm.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6332d1d. Configure here.
| if err != nil { | ||
| loc = time.UTC | ||
| } | ||
| date := day.In(loc).Format("2006-01-02") |
There was a problem hiding this comment.
Fallback fetches the wrong delivery date
Medium Severity
NordPoolProvider.Fetch reformats day in Europe/Stockholm, but fetchAndStore and the primary providers use day's own calendar date. When those dates differ, the fallback requests the wrong Nord Pool day and published prices are missed.
Reviewed by Cursor Bugbot for commit 6332d1d. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6332d1dccb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sp.Currency = currency | ||
| sp.FX = fx | ||
| p = sp | ||
| p = withFallback(sp, np) |
There was a problem hiding this comment.
Wrapping every Sourceful configuration applies this fallback to zones that Nord Pool does not publish, including configured zones such as ES, PT, and the IT-* areas, as well as identifiers that differ between the providers. When Sourceful is empty for one of these zones, the secondary request cannot supply matching entries, so the advertised fallback still leaves the planner using predicted prices. Translate supported area identifiers and only install this fallback for Nord Pool-covered zones.
Useful? React with 👍 / 👎.
| case <-catch.C: | ||
| s.fetchAndStore(ctx) | ||
| catch.Reset(time.Until(nextDayAheadCatch(time.Now().Add(time.Minute)))) |
There was a problem hiding this comment.
If tomorrow's prices are not available at exactly 13:05, this branch performs one fetch and immediately resets the catch timer for the following day. Nord Pool publication can occur later than 13:05, and the independent hourly ticker is aligned to process startup, so if its next tick also precedes publication the real prices remain unavailable for almost another hour—the delay this catch is intended to avoid. Schedule short retries after an empty publication catch rather than abandoning the catch window after one request.
Useful? React with 👍 / 👎.
miravoss26
left a comment
miravoss26
left a comment
There was a problem hiding this comment.
Adds a Nord Pool fallback for the sourceful and elprisetjustnu price providers: when the primary returns an error or an empty day (e.g. ENTSO-E down, or the day just isn't in the harvest cache yet), fallbackProvider retries against Nord Pool's public dataportal (no API key) before giving up. Also adds a daily 13:05 Europe/Stockholm timer alongside the existing hourly tick, so tomorrow's day-ahead gets picked up close to Nord Pool's actual publication time instead of up to an hour late.
Correctness read:
fallbackProvider.Fetch— falls through to secondary on eithererr != nilorlen(rows) == 0from primary, which covers both failure modes that actually happen (an outage, and a day not yet published). Error handling on the secondary path is sensible (prefers primary's error over secondary's if both fail).NordPoolProvider.Fetch— 404 mapped to(nil, nil)(not-yet-published, not an error) rather than bubbling up; currency mismatch between requested and returned is checked explicitly rather than trusting the API; price is correctly divided by 1000 (Nord Pool quotes per MWh) before optional FX conversion; zone lookup falls back to a case-insensitive scan. All sensible defensive choices for a public unauthenticated API you don't control the shape of.nextDayAheadCatch+ its test cover both sides of the 13:05 boundary correctly.- One gap worth naming, not blocking:
cfg.Provider == "entsoe"(direct ENTSO-E-key mode) is left unwrapped — onlysourcefulandelprisetjustnuget the Nord Pool fallback. Doc comment sayssourcefulis the default or deployed provider, so this likely covers the actual 08-31 incident (ENTSO-E-503 outage on the Sourceful-harvest path), but a direct-entsoe-configured deployment would still have no fallback.
Security: clean. Public API, no key, no secrets. zone/date/apiCur are query-escaped before going into the URL. No new dependencies (stdlib only). Response body is capped at 512 bytes before logging on a non-200.
Tests are hermetic (httptest.Server, no live network) and cover the cases that matter: parsing, empty/404 day, fallback triggering, currency-mismatch rejection, and the catch-timer boundary. CI green.
Safe to merge from my read.
|
Maybe also fallback on the provider we have in the list already if it does not answer - atleast momentarily? |
|
@HuggeK on the extra fallback: This PR already does that for the two providers that can 404 an unpublished day: What it does not do is chain the other name in the settings list (sourceful → elpriset → Nord Pool, or the reverse). I would keep it that way:
So: momentary Sourceful silence is already covered. A third hop through the other listed provider would not have filled that night's gap faster than Nord Pool. |
3 participants
Summary
The Plan chart marked tonight 00:00 as predicted even though Nord Pool had already published tomorrow.
That was not a drawing bug. Default Sourceful harvest for SE3 was unhealthy: last harvested 2026-08-30 17:36 UTC (~24 h stale). Nord Pool published 2026-09-01 at 13:23 CEST. Sourceful then 404s for that day, so Core fills the gap with the ML twin.
This PR:
Test plan
price: primary empty, using Nord Pool day-aheadwhen the harvest is down.make verifyis green.Note
Medium Risk
Changes live price ingestion and scheduling for default Swedish/EU installs; wrong fallback or FX could mislabel costs, but behavior is gated on empty primary responses and covered by new tests.
Overview
Fixes tomorrow’s plan showing predicted spot prices after midnight when Nord Pool has already published day-ahead but the Sourceful harvest cache is still empty (404).
Sourceful and elprisetjustnu are now wrapped with a primary → Nord Pool fallback: if the primary returns no rows for a day, FTW fetches from Nord Pool’s public day-ahead dataportal (no API key), with currency/FX aligned to install config. The configured provider name stays the primary unless Nord Pool actually supplies the data (logged as
price: primary empty, using Nord Pool day-ahead). ENTSO-E is unchanged.The price fetch loop keeps the hourly tick and adds a 13:05 Europe/Stockholm one-shot timer so tomorrow’s auction is picked up soon after publication instead of waiting up to an hour.
Reviewed by Cursor Bugbot for commit 6332d1d. Bugbot is set up for automated code reviews on this repo. Configure here.