feat(weather): add STRÅNG as an irradiance source and declare source coverage - #734
feat(weather): add STRÅNG as an irradiance source and declare source coverage#734HuggeK wants to merge 6 commits into
Conversation
|
Rebased onto #962 ( Conflict resolution:
Force-pushed HuggeK |
…coverage STRANG becomes a real irradiance source rather than an unreferenced client, every external data source now says where in the world it works, and the location picker moves off Leaflet. STRANG's parameter set was mapped against the live API because SMHI's apidocs pages 404: exactly 116-122 exist. Identification was confirmed by physics rather than by guessing - at solar noon 121 + 122 = 723.0 + 87.5 = 810.5, exactly parameter 117, and 119 caps at 60, i.e. minutes within the hour. STRANG publishes no cloud cover; it is a radiation model. Cloudiness is instead derived from sunshine duration as 1 - minutes/60, which is observed rather than inferred but coarser: blind to thin cirrus, undefined at night. CloudCover() therefore returns an explicit unknown instead of defaulting to clear, because those two lead to opposite decisions. The new coverage registry makes an existing silence explicit. STRANG is Nordic-only and every price provider is European, so sites elsewhere were getting empty results with no explanation (srcfl#726). GET /api/data-sources now reports area, countries, licence and whether each source reaches this site, and the Weather tab renders it under the map. Bounds are advisory: STRANG's grid is rotated, so a lat/lon box can only ever be a superset - all four in-box corners were probed and returned no data. False is definitive, true means worth trying. Scoring now declines to start outside the domain instead of retrying nightly forever. Stacked on srcfl#718, which carries the plane-of-array wiring this builds on. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
…a CDN Upstream ships Leaflet on the box (srcfl#910) for a reason that applies just as hard to its replacement: the box UI must not execute third-party JS from a CDN, and the map must load when the gateway cannot reach the internet. So MapLibre GL JS 6.0.0 moves into web/vendor/maplibre — the ESM entry, its code-split chunk, its worker, the stylesheet and the license — and the loader imports it same-origin, which also spares MapLibre its blob: worker detour. The now-unused vendored Leaflet goes away with its test; maplibre-vendor.test.mjs pins the new contract, including the per-request referrerPolicy opt-in that keeps OSM volunteer tile servers from 403ing under the box's Referrer-Policy: no-referrer. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
e92acd2 to
ba47f6a
Compare
The package comment linked the legacy strang.smhi.se site and claimed the apidocs 404 — they live at opendata.smhi.se/metanalys/strang, with the model described under smhi.se/forskning. The empirical parameter-code confirmation stays: the docs name the quantities, not the code-to-quantity mapping. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
http.ServeFile resolves Content-Type through the operating system's MIME table — the registry on Windows — and a host that maps .mjs to text/plain does not merely mislabel the vendored MapLibre entry: the app sends X-Content-Type-Options: nosniff, so the browser is required to refuse the module, and the weather map dies with 'failed to fetch dynamically imported module'. Seen live on a Windows host serving the demo build. handleStatic now pins the Content-Type for every asset kind the web tree ships (.html .css .js .mjs .svg .png .jpg), so nothing depends on host MIME state. Covered by a test that would have caught this wherever it ran. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
… chart Price-driven planning is Europe-only and nothing said so — a site outside ENTSO-E got an empty price curve with no explanation (#726). - go/internal/coverage is the registry: every forecast and price source declares its area, country list, licence, and a lat/lon box. - GET /api/data-sources answers for this site, or for a pin still being dragged. covers:false is definitive; covers:true means worth trying. - The Weather tab renders that under the map. The Price tab flags a location outside every European bidding zone. - docs/data-coverage.md is the prose; the registry is the source of truth. STRÅNG scoring and roof geometry are not shipped, so they are not advertised. #734 already carries a later version of this plus STRÅNG and should rebase onto this when it lands. Part of #726.
Same four-file code-split layout as 6.0.0, from the maplibre-gl npm package's dist/. The entry still imports its shared and worker chunks by the vendored names and the shared chunk still honours the per-request referrerPolicy the OSM tiles depend on. Map mount, building picker and the TerraDraw PV drawing tool re-verified in headless Edge on the new build; served entry hash matches the tarball byte for byte. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
miravoss26
left a comment
There was a problem hiding this comment.
Large PR (4.4k additions across STRÅNG irradiance, coverage metadata, PV performance calibration, and a Leaflet→MapLibre vendor swap) — reviewed in full, security screen + correctness pass.
Security screen:
- New external destination:
opendata-download-metanalys.smhi.se(SMHI's public STRÅNG API) — matches the PR's stated purpose, no auth/keys needed, read-only. - New SQL (
pv_performance_dailyload/save instate/store.go) uses parameterized queries (WHERE day = ?,db.Exec(q, ...)) — no string-built SQL. - New endpoints
GET /api/data-sourcesandGET /api/pv/performanceare registered through the existings.handle(method, Read, handler)permission wrapper, same tier as neighboring reads — no new/weakened auth pattern. - MapLibre is vendored same-origin under
web/vendor/maplibre/(no CDN), which reduces surface vs the Leaflet-from-CDN baseline; the referrer-policy opt-in for OSM tiles is carried over. I can't independently re-verify the claimed sha256-against-npm-registry match from here — worth a human spot-check before merge given it's an 806-line minified bundle, but nothing in the vendor README/LICENSE diff itself is suspicious. - No secrets, no
exec/eval, no obviously new authz surface elsewhere in the diff.
Correctness:
PVBand'slow ≤ base ≤ high ≤ 0invariant is asserted over a table including NaN/±Inf — the right shape for a robust-band extraction.- Calibration factor is refused outside a plausible band rather than silently rescaling a misconfigured site's forecast — good failure mode (config fault surfaces instead of getting hidden).
- STRÅNG parameter-code identification is cross-checked two independent ways (121+122=117, 119 caps at 60) rather than taken from the model page alone — solid provenance for a code mapping SMHI doesn't publish directly.
CloudCover's explicit unknown return (vs. defaulting clear/overcast) below 5° solar elevation is the correct failure mode for a value two call sites would otherwise treat as opposite decisions.- Coverage
covers: trueis documented as advisory ("worth asking," upstream stays authoritative) since the model grid doesn't align to a lat/lon box — reasonable given all four declared-box corners were probed live per the description.
Test evidence in the PR body (Go: 11+14+6+pvperf/calibration/PVBand suites, go vet clean; Web: 462/463, the one failure pre-existing/Windows-only) plus a reproducible end-to-end runbook against live SMHI data. Didn't re-run the suite myself, but the diff matches what the description claims.
Safe to merge from my read — the one open item is the human spot-check on the vendored MapLibre bundle's integrity claim noted above.
frahlg
left a comment
There was a problem hiding this comment.
Assessed as the base of the weather/roof stack (#734 → #735 → #826 → #1052). Do not review those three as independent 10k-line PRs; their GitHub diffs against master are this PR plus a small increment.
Go / STRÅNG / coverage — looks right.
- Parameter identification is pinned in comments and tests (121+122=117, 119 caps at 60).
CloudCoverreturns unknown below 5° solar elevation instead of defaulting clear/overcast.ErrOutsideDomainstops the nightly scorer from retrying forever outside the Nordic box.PVBandis in site convention (generation negative);low ≤ base ≤ high ≤ 0and NaN/Inf collapse to a zero-width night band.- Calibration is a median+MAD, refused outside 0.30–1.30 so a misconfigured nameplate cannot silently rescale the forecast.
GET /api/data-sourcesandGET /api/pv/performanceareRead. SQL instate/pvperf.gois parameterized.- Coverage bounds are documented as advisory supersets. Matches what the product needed.
Before merge, two real items:
-
Fold #1051's zone-table lockstep into this PR, then we close #1051. Same
coveragepackage, same/api/data-sources. #1051'sTestEuropeanPriceCountriesMatchZoneTableplusME/UA(and no inventedIE— Ireland is not inprices/zones.go) is the better price-country list. This branch currently hasIEand is missingME/UA. Hugo said the coverage fix lives here; please take that test across so we do not land two registries. -
Browser check of the Weather tab (AGENTS.md). MapLibre vendor swap, coverage panel, Produced overlay. I have not rendered it. The
maplibre-vendor.test.mjspins same-origin + OSMreferrerPolicy; that is not a visual check. Themaplibre-670changeset already says the map was re-verified on 6.7.0 — confirm that still holds on this tip, including WebGL-fail fallback (numeric lat/lon stay authoritative).
Nits, not blocking:
- PR body still says MapLibre 6.0.0; the changeset and vendor test say 6.7.0. Update the body.
.changeset/maplibre-670.mdmentions the building picker and PV array drawing tool, which are #735/#826. Trim that sentence so a 734-only changelog does not claim them.
CI is green. Once (1) is in and (2) has a human look, this is the one to land; #735 waits on it.
…urope The coverage API and settings panel are gone — #734 already owns that surface. What a non-European site actually needed was a price source. price.provider: static takes a flat rate in the install currency, plus optional local-time windows (overnight wrap, optional weekdays). Grid tariff and VAT still apply. Live day-ahead remains Europe-only. Part of #726.
Makes SMHI STRÅNG a real irradiance source rather than an unreferenced client,
gives every external data source a machine-readable statement of where in the
world it works, and moves the location picker off Leaflet onto vendored
MapLibre.
Second half of the split @frahlg asked for — #718 carried the plane-of-array
wiring and has merged. Based on
master; the SI-units branch this stack usedto sit on merged as #962.
STRÅNG's parameters, measured rather than assumed
STRÅNG is documented on SMHI's
model page
and in the
open-data explorer,
with the API documentation at
opendata.smhi.se/metanalys/strang.
The model page names five quantities but not the API's parameter codes, so the
code-to-quantity mapping was pinned against the live API: probing 100–130
returned data for exactly 116–122, and names were confirmed from their
magnitudes on a clear day at Stockholm (2026-06-21):
Two independent checks pin the identification: 121 + 122 = 723.0 + 87.5 =
810.5, exactly 117 (direct + diffuse = global), and 119 caps at exactly 60 —
minutes within the hour.
Cloud cover: not published, but derivable
STRÅNG is a radiation model and has no cloud-cover parameter. It does publish
sunshine duration — minutes per hour with the direct beam above the WMO
120 W/m² threshold — so cloudiness comes out as
1 − minutes/60. That is anobserved quantity rather than an inferred cloud field, coarser in exactly one
direction: thin cirrus that dims without blocking is invisible to it.
Why CloudCover takes a location
Sunshine duration is zero at night for the trivial reason that there is no sun,
and near sunrise and sunset the beam crosses ten or more air masses and cannot
reach 120 W/m² under a spotless sky — the first working version reported 100%
cloud cover at midnight, confidently, every night. So
CloudCover(lat, lon)declines to answer unless the sun clears 5° of elevation at some point in
the hour, and returns an explicit unknown rather than defaulting to clear
or overcast — those lead to opposite decisions at a call site. It samples the
hour's start, midpoint and end, because the hour in which the sun crosses the
line would otherwise be discarded despite carrying a real half-hour of
sunshine.
Coverage metadata (#726)
New
go/internal/coverageregistry andGET /api/data-sources: everyforecast, irradiance and price source declares its area, country list, licence,
whether it needs a key, and whether it reaches this site. The Weather tab
renders it under the map and flags anything that does not apply.
This makes an existing silence explicit — STRÅNG is Nordic-only and every price
provider European, so sites elsewhere were getting empty results with no
explanation:
PV performance scoring now declines to start outside the STRÅNG domain instead
of retrying nightly forever.
Why the bounds are advisory, and why that is not a cop-out
STRÅNG's model grid is rotated relative to lat/lon, so any lat/lon box is a
superset of the real domain — all four corners of the declared box were
probed live and every one returned no data.
covers: falseis thereforedefinitive, while
covers: truemeans "worth asking"; the upstream API staysthe final word. Nothing here is a safety input — it decides what the UI shows
and whether a pointless nightly fetch is skipped.
Also in this PR
GET /api/pv/performance, a dashed "expected (STRÅNG)" overlay on theProduced tile, and a calibration factor fed back into the forward forecast.
The factor is a median with a median-absolute-deviation spread, so one snowy
day cannot move it, and it is refused outside a plausible band — a site
reading at 10% of nameplate is a configuration fault, and silently rescaling
the forecast would hide it.
mpc.PVBand— extracts the site-signed robust band into one testedfunction, discharging the correctness risk flagged in RFC: SMHI STRÅNG irradiance + Lantmäteriet roof geometry for PV prediction, performance scoring & map-drawn arrays #717. Generation is
negative, so
lowis the optimistic bound; the invariantlow ≤ base ≤ high ≤ 0is asserted over a table including NaN and ±Inf.web/vendor/maplibre/per the ship-it-on-the-box policy from fix(web): escape loadpoint HTML and ship Leaflet on the box #910 — theweather tab loads it same-origin and nothing comes from a CDN. The vendored
Leaflet and its test are removed;
web/maplibre-vendor.test.mjspins the newcontract. Same OSM raster tiles and attribution as before, and the numeric
lat/lon fields stay authoritative, so a WebGL failure costs the picker and
nothing else.
What the MapLibre vendor swap carries over
transformRequestreturnsreferrerPolicy: "strict-origin-when-cross-origin",and the vendored build passes it into its
Requests, so volunteer tileservers still see an origin under the box's
Referrer-Policy: no-referrer.(sha256 against the registry metadata; the sha384 SRI pins the old CDN loader
carried match too).
blob:workerdetour.
Testing
Go: 11 coverage tests (the probe results are encoded as assertions, so widening
the box fails them and says why), 14 STRÅNG tests including the night,
polar-night and near-horizon cases, 6 for the new endpoint, plus the pvperf,
calibration and PVBand suites;
go vetclean. Web: 462/463 — the one failureis
theme-tokens.test.mjs, Windows-only and red on a cleanmastertoo.Verified end-to-end against a seeded local instance: performance ratio 0.90,
calibration factor 0.92 (σ 0.033, 21 days, applied), and the live STRÅNG fetch
above.
Part of the roadmap in discussion #717. Closes part of #726. The research
behind the calibration loop — when a simulation-based expectation holds up
against models built purely on site history, and when it does not — is in
discussion #1050.
Seen running
The coverage panel answering for Stockholm — every forecast source, STRÅNG for
irradiance, all three price sources:
And the scoring loop, three minutes after boot, against live SMHI data: the
dashed line is weather-expected production over the seeded actuals, and the
factor (0.677 over 24 scored days, σ_rel 0.25) is applied to the forward
forecast, not just displayed.
Runbook: how this was demoed (reproducible)
go build -o ftw-demo.exe ./cmd/ftwfrom the branch, plusgo run ./cmd/sim-ferroamp.provider: open_meteo, twopv_arrays(7 kW S @ 35°, 3 kW W @ 35°),roofmodel.enabledwithcommand: python,module_dir: roofmodel../ftw-demo.exe -config config.demo.yaml -backfill 30— 518 400 synthetichistory rows from the repo's own generator.
from SMHI's production STRÅNG API, live and scored all 30 days.
screenshots are unretouched viewport captures.
Everything else — every line of core, the module, the UI — was the shipped code
from this branch.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Wepiu9kFxZkLTEZhoTJTx4