Skip to content

feat(weather): add STRÅNG as an irradiance source and declare source coverage - #734

Open
HuggeK wants to merge 6 commits into
srcfl:masterfrom
HuggeK:strang-source
Open

feat(weather): add STRÅNG as an irradiance source and declare source coverage#734
HuggeK wants to merge 6 commits into
srcfl:masterfrom
HuggeK:strang-source

Conversation

@HuggeK

@HuggeK HuggeK commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Makes SMHI STRÅNG a real irradiance source rather than an unreferenced client,
gives every external data source a machine-readable statement of where in the
world it works
, and moves the location picker off Leaflet onto vendored
MapLibre.

Second half of the split @frahlg asked for — #718 carried the plane-of-array
wiring and has merged. Based on master; the SI-units branch this stack used
to sit on merged as #962.

STRÅNG's parameters, measured rather than assumed

STRÅNG is documented on SMHI's
model page
and in the
open-data explorer,
with the API documentation at
opendata.smhi.se/metanalys/strang.
The model page names five quantities but not the API's parameter codes, so the
code-to-quantity mapping was pinned against the live API: probing 100–130
returned data for exactly 116–122, and names were confirmed from their
magnitudes on a clear day at Stockholm (2026-06-21):

Code Quantity Noon value
116 CIE-weighted UV (mW/m²) 146.6
117 Global horizontal (GHI) 810.5
118 Direct normal (DNI) 917.2
119 Sunshine duration (min/h) 60.0
120 PAR 357.9
121 Direct horizontal 723.0
122 Diffuse (DHI) 87.5

Two independent checks pin the identification: 121 + 122 = 723.0 + 87.5 =
810.5
, exactly 117 (direct + diffuse = global), and 119 caps at exactly 60 —
minutes within the hour.

Cloud cover: not published, but derivable

STRÅNG is a radiation model and has no cloud-cover parameter. It does publish
sunshine duration — minutes per hour with the direct beam above the WMO
120 W/m² threshold — so cloudiness comes out as 1 − minutes/60. That is an
observed quantity rather than an inferred cloud field, coarser in exactly one
direction: thin cirrus that dims without blocking is invisible to it.

Why CloudCover takes a location

Sunshine duration is zero at night for the trivial reason that there is no sun,
and near sunrise and sunset the beam crosses ten or more air masses and cannot
reach 120 W/m² under a spotless sky — the first working version reported 100%
cloud cover at midnight, confidently, every night. So CloudCover(lat, lon)
declines to answer unless the sun clears 5° of elevation at some point in
the hour, and returns an explicit unknown rather than defaulting to clear
or overcast — those lead to opposite decisions at a call site. It samples the
hour's start, midpoint and end, because the hour in which the sun crosses the
line would otherwise be discarded despite carrying a real half-hour of
sunshine.

Coverage metadata (#726)

New go/internal/coverage registry and GET /api/data-sources: every
forecast, irradiance and price source declares its area, country list, licence,
whether it needs a key, and whether it reaches this site. The Weather tab
renders it under the map and flags anything that does not apply.

This makes an existing silence explicit — STRÅNG is Nordic-only and every price
provider European, so sites elsewhere were getting empty results with no
explanation:

Stockholm    forecast: all 4 OK | irradiance: OK STRÅNG | price: all 3 OK
Berlin       forecast: all 4 OK | irradiance: NO STRÅNG | price: OK ENTSO-E, NO Elpriset
Sydney       forecast: all 4 OK | irradiance: NO STRÅNG | price: none

PV performance scoring now declines to start outside the STRÅNG domain instead
of retrying nightly forever.

Why the bounds are advisory, and why that is not a cop-out

STRÅNG's model grid is rotated relative to lat/lon, so any lat/lon box is a
superset of the real domain — all four corners of the declared box were
probed live and every one returned no data. covers: false is therefore
definitive, while covers: true means "worth asking"; the upstream API stays
the final word. Nothing here is a safety input — it decides what the UI shows
and whether a pointless nightly fetch is skipped.

Also in this PR

  • PV performance scoring + calibration — nightly backfill,
    GET /api/pv/performance, a dashed "expected (STRÅNG)" overlay on the
    Produced tile, and a calibration factor fed back into the forward forecast.
    The factor is a median with a median-absolute-deviation spread, so one snowy
    day cannot move it, and it is refused outside a plausible band — a site
    reading at 10% of nameplate is a configuration fault, and silently rescaling
    the forecast would hide it.
  • mpc.PVBand — extracts the site-signed robust band into one tested
    function, discharging the correctness risk flagged in RFC: SMHI STRÅNG irradiance + Lantmäteriet roof geometry for PV prediction, performance scoring & map-drawn arrays #717. Generation is
    negative, so low is the optimistic bound; the invariant
    low ≤ base ≤ high ≤ 0 is asserted over a table including NaN and ±Inf.
  • MapLibre GL JS 6.0.0 replaces Leaflet, vendored under
    web/vendor/maplibre/
    per the ship-it-on-the-box policy from fix(web): escape loadpoint HTML and ship Leaflet on the box #910 — the
    weather tab loads it same-origin and nothing comes from a CDN. The vendored
    Leaflet and its test are removed; web/maplibre-vendor.test.mjs pins the new
    contract. Same OSM raster tiles and attribution as before, and the numeric
    lat/lon fields stay authoritative, so a WebGL failure costs the picker and
    nothing else.
What the MapLibre vendor swap carries over
  • The OSM tile referrer opt-in from the Leaflet fix is ported: MapLibre's
    transformRequest returns referrerPolicy: "strict-origin-when-cross-origin",
    and the vendored build passes it into its Requests, so volunteer tile
    servers still see an origin under the box's Referrer-Policy: no-referrer.
  • All vendored files verified byte-identical to the npm-published copies
    (sha256 against the registry metadata; the sha384 SRI pins the old CDN loader
    carried match too).
  • Same-origin serving also removes MapLibre's cross-origin blob: worker
    detour.

Testing

Go: 11 coverage tests (the probe results are encoded as assertions, so widening
the box fails them and says why), 14 STRÅNG tests including the night,
polar-night and near-horizon cases, 6 for the new endpoint, plus the pvperf,
calibration and PVBand suites; go vet clean. Web: 462/463 — the one failure
is theme-tokens.test.mjs, Windows-only and red on a clean master too.

Verified end-to-end against a seeded local instance: performance ratio 0.90,
calibration factor 0.92 (σ 0.033, 21 days, applied), and the live STRÅNG fetch
above.

Part of the roadmap in discussion #717. Closes part of #726. The research
behind the calibration loop — when a simulation-based expectation holds up
against models built purely on site history, and when it does not — is in
discussion #1050.

Seen running

The coverage panel answering for Stockholm — every forecast source, STRÅNG for
irradiance, all three price sources:

Weather tab with the data-sources coverage panel

And the scoring loop, three minutes after boot, against live SMHI data: the
dashed line is weather-expected production over the seeded actuals, and the
factor (0.677 over 24 scored days, σ_rel 0.25) is applied to the forward
forecast, not just displayed.

Produced tile with the dashed expected-from-STRÅNG overlay

Runbook: how this was demoed (reproducible)
  1. go build -o ftw-demo.exe ./cmd/ftw from the branch, plus
    go run ./cmd/sim-ferroamp.
  2. A demo config: Stockholm (59.3293, 18.0686), provider: open_meteo, two
    pv_arrays (7 kW S @ 35°, 3 kW W @ 35°), roofmodel.enabled with
    command: python, module_dir: roofmodel.
  3. ./ftw-demo.exe -config config.demo.yaml -backfill 30 — 518 400 synthetic
    history rows from the repo's own generator.
  4. Launch, and let the pvperf boot timer fire: it fetched 745 hourly rows
    from SMHI's production STRÅNG API, live
    and scored all 30 days.
  5. Driven with headless Edge over the DevTools protocol — nothing installed;
    screenshots are unretouched viewport captures.

Everything else — every line of core, the module, the UI — was the shipped code
from this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wepiu9kFxZkLTEZhoTJTx4

@frahlg

frahlg commented Aug 19, 2026

Copy link
Copy Markdown
Member

Rebased onto #962 (si-core-units) so STRÅNG lands on watts and 0–1 fractions.

Conflict resolution:

  • Weather tab keeps MapLibre from current master, plus this PR's coverage panel.
  • Forecast tests keep the SI nameplate/Björn cases and the STRÅNG calibration hook (RatedW: 10000, not KWp: 10).
  • Routes keep current auth tiers; added GET /api/pv/performance and GET /api/data-sources as Read.
  • pvperf.Array stores RatedW (watts). Expected energy uses units.PVFromIrradiance.

Force-pushed HuggeK strang-source (92372869). Base is now this SI PR. Still draft.

claude and others added 3 commits September 2, 2026 12:28
…coverage

STRANG becomes a real irradiance source rather than an unreferenced client,
every external data source now says where in the world it works, and the
location picker moves off Leaflet.

STRANG's parameter set was mapped against the live API because SMHI's apidocs
pages 404: exactly 116-122 exist. Identification was confirmed by physics
rather than by guessing - at solar noon 121 + 122 = 723.0 + 87.5 = 810.5,
exactly parameter 117, and 119 caps at 60, i.e. minutes within the hour.

STRANG publishes no cloud cover; it is a radiation model. Cloudiness is
instead derived from sunshine duration as 1 - minutes/60, which is observed
rather than inferred but coarser: blind to thin cirrus, undefined at night.
CloudCover() therefore returns an explicit unknown instead of defaulting to
clear, because those two lead to opposite decisions.

The new coverage registry makes an existing silence explicit. STRANG is
Nordic-only and every price provider is European, so sites elsewhere were
getting empty results with no explanation (srcfl#726). GET /api/data-sources now
reports area, countries, licence and whether each source reaches this site,
and the Weather tab renders it under the map. Bounds are advisory: STRANG's
grid is rotated, so a lat/lon box can only ever be a superset - all four
in-box corners were probed and returned no data. False is definitive, true
means worth trying. Scoring now declines to start outside the domain instead
of retrying nightly forever.

Stacked on srcfl#718, which carries the plane-of-array wiring this builds on.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>
…a CDN

Upstream ships Leaflet on the box (srcfl#910) for a reason that applies just as
hard to its replacement: the box UI must not execute third-party JS from a
CDN, and the map must load when the gateway cannot reach the internet. So
MapLibre GL JS 6.0.0 moves into web/vendor/maplibre — the ESM entry, its
code-split chunk, its worker, the stylesheet and the license — and the
loader imports it same-origin, which also spares MapLibre its blob: worker
detour. The now-unused vendored Leaflet goes away with its test;
maplibre-vendor.test.mjs pins the new contract, including the per-request
referrerPolicy opt-in that keeps OSM volunteer tile servers from 403ing
under the box's Referrer-Policy: no-referrer.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
claude and others added 2 commits September 2, 2026 14:30
The package comment linked the legacy strang.smhi.se site and claimed the
apidocs 404 — they live at opendata.smhi.se/metanalys/strang, with the model
described under smhi.se/forskning. The empirical parameter-code confirmation
stays: the docs name the quantities, not the code-to-quantity mapping.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
http.ServeFile resolves Content-Type through the operating system's MIME
table — the registry on Windows — and a host that maps .mjs to text/plain
does not merely mislabel the vendored MapLibre entry: the app sends
X-Content-Type-Options: nosniff, so the browser is required to refuse the
module, and the weather map dies with 'failed to fetch dynamically imported
module'. Seen live on a Windows host serving the demo build.

handleStatic now pins the Content-Type for every asset kind the web tree
ships (.html .css .js .mjs .svg .png .jpg), so nothing depends on host MIME
state. Covered by a test that would have caught this wherever it ran.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>
frahlg added a commit that referenced this pull request Sep 2, 2026
… chart

Price-driven planning is Europe-only and nothing said so — a site outside
ENTSO-E got an empty price curve with no explanation (#726).

- go/internal/coverage is the registry: every forecast and price source
  declares its area, country list, licence, and a lat/lon box.
- GET /api/data-sources answers for this site, or for a pin still being
  dragged. covers:false is definitive; covers:true means worth trying.
- The Weather tab renders that under the map. The Price tab flags a
  location outside every European bidding zone.
- docs/data-coverage.md is the prose; the registry is the source of truth.

STRÅNG scoring and roof geometry are not shipped, so they are not
advertised. #734 already carries a later version of this plus STRÅNG
and should rebase onto this when it lands.

Part of #726.
@HuggeK
HuggeK marked this pull request as ready for review September 2, 2026 16:46
Same four-file code-split layout as 6.0.0, from the maplibre-gl npm
package's dist/. The entry still imports its shared and worker chunks by
the vendored names and the shared chunk still honours the per-request
referrerPolicy the OSM tiles depend on. Map mount, building picker and
the TerraDraw PV drawing tool re-verified in headless Edge on the new
build; served entry hash matches the tarball byte for byte.

Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com>

@miravoss26 miravoss26 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Large PR (4.4k additions across STRÅNG irradiance, coverage metadata, PV performance calibration, and a Leaflet→MapLibre vendor swap) — reviewed in full, security screen + correctness pass.

Security screen:

  • New external destination: opendata-download-metanalys.smhi.se (SMHI's public STRÅNG API) — matches the PR's stated purpose, no auth/keys needed, read-only.
  • New SQL (pv_performance_daily load/save in state/store.go) uses parameterized queries (WHERE day = ?, db.Exec(q, ...)) — no string-built SQL.
  • New endpoints GET /api/data-sources and GET /api/pv/performance are registered through the existing s.handle(method, Read, handler) permission wrapper, same tier as neighboring reads — no new/weakened auth pattern.
  • MapLibre is vendored same-origin under web/vendor/maplibre/ (no CDN), which reduces surface vs the Leaflet-from-CDN baseline; the referrer-policy opt-in for OSM tiles is carried over. I can't independently re-verify the claimed sha256-against-npm-registry match from here — worth a human spot-check before merge given it's an 806-line minified bundle, but nothing in the vendor README/LICENSE diff itself is suspicious.
  • No secrets, no exec/eval, no obviously new authz surface elsewhere in the diff.

Correctness:

  • PVBand's low ≤ base ≤ high ≤ 0 invariant is asserted over a table including NaN/±Inf — the right shape for a robust-band extraction.
  • Calibration factor is refused outside a plausible band rather than silently rescaling a misconfigured site's forecast — good failure mode (config fault surfaces instead of getting hidden).
  • STRÅNG parameter-code identification is cross-checked two independent ways (121+122=117, 119 caps at 60) rather than taken from the model page alone — solid provenance for a code mapping SMHI doesn't publish directly.
  • CloudCover's explicit unknown return (vs. defaulting clear/overcast) below 5° solar elevation is the correct failure mode for a value two call sites would otherwise treat as opposite decisions.
  • Coverage covers: true is documented as advisory ("worth asking," upstream stays authoritative) since the model grid doesn't align to a lat/lon box — reasonable given all four declared-box corners were probed live per the description.

Test evidence in the PR body (Go: 11+14+6+pvperf/calibration/PVBand suites, go vet clean; Web: 462/463, the one failure pre-existing/Windows-only) plus a reproducible end-to-end runbook against live SMHI data. Didn't re-run the suite myself, but the diff matches what the description claims.

Safe to merge from my read — the one open item is the human spot-check on the vendored MapLibre bundle's integrity claim noted above.

@frahlg frahlg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessed as the base of the weather/roof stack (#734#735#826#1052). Do not review those three as independent 10k-line PRs; their GitHub diffs against master are this PR plus a small increment.

Go / STRÅNG / coverage — looks right.

  • Parameter identification is pinned in comments and tests (121+122=117, 119 caps at 60).
  • CloudCover returns unknown below 5° solar elevation instead of defaulting clear/overcast.
  • ErrOutsideDomain stops the nightly scorer from retrying forever outside the Nordic box.
  • PVBand is in site convention (generation negative); low ≤ base ≤ high ≤ 0 and NaN/Inf collapse to a zero-width night band.
  • Calibration is a median+MAD, refused outside 0.30–1.30 so a misconfigured nameplate cannot silently rescale the forecast.
  • GET /api/data-sources and GET /api/pv/performance are Read. SQL in state/pvperf.go is parameterized.
  • Coverage bounds are documented as advisory supersets. Matches what the product needed.

Before merge, two real items:

  1. Fold #1051's zone-table lockstep into this PR, then we close #1051. Same coverage package, same /api/data-sources. #1051's TestEuropeanPriceCountriesMatchZoneTable plus ME/UA (and no invented IE — Ireland is not in prices/zones.go) is the better price-country list. This branch currently has IE and is missing ME/UA. Hugo said the coverage fix lives here; please take that test across so we do not land two registries.

  2. Browser check of the Weather tab (AGENTS.md). MapLibre vendor swap, coverage panel, Produced overlay. I have not rendered it. The maplibre-vendor.test.mjs pins same-origin + OSM referrerPolicy; that is not a visual check. The maplibre-670 changeset already says the map was re-verified on 6.7.0 — confirm that still holds on this tip, including WebGL-fail fallback (numeric lat/lon stay authoritative).

Nits, not blocking:

  • PR body still says MapLibre 6.0.0; the changeset and vendor test say 6.7.0. Update the body.
  • .changeset/maplibre-670.md mentions the building picker and PV array drawing tool, which are #735/#826. Trim that sentence so a 734-only changelog does not claim them.

CI is green. Once (1) is in and (2) has a human look, this is the one to land; #735 waits on it.

frahlg added a commit that referenced this pull request Sep 3, 2026
…urope

The coverage API and settings panel are gone — #734 already owns that
surface. What a non-European site actually needed was a price source.

price.provider: static takes a flat rate in the install currency, plus
optional local-time windows (overnight wrap, optional weekdays). Grid
tariff and VAT still apply. Live day-ahead remains Europe-only.

Part of #726.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants