http4s mtls dev toggle#2876
Open
constantine2nd wants to merge 15 commits into
Open
Conversation
Turning on the dev-only in-process mTLS termination previously meant editing five props. Reduce it to a single switch, and stop the switch from silently breaking local logins. - Http4sMtls: only mtls.enabled is required now. The four store props fall back to the checked-in dev pair (server.jks / server.trust.jks), each fallback logged at WARN naming the prop and its OBP_* environment equivalent, so a default is never silent. A missing store now fails with the resolved absolute path and the working directory in the message -- the defaults are repo-relative, so "launched from the wrong directory" is the realistic failure and a bare FileNotFoundException did not say so. - Http4sMtls.keyStoreTypeOf: load .p12/.pfx as PKCS12 and everything else as JKS, for keystore and truststore alike. The type cannot be sniffed cheaply and guessing wrong fails with an opaque parse error. This makes localhost_san_dns_ip.pfx usable -- the only local cert carrying DNS:localhost + IP:127.0.0.1, where the server.jks default has no SAN at all and relies on clients falling back to CN. - scripts/mtls_env.sh: exports the OBP_MTLS_* overrides. No new machinery was needed -- APIUtil.getPropsValue already reads OBP_<NAME> from the environment ahead of the props file, so every prop is env-overridable. Pre-exported values win. It also pins local_identity_provider. Enabling TLS means hostname must become https://, but hostname is the default for local_identity_provider (constant.scala), which is the provider column every AuthUser / ResourceUser row is keyed on -- moving the scheme alone gives existing local users a different provider string and orphans them. The pin reads the hostname via Lift's own resolution order: Lift maps Development to an EMPTY modeName, so development.default.props is never loaded and default.props is. Reading the wrong file there reintroduces exactly the orphaning this pin prevents. - scripts/java_env.sh: select a JDK >= 17 before any Maven work. The build compiles with -release 17 and a default JDK 11 fails with "'17' is not a valid choice for '-release'", which reads like a source error. An adequate JAVA_HOME is honoured; an inadequate one is replaced with an announcement. - Both build_and_run scripts: --mtls flag following the existing --clean/--online/--no-flush/--background idiom. mtls_env.sh is sourced after the build so it can never affect compilation. - Tests: extension-to-store-type mapping, and buildSslContext loading the .pfx. Verified live: plain HTTP rejected (curl 52), certless handshake rejected under client_auth=need (curl 56), 200 with a real client certificate, and a spoofed PSD2-CERT header replaced by the handshake certificate (1252-char exact match). Both scripts build and boot from a JDK 11 shell.
…ipts Both build_and_run scripts run under `set -e`, and both call mvn without guarding it. A failing build therefore aborted the script at the mvn line, so everything written to handle that failure never ran: - flushall_fast_build_and_run.sh captured BUILD_EXIT_CODE=$? and had a whole "detect incremental compilation cache issue, retry with a clean build" block keyed off it. None of it was reachable. The script exited silently with no diagnosis and no retry. - flushall_build_and_run.sh could not reach its BUILD_EXIT check, so it never printed the log tail it promises on failure. Bracket each mvn invocation with set +e / set -e so the exit code is captured and the existing handling runs. The symptom is easy to misread: the scripts pipe their output, so the exit code a caller sees is the pipeline's, not the script's -- a failed build could look like a successful run that simply produced no server. Verified by putting a failing mvn on PATH and running the real script: it now reports the failure, detects the cache-issue signature, retries with a clean build, tails the log and exits 1.
…sable
With requirePsd2Certificates=ONLINE, any request reaching passesPsd2Aisp
without a TPP-Signature-Certificate header produced
500 OBP-50000: Unknown Error.: Illegal base64 character 2d
getCertificateFromTppSignatureCertificate read the header via getHeaderValue,
which substitutes SecureRandomUtil.csprng.nextLong().toString when a header is
absent. That is a serviceable "never matches" sentinel for the string
comparisons it was written for (Digest, Signature), but the value here goes
straight into Base64.getDecoder: a negative Long renders with a leading '-',
which is 0x2d, and the decoder throws.
The bug was also flaky by construction. Only about half of calls got a negative
Long; the rest base64-decoded to garbage and failed later in getCertificatePem
with a different error. Same missing header, two symptoms.
Make getCertificateFromTppSignatureCertificate return Box[X509Certificate]
instead of throwing, and look the header up directly rather than through
getHeaderValue. Each failure mode is now a Failure: header absent, not base64,
base64 of something that is not a PEM, or a PEM that is not a certificate.
Call sites:
- APIUtil.passesPsd2ServiceProviderCommon: fail closed. passesPsd2ServiceProvider
already maps a Failure to 401 -- this is the path that produced the 500.
- BerlinGroupSigning.verifySignedRequest: checkRequestIsSigned only proves the
header is present, not usable, so an unusable one is a 401.
- BerlinGroupSigning.getOrCreateConsumer: leave the caller's result untouched.
Consumer creation is a side effect of a signed request, not the thing being
authorised; verifySignedRequest is what rejects a bad certificate.
- BerlinGroupCheck: without a certificate there is no serial number to compare
keyId.SN against, so it is an invalid signature header (400).
Found by running a real TPP (OBP-Hola) through the UK Open Banking consent
journey. Verified: the identical POST to
/open-banking/v3.1/account-access-consents now returns 401 with no "Illegal
base64" in the log; RegulatedEntityTest passes.
Note this changes only the failure mode, not the gate: a caller with no
TPP-Signature-Certificate is still rejected, just with the correct status.
…lly uses BEHAVIOUR CHANGE for UK Open Banking endpoints when requirePsd2Certificates=ONLINE. Read the note at the end before upgrading. The PSD2 gate (passesPsd2Aisp / passesPsd2Pisp) identified the TPP from the TPP-Signature-Certificate header regardless of which API standard was being served. That header is Berlin Group NextGenPSD2. UK Open Banking does not use it at all: OBIE identifies the TPP by the mTLS transport certificate, which reaches OBP as PSD2-CERT -- set by the reverse proxy that terminates mTLS, or by bootstrap.http4s.Http4sMtls in development. A correctly behaving OBIE client was therefore rejected for omitting a header its specification never mentions. BerlinGroupCheck.validate already scopes every OTHER piece of Berlin Group machinery to Berlin Group URLs -- the mandatory headers, the request-signature verification, the on-the-fly consumer creation. The PSD2 gate was the single place that crossed that boundary. tppCertificateForStandard brings it in line: UK Open Banking URLs read PSD2-CERT, everything else keeps the previous code path byte for byte. One call site changes, which covers all 10 UK enforcement sites (5 in v3.1.0, 5 in v4.0.1) and any added later. Berlin Group (37 sites) and the OBP-native endpoints (4) are untouched. Only the certificate SOURCE changes, not what is done with it: both branches feed the same regulated-entity lookup, which matches on issuer CN + serial number and works with any X509 certificate. No eIDAS QCStatement is required -- that is only needed by the CERTIFICATE mode, which is unaffected. Verified end to end against a real TPP (OBP-Hola) over mutual TLS: - before registering the entity: 401 OBP-34102 (regulated entity not found), proving the mTLS certificate is now the one being looked up, where previously the same call returned OBP-20306 (no certificate in the request at all); - after registering it: the UK v4.0.1 consent is created, Status AWAITINGAUTHORISATION, with the gate logging the matched entity and PSP_AI; - /berlin-group/v1.3/consents still rejects on missing tpp-signature-certificate, unchanged. RegulatedEntityTest and the mTLS suites pass (13 tests). No test in this repo sends TPP-Signature-Certificate to a UK endpoint -- only the Berlin Group suites reference that header. UPGRADE NOTE. Under eIDAS a TPP holds two different certificates: the QWAC, used for TLS and therefore surfacing as PSD2-CERT, and the QSEAL, used to sign and therefore carried in TPP-Signature-Certificate. They have different serial numbers and often different issuing CAs. Since regulated entities are matched on issuer CN + serial, a deployment that runs UK endpoints with requirePsd2Certificates=ONLINE and onboarded its TPPs by QSEAL will now match a different entity, or none: - no PSD2-CERT present (no mTLS proxy) -> previously working TPPs get 401; - QWAC registered as a different entity -> that entity's services decide the call. This cannot authorise an unregistered party: both paths require a CA + serial match and fail closed. Deployments on requirePsd2Certificates=NONE are unaffected because the gate does not run. Register the QWAC alongside the QSEAL to migrate.
Two cosmetic loose ends from 3c863d3, no behaviour change. - APIUtil imported getCertificateFromTppSignatureCertificate for a call site that now qualifies it through BerlinGroupSigning, leaving the import as the only reference to a name the file no longer uses. - BerlinGroupSigning.getOrCreateConsumer gained a `case Full(certificate) =>` whose body was left at the indentation of the `else` block it came from, so ~55 lines sat one level out from the case that owns them and read as if they followed the match rather than belonging to a branch of it. Verified with a full compile.
Not implemented — a proposal, marked as such at the top of the document and at both links to it. Production terminates mutual TLS at nginx and forwards the TPP certificate to OBP as the PSD2-CERT header over a plain HTTP hop. The proposal is to run mutual TLS on that hop too, so OBP sees a certificate for nginx as well as the forwarded one for the App. The question it raises — OBP now knows two parties by certificate, is that two Consumers? — is answered no: the App certificate is an identity we authorise on, the proxy certificate is a trust decision made once and discarded. Consumer is what rate limiting, metering, entitlements and consent binding key on, so admitting the proxy as one would make every request appear to come from a single Consumer. Four deployments are in scope and all are treated as legitimate: OBP as the TLS edge or behind nginx, in development or in production. They collapse onto one code path by asking, per request, whether the TLS peer is the caller or a forwarder trusted to name the caller — the X-Forwarded-For trust model applied to certificates. The distinguishing config is one axis, an allowlist that is empty for edge deployments, so run mode stops being a topology axis and the Development gate on the existing middleware has nothing left to encode. Two findings worth the reader's attention: - The PSD2-CERT path performs no chain or revocation validation. CertificateVerifier (PKIX + CRL) is reached only from the Berlin Group signature path, and JSSE disables revocation by default, so a deployment where OBP is the public TLS edge would accept a revoked but unexpired TPP certificate. nginx does this check today; taking the edge takes the obligation with it. Established by call site, not yet by test. - The same certificate arrives in three encodings depending on who terminated TLS, and three separate places downstream compensate ad hoc. Normalising once on ingress makes each an exact comparison and removes the encoding bugs that only appear in production. Linked from MTLS_DEV_MODE.md at the production-deployment rule that motivates it and from that document's file table.
Section 11. Five sequenced changes, still a proposal — no code accompanies it. Two findings drive the sequencing. First, the certificate middleware is wired only when mtls.enabled, so in the topology production actually runs today — nginx over a plain HTTP hop — no certificate middleware runs at all. The work cannot extend the existing wrapper in place; the resolution middleware has to wrap httpApp unconditionally, with the TLS branch only supplying the peer certificate. Second, expressing the peer-vs-caller rule as a pure function makes the decision table a unit test with no server and no TLS, which is what keeps the later phases cheap. The first two PRs are behaviour-preserving in every existing deployment: an empty forwarder allowlist reproduces dev-as-edge exactly, and the legacy plain-hop prop defaults to what production does today. All of the actual change is deferred to per-environment configuration, gated on the logs showing requests resolving as forwarded before the insecure default is switched off anywhere. Records the one genuine regression path found while planning: normalising the PSD2-CERT header at ingress would break Consumers registered with a non-canonical PEM, which today match through the raw-value lookup that runs first. Both sides of the comparison have to be normalised, which is strictly more permissive than today. Noted explicitly because moving the normalisation and deleting the fallback looks equivalent and is not. Three decisions are called out as needing an answer before the main PR starts, one of them — how a forwarder is identified — because it is the only choice that is hard to reverse once operators have deployed the config format.
First phase of docs/MTLS_TOPOLOGIES.md (§11.2). No behaviour change is intended for any certificate that authenticates today; what changes is that several more now do. The same certificate reaches OBP in whichever encoding the deployment's TLS terminator happens to produce: nginx percent-encodes $ssl_client_escaped_cert, HAProxy rebuilds a single-line PEM, the dev-mode in-process terminator injects canonical PEM, a hand-built client may send bare base64. Everything downstream compares certificates as strings -- the Consumer lookup, the consent/Consumer match, the PSD2 regulated-entity gate -- so each encoding behaves like a different certificate. That is the mechanism behind "it works in development and fails in production", and it is why the proxy documentation had to require a decoding step before OBP would accept nginx's output at all. Psd2CertIngress parses the header once and rewrites it to one canonical form, so those comparisons become exact. nginx's percent-encoded form is now understood directly and the njs step is no longer needed. A header that is not a parseable certificate is passed through untouched: normalisation is not authentication, and rejecting here would move an access-control decision into a formatting concern and take the error code away from the layer that owns it. Wired unconditionally into Http4sApp.httpApp rather than into Http4sServer's mtls.enabled branch. The deployment that needs this most -- a proxy forwarding the header over a plain HTTP hop -- runs no TLS middleware at all, so the existing branch could never have covered it. Http4sMtls.toPem now delegates to CertificateUtil.toPem so an injected certificate and a forwarded one are byte-identical by construction rather than by coincidence. Percent-decoding is hand-rolled because java.net.URLDecoder maps '+' to a space. That is correct for form encoding and silently corrupts a base64 payload; the test asserts a certificate carrying a '+' survives. The plan called for deleting the ad-hoc normalisations this makes redundant. Kept instead: the Consumer lookup reads the database, so only the request side can be normalised -- the stored value is whatever was pasted at registration. The consent check's removeBreakLines comparison is therefore now an alternative to comparePemX509Certificates rather than a replacement, since neither subsumes the other for every stored form and accepting either cannot break a Consumer that matched before. Doing this properly means normalising stored certificates at write time plus a migration, which is its own change. Tests: the encoding table, the '+' case, non-certificates, middleware pass- through, and the regression this had to be designed around -- a Consumer registered with a single-line PEM still matching once the header is canonical. 17 tests in the mTLS suites, plus 54 in the consent, regulated-entity and consumer-resolution suites, all pass.
Taken 2026-07-23. Written into the sections the code will be built against rather than kept as an answered-questions list, so PR 2 has one place to read. - A forwarder is recognised by issuer CA + subject DN. §5.1 now carries the concrete prop format: indexed pairs, because a DN contains commas and any comma-separated list of DNs is ambiguous the first time someone writes a real one. Records the two things the format depends on — canonical X500Principal comparison does not reorder RDNs, so a differently-ordered DN silently fails to match, and a rejected peer must log its canonical issuer and subject, because the failure mode is every TPP behind that proxy going anonymous at once and it should be one log line to diagnose. - trust_forwarded_header_without_tls defaults to true, preserving current behaviour, with a boot warning so the permissive state is noisy rather than silent. A false default was rejected: it would need every deployment to set the prop in the same release, and anything missed fails closed, which means TPP traffic stops. - prod-as-edge was enumerated for completeness. Nobody is asking for it and the nginx to OBP connection is already secured by other means, so PR 5 is not scheduled and the revocation gap stays documented rather than blocking. Kept in the document, with the condition that reopens it stated: any deployment making OBP the public TLS edge. - Stored certificates are not normalised at write time. PR 1 normalises the request side; the fallbacks on the stored side stay. Two open questions are resolved by decision 3 and struck through with their answers. One is added in their place: "already secured by other means" needs to be pinned down, because a hop carrying TLS without client authentication puts that deployment on the fourth row of the §3 table rather than the fifth, where trust_forwarded_header_without_tls — which keys off no TLS at all — is not the prop that governs it.
…orwarder
Second phase of docs/MTLS_TOPOLOGIES.md (§11.3). Behaviour-preserving in every
existing deployment: the shipped defaults — an empty forwarder allowlist and
mtls.trust_forwarded_header_without_tls=true — reproduce exactly what
development-as-edge and production-behind-a-plain-hop did before. All of the
change is deferred to per-environment configuration (§11.5).
OBP terminates TLS in some deployments and sits behind a proxy that terminates
it in others. These look like different problems only while the question is
"which deployment is this". The question that unifies them is asked per request:
is my TLS peer the caller, or a forwarder I trust to tell me who the caller is?
That is the X-Forwarded-For trust model applied to certificates, and it collapses
development-as-edge and production-behind-nginx onto one code path distinguished
by one axis — whether the allowlist is empty. There is no topology mode to keep
in step with the configuration, because the topology is inferred from it.
PeerTrust.resolve is deliberately pure and free of http4s: the branch that
matters most in production, a forwarded header arriving over a hop with no
client certificate, is otherwise the hardest one to exercise. CallerCertificate
is the shell that connects it to a request, and it replaces
Http4sMtls.injectClientCertificate, which implemented one branch of the rule —
"overwrite the header with the handshake certificate" — as though it were the
whole of it. Enabled in production unchanged, that would have replaced the App's
certificate with nginx's and made every request arrive as the proxy; a test now
pins it.
Wired unconditionally into Http4sApp.httpApp rather than Http4sServer's
mtls.enabled branch, for the reason PR 1 had to move too: the deployment where
the answer is least obvious runs no TLS middleware at all.
The run-mode gate is gone. It encoded an assumption about deployment that all
four supported topologies falsify, and terminating mTLS in production is now a
supported one. What it was really standing in for — the dev keypair in this
public repository reaching a production server — is now checked directly, by
digest, so the store is recognised wherever it has been copied to. A test
asserts the digests still match the files, so regenerating the dev pair fails
the build rather than silently disarming the check.
Two things surfaced while building it, both now pinned by tests:
- The empty string is a valid X.500 name. X500Principal("") parses and
canonicalises to "", so a blank trusted_proxy subject would have become a rule
matching any certificate with an empty subject rather than a configuration
error.
- An HTTP header value cannot contain newlines, so a proxy can only ever forward
a single-line PEM — the client rejects canonical multi-line PEM before it
reaches the wire. This is what makes PR 1's ingress canonicalisation
load-bearing, and it is why its regression analysis holds: the header side of
every stored-certificate comparison was always single-line.
Deviation from the plan: the sketched ADT's fourth case, Rejected, is not
implemented because nothing produces it. A peer that is not a trusted forwarder
is simply the caller, and an unusable certificate belongs to the authorisation
layer to reject. Three cases, no dead branch.
Tests: the §3 decision table row by row without a server, the middleware wiring,
the production topology over a real handshake, and the dev-store digest guard.
38 in the mTLS suites; 106 across the consent, regulated-entity,
consumer-resolution and DirectLogin suites; 114 in Http4s700RoutesTest as a
whole-chain smoke test, since this middleware now runs for every request.
The mode is honoured in every run mode since the run-mode gate was replaced by the dev-keystore digest check. The documented boot log was updated with it; this line was not, leaving the server telling operators the opposite of the manual.
The quick start opened by telling the reader to generate a keypair, a client certificate and a truststore, on the grounds that the repository's truststore "contains no client certificate you own a private key for". That has not been true since localhost_san_dns_ip.pfx was added: server.trust.jks trusts CN=TESOBE CA (alias mykey, CA:TRUE), and the .pfx is a client identity signed by exactly that CA with its private key included. Keystore, truststore and a usable client identity have been lining up out of the box, and the note said otherwise. Adds step 0: start the server, unpack the .pfx into a PEM pair, make the call. Both commands were run against a live mTLS server from the repository root and returned 200 and curl exit 56 respectively, which is also what the note's replacement claims. Generating certificates is still documented, now with the reasons to bother: a specific subject for a Consumer or regulated-entity lookup, or a second identity to test rejection with. The remaining real limitation of the checked-in pair is recorded where it belongs -- the server certificate has no SAN, so either use -k or point mtls.keystore.path at the .pfx, which carries DNS:localhost and IP:127.0.0.1.
Standalone: no production code changes, and the props still default to the old server.jks / server.trust.jks pair, which keeps working. The checked-in certificates had drifted into naming nothing in particular. The identity used as the calling TPP is localhost_san_dns_ip.pfx, subject CN=localhost -- a host name for a party, and the same CN the server itself presents, so a log line saying the caller is CN=localhost reads as the server talking to itself. The CA that makes it all work hides behind keytool's default alias "mykey", which is how a certificate two commits ago was diagnosed as untrusted when it was not. Of the seven entries in server.trust.jks five have expired -- in 2021, 2022 and 2023 -- and two are public web CAs, which in a client-auth truststore means anything Let's Encrypt ever signed would authenticate as a caller. Expired, so inert; still not a file to copy toward production. This matters more since mtls.trusted_proxy.N.subject made a DN into a configuration value an operator reads and types. Three certificates all called CN=localhost cannot express "the peer is the proxy, the header names the App", which is exactly what the dev-behind-nginx harness has to demonstrate. So: one CA, and one identity per role with the role in the filename, the alias and the DN -- obp-server (serverAuth, SAN DNS:localhost + IP:127.0.0.1), tpp-client (clientAuth, O=Example TPP Ltd so the App is a different organisation from the bank), proxy-client (the forwarder the harness needs), and expired-tpp, which is expired on purpose because certificate expiry had no coverage at all. The truststore holds the CA and nothing else. PKCS12 throughout, PEM alongside, since Java wants one and curl and nginx want the other. scripts/generate_dev_certs.sh regenerates the set, because fixtures nobody can rebuild are how the old ones ended up expired. The CA private key is deliberately not kept: everything is reproducible from the script, and a signing key is the one thing worth withholding even from a set that is public by design. DevCertificateSetTest asserts the properties over a real handshake -- the TPP identity authenticating, the expired one rejected, a certless handshake rejected, the proxy identity forwarding the App's certificate, and the server certificate verifying by hostname with no verifier override, which is what makes the documented curl work without -k. It also pins the subjects, because docs/MTLS_TOPOLOGIES.md shows operators typing them. .gitignore excluded obp-api/src/test/resources/** wholesale; the four existing certificates are tracked only because they predate the rule. Without an exception the regenerated set would have been silently untracked and every machine except the one that ran the script would fail these tests.
…st it Reverses one decision from the previous commit and regenerates the set. The CA key was withheld on the reasoning that everything is reproducible from the script, so a signing key need not be committed even in a set that is public by design. That holds within this repository and fails across two: OBP-Hola is the TPP that calls this API over mutual TLS, and its client certificate has to be one this server's truststore accepts. With the truststore holding only CN=OBP Dev CA, and no key for that CA anywhere, nothing could mint such a certificate. "Regenerate the whole set from a shared script" is not a workable substitute when the two halves live in different repositories. Keeping the key also matches the arrangement being modelled: the TPP holds its own private key and the bank's CA signs the certificate. OBP-Hola generates a keypair and signs against this CA, rather than being handed an identity someone else generated. The key is safe to commit only because it cannot matter: it is named OBP Dev CA wherever it appears, and Http4sMtls refuses to boot a Production server on these stores at all. That guard is what makes the whole set publishable, and it is worth not weakening.
The Scala boot log dropped '(dev-only)' when the run-mode gate was replaced by the dev-keystore digest check, but the three shell messages around it did not, so starting a server printed the old claim twice on the way to printing the new one.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.