gh-159811: Raise OpenSSL errors in _hashlib.HMAC after re-attaching the thread state

`hashlib_openssl_HMAC_update_once()` set an exception when the OpenSSL
HMAC update function failed, but it is called inside a GIL-released
region, with the object's mutex held, by `_hashlib.HMAC.update()` and
`_hashlib.hmac_new()` when the data exceeds HASHLIB_GIL_MINSIZE bytes.
Setting the exception with the thread state detached crashed the
interpreter instead of raising an exception.

Only report the failure in `hashlib_openssl_HMAC_update_once()` and set
the exception in `_hmac_update()` once the thread state is re-attached
and the mutex is released, as it was done before gh-134531. The OpenSSL
error queue is per thread, so the exception type and message are
unaffected.

Author: picnixz

Misc/NEWS.d/next/Library/2026-10-19-23-25-29.gh-issue-159811.P6yy0z.rst

@@ -0,0 +1,3 @@
+Fix a crash in :mod:`hmac` when the OpenSSL routine used by
+:meth:`hmac.HMAC.update` or by :func:`hmac.new` fails on data larger
+than 2048 bytes. Patch by Bénédikt Tran.

Modules/_hashopenssl.c

@@ -1976,14 +1976,18 @@ hashlib_HMAC_get_hashlib_digest_name(HMACobject *self)
 #endif
 }
 
+/*
+ * Update the HMAC context 'ctx' with the content of 'v'.
+ *
+ * Return 0 on success and -1 if PY_HMAC_update() failed. No exception
+ * is set, so that this can be called without an attached thread state.
+ */
 static int
 hashlib_openssl_HMAC_update_once(PY_HMAC_CTX_TYPE *ctx, const Py_buffer *v)
 {
-    if (!PY_HMAC_update(ctx, (const unsigned char *)v->buf, (size_t)v->len)) {
-        notify_smart_ssl_error_occurred_in(Py_STRINGIFY(PY_HMAC_update));
-        return -1;
-    }
-    return 0;
+    return PY_HMAC_update(ctx, (const unsigned char *)v->buf, (size_t)v->len)
+        ? 0
+        : -1;
 }
 
 /* Thin wrapper around PY_HMAC_CTX_free that allows to pass a NULL 'ctx'. */
@@ -2243,7 +2247,11 @@ _hmac_update(HMACobject *self, PyObject *obj)
         r = hashlib_openssl_HMAC_update_once(self->ctx, &view)
     );
     PyBuffer_Release(&view);
-    return r;
+    if (r < 0) {
+        notify_smart_ssl_error_occurred_in(Py_STRINGIFY(PY_HMAC_update));
+        return -1;
+    }
+    return 0;
 }
 
 /*[clinic input]