[3.15] gh-160268: Fix tokenize on a mismatched closing bracket in an f-string (GH-160272) (#160335)

Since gh-105042, `tokenize` tolerates mismatched brackets. That also let
a mismatched closing bracket pop the brace that opened an f-string
replacement field, which the f-string state relies on. In `f'{]a'`, this
triggered an assertion failure in _PyLexer_scan_string.

Check the replacement field brace in both modes.
(cherry picked from commit 86758d1)

Author: lysnikolaou

Misc/NEWS.d/next/Core_and_Builtins/2026-10-22-17-37-27.gh-issue-160268.gJ2iDx.rst

@@ -0,0 +1,3 @@
+Fix a crash in :mod:`tokenize` when a mismatched closing bracket follows the
+opening brace of an f-string or t-string replacement field, as in ``f'{]a'``.
+It now raises the same "unmatched" error as the parser.

Parser/lexer/lexer.c

@@ -1293,9 +1293,9 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
         if (tok->level > 0) {
             tok->level--;
             int opening = tok->parenstack[tok->level];
-            if (!tok->tok_extra_tokens && !((opening == '(' && c == ')') ||
-                                            (opening == '[' && c == ']') ||
-                                            (opening == '{' && c == '}'))) {
+            if (!((opening == '(' && c == ')') ||
+                  (opening == '[' && c == ']') ||
+                  (opening == '{' && c == '}'))) {
                 /* If the opening bracket belongs to an f-string's expression
                 part (e.g. f"{)}") and the closing bracket is an arbitrary
                 nested expression, then instead of matching a different
@@ -1309,7 +1309,10 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
                             "%c-string: unmatched '%c'", TOK_GET_STRING_PREFIX(tok), c));
                     }
                 }
-                if (tok->parenlinenostack[tok->level] != tok->lineno) {
+                if (tok->tok_extra_tokens) {
+                    /* Allow other mismatched brackets. */
+                }
+                else if (tok->parenlinenostack[tok->level] != tok->lineno) {
                     return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
                             "closing parenthesis '%c' does not match "
                             "opening parenthesis '%c' on line %d",