tokenize crashes on a mismatched closing bracket in an f-string replacement field #160268

Bug report

The following snippets crashes with an assertion error on the debug build:

import io, tokenize
list(tokenize.generate_tokens(io.StringIO("f'{]a'").readline))

The ] closes the replacement field's { without an error, because tokenize tolerates mismatched brackets (gh-105042). The f-string state still expects that { on the bracket stack. Release builds don't crash; they read a stale stack entry instead.

The parser rejects the same input with SyntaxError: f-string: unmatched ']', and tokenize mode should do the same.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-160268: Fix tokenize on a mismatched closing bracket in an f-string #160272
• [3.15] gh-160268: Fix tokenize on a mismatched closing bracket in an f-string (GH-160272) #160335
• [3.14] gh-160268: Fix tokenize on a mismatched closing bracket in an f-string (GH-160272) #160336