Repository navigation
C API: Detect invalid UCS4 characters in debug mode #158445
Description
Currently, PyUnicode_FromKindAndData() and PyUnicodeWriter_WriteUCS4() create an invalid string if a character is not in the [U+0000; U+10ffff] range. Instead, I propose to raise an exception in this case.
Example creating an invalid string:
$ python3.15
>>> import _testcapi, sys
>>> invalid_ucs4 = (0x11_0000).to_bytes(4, byteorder=sys.byteorder)
>>> s = _testcapi.unicode_fromkindanddata(4, invalid_ucs4*5)
>>> list(s)
SystemError: invalid maximum character passed to PyUnicode_New
>>> s[0]
SystemError: invalid maximum character passed to PyUnicode_New
Linked PRs
- gh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData() #158447
- gh-158445: Detect invalid UCS4 characters in debug mode #158502
- gh-158445: Add C API tests on build information functions #158603
- gh-158445: Skip invalid UCS-4 tests if Python is built with assertions #158604
- gh-158445: Allocate memory in the heap in Py_GetVersion() #158608
- [3.15] gh-158445: Add C API tests on build information functions #159515
- Revert "gh-158445: Allocate memory in the heap in Py_GetVersion() (#158608)" #159519
- [3.15] gh-158445: Detect invalid UCS4 characters in debug mode (#158502) #159524
- [3.14] gh-158445: Add C API tests on build information functions (#159515) #159525
- [3.14] gh-158445: Detect invalid UCS4 characters in debug mode (#158502) (#159524) #159540
Activity
vstinner commentedon Sep 29, 2026
With 32-bit
wchar_t(ex: Unix),PyUnicode_FromWideChar()andPyUnicodeWriter_WriteWideChar()already raise a ValueError if a character is outside the [U+0000; U+10ffff] range.pythongh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData()
pythongh-158445: Reject invalid UCS4 in PyUnicode_FromKindAndData()
vstinner commentedon Sep 29, 2026
Other Python and C functions raising an exception if the code pointer is negative or is greater than 0x10ffff:
chr()raisesValueError.PyUnicode_FromOrdinal(),PyUnicode_Fill()andPyUnicodeWriter_WriteChar()raiseValueError.PyUnicode_FromFormat("%c")raises OverflowError.PyUnicode_New()raisesSystemError.PyUnicode_WriteChar()raises ValueError if the character is greater than the string maximum character.pythongh-158445: Detect invalid UCS4 characters in debug mode
pythongh-158445: Detect invalid UCS4 characters in debug mode
gh-158445: Detect invalid UCS4 characters in debug mode (#158502)
[-]C API: Reject invalid UCS-4 string in PyUnicode_FromKindAndData[/-][+]C API: Detect invalid UCS4 characters in debug mode[/+]vstinner commentedon Sep 30, 2026
Since checking for invalid characters at runtime was rejected, I repurpose the issue to "Detect invalid UCS4 characters in debug mode".
encukou commentedon Oct 2, 2026
This broke several buildbots.
gh-158445: Add C API tests on build information functions (#158603)
vstinner commentedon Oct 2, 2026
Oh, I didn't get buildbot failures report on my PR, so I didn't notice. Thanks for the notification.
So test_capi fails on 2 buildbots: Python is built in release mode with assertion using
./configure (...) CFLAGS=-UNDEBUGI wrote a first PR #158603 to fix
support.built_with_c_assertions()(merged). Previously,-UNDEBUGcompiler flag was not handled properly.I wrote a second PR #158604 to fix test_capi.test_unicode.
gh-158445: Skip invalid UCS-4 tests if Python is built with assertions (
vstinner commentedon Oct 2, 2026
My latest change should fix the 2 buildbots.
encukou commentedon Oct 2, 2026
Thank you!
gh-158445: Allocate memory in the heap in Py_GetVersion() (#158608)
vstinner commentedon Oct 3, 2026
Both workers are back to green.
Revert "gh-158445: Allocate memory in the heap in Py_GetVersion() (#1…
[3.15] gh-158445: Add C API tests on build information functions (#15…
[3.14] gh-158445: Add C API tests on build information functions (#15…
[3.15] gh-158445: Detect invalid UCS4 characters in debug mode (#158502…
[3.14] gh-158445: Detect invalid UCS4 characters in debug mode (#158502…