Skip to content

Avoid removed Py_FileSystemDefaultEncoding in load_library - #287

Open
ozeranskii wants to merge 2 commits into
python-cffi:mainfrom
ozeranskii:fix-py316-fsdefaultencoding
Open

ozeranskii wants to merge 2 commits into
python-cffi:mainfrom
ozeranskii:fix-py316-fsdefaultencoding

Conversation

@ozeranskii

@ozeranskii ozeranskii commented Oct 18, 2026 •

Copy link
Copy Markdown

CPython 3.16 removed Py_FileSystemDefaultEncoding from the headers (python/cpython#158020), so _cffi_backend.c no longer compiles there:

src/c/_cffi_backend.c:4541:22: error: use of undeclared identifier 'Py_FileSystemDefaultEncoding'

This replaces the "et" conversion in b_do_dlopen with "O&" + PyUnicode_FSConverter. That converter encodes with the interpreter's filesystem encoding, rejects embedded null characters, releases its result if a later argument fails to parse, and has the same behaviour on every supported version (3.10+), so no version checks are needed. The None, cdata and Windows wide-char paths are unchanged.

PyUnicode_EncodeFSDefault, as suggested in #86, would be shorter. It does not check for embedded nulls, though, so load_library("/usr/lib/libc.so.6\0x") would silently load the truncated path. Today that raises TypeError; with this change it raises ValueError: embedded null byte, like open(). The new test covers this and the existing rejection of bytes. Non-ASCII file names are already covered by testing/cffi1/test_re_python.py and testing/cffi0/test_ownlib.py.

Addresses the Py_FileSystemDefaultEncoding part of #180 (_PyLong_Sign remains). Refs #86.

Tested on macOS arm64 with python setup.py build_ext -i && pytest src/c/test_c.py on 3.10 through 3.15t (unchanged: 225 passed, 6 skipped) and on 3.16.0a1 / 3.16.0a1t (previously failed to build, now 225 passed, 6 skipped). I have not tested on Windows, where a str goes through the separate dlopenWinW branch, which this does not touch. That branch already truncates at an embedded null, so the test skips that check on win32. This PR does not add 3.16 to CI, since neither cibuildwheel nor setup-python ship 3.16 yet. I'm happy to send that as a follow-up once they do.

ozeranskii added a commit to ozeranskii/cffi that referenced this pull request Oct 18, 2026
ozeranskii added a commit to ozeranskii/cffi that referenced this pull request Oct 18, 2026
@ozeranskii
ozeranskii force-pushed the fix-py316-fsdefaultencoding branch from 9159b58 to c12ae81 Compare October 18, 2026 18:47
ozeranskii added a commit to ozeranskii/cffi that referenced this pull request Oct 19, 2026
@ozeranskii
ozeranskii force-pushed the fix-py316-fsdefaultencoding branch from c12ae81 to 46d7a1e Compare October 19, 2026 17:59
CPython 3.16 removed Py_FileSystemDefaultEncoding from the headers, so
_cffi_backend.c no longer compiled. Convert the file name with
PyUnicode_FSConverter instead, which uses the filesystem encoding,
rejects embedded null characters and is available on all supported
versions. The encoded name is kept in *p_temp, which the callers
already release.
@ozeranskii
ozeranskii force-pushed the fix-py316-fsdefaultencoding branch from 46d7a1e to 38dd2f6 Compare October 19, 2026 18:36
@ozeranskii

ozeranskii commented Oct 19, 2026 •

Copy link
Copy Markdown
Author

@mattip, could you approve the CI workflows for the latest push? The first run failed only in clang_TSAN, on a non-ASCII dlopen test I've since removed: it loaded and closed a copy of the extension from several threads at once, and TSan flagged that inside the dynamic loader. Non-ASCII file names are already covered by test_re_python.py and test_ownlib.py. All other jobs passed, including Windows and iOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant