Repository navigation
UnicodeDecodeError from juliacall IPython extension #679
Description
Affects: JuliaCall
Describe the bug
Various users have reported stray UnicodeDecodeErrors when using PySR (=>juliacall) in a Jupyter session:
- [BUG]: Frequent unicodedecode error astroautomata/PySR#1043 (@jobs-git)
- Getting a uft-8 Error astroautomata/PySR#945 (@AaronRoseDA @DorofeevKirillDev @GongJr0)
I think all of these users were on Windows. The error looks like this:
Error in callback _flush_stdio (for post_execute), with arguments args (),kwargs {}:
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
File ~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:262, in __call__(self, *args, **kwargs)
260 return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlany_dir)))
261 def __call__(self, *args, **kwargs):
--> [262](~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:262) return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
263 def __bool__(self):
264 return True
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 4095: unexpected end of dataThe workaround seems to be to set PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=no before loading. I think it's worth figuring out the root cause of the issue, though.
One quick fix is that we have PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=auto by default, which would get set to no on Windows, or yes on Linux/macOS. Although the downside is this would make notebooks non-reproducible on different operating systems, so it's probably better to fix the root cause.
cc @cjdoris - any ideas?
Activity
cjdoris commentedon Sep 27, 2025
This is happening in the IPython callback that is called after each cell which flushes stdout and stderr. We redirect these streams so that they are sent to the kernel and so are visible in the notebook (otherwise
printdoesn't work).The
unexpected end of datamakes me think that the redirected stream is being flushed while only part of a UTF-8 character has actually been written to it. Because Python'ssys.stdoutis a text IO, you write strings to it, and Python strings only accept valid unicode, and so constructing the string to write fails because there is a partial character at the end.The correct answer is probably to change
flush(::PyIO)to not necessarily flush the entire buffer, but to leave any partial character at the end in the buffer until the next flush.fix #679 - defer printing of partial characters at end of buffer when…
fix JuliaPy#679 - defer printing of partial characters at end of buff…