IDLE: in shell, time.sleep ignores _thread.interrupt_main()

@terryjreedy

BPO	29926
Nosy	@terryjreedy, @pitrou, @njsmith, @vadmium, @eryksun, @mlouielu
PRs	gh-74112: IDLE: Fix blocking function ignore SIGINT #2466
Files	int-unix.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2017-03-27.22:51:11.745>
labels = ['expert-IDLE', 'type-bug', '3.9']
title = 'IDLE: in shell, time.sleep ignores _thread.interrupt_main()'
updated_at = <Date 2019-09-19.18:42:59.551>
user = 'https://bugs.python.org/Mark'

bugs.python.org fields:

activity = <Date 2019-09-19.18:42:59.551>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2017-03-27.22:51:11.745>
creator = 'Mark'
dependencies = []
files = ['46771']
hgrepos = []
issue_num = 29926
keywords = ['patch']
message_count = 31.0
messages = ['290663', '290675', '290677', '290679', '290715', '290728', '290743', '290744', '290749', '290751', '290761', '291049', '291052', '291054', '297157', '297163', '297176', '297241', '297242', '297286', '297288', '297291', '297307', '297323', '297330', '297334', '297345', '297347', '297349', '297352', '297477']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'pitrou', 'njs', 'martin.panter', 'Mark', 'eryksun', 'louielu']
pr_nums = ['2466']
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29926'
versions = ['Python 3.9']

Linked PRs

For Windows the event should be set after trip_signal(SIGINT). That way the flag is guaranteed to be tripped when PyErr_CheckSignals is called from the time module's pysleep() function. This in turn calls the default SIGINT handler that raises a KeyboardInterrupt.

If you want to trigger the standard signal handling logic, then raise(SIGINT) is also an option. On unix it probably won't help because of #66094, but using raise() here + fixing 21895 by using pthread_kill in the c level signal handler would together give a pretty simple and robust way to pretend that the user hit control-C.

But... interrupt_main isn't documented to raise a SIGINT, it's documented to raise KeyboardInterrupt. These are very different, e.g. if the user has installed a custom handler or even set SIGINT to SIG_IGN or masked it. (In the later two cases, using pthread_kill to send a signal to the main thread *won't* interrupt any ongoing syscall.) This is *much* more difficult to make work correctly. And actually what IDLE wants is a fake control-C anyway!

So I'd suggest either redefining the semantics of interrupt_main to be "acts as if SIGINT was received" or else abandoning interrupt_main to its slightly-buggy semantics and adding a new function that does raise(SIGINT) and switching IDLE to use this new function. And fix #66094 in either case.

(oh, in case it wasn't obvious: the advantage of raise() over kill() and pthread_kill() is that raise() works everywhere, including Windows, so it would avoid platform specific logic. Or if you don't like raise() for some reason then you can get the same effect by doing handler = PyOS_getsig(SIGINT); if (handler) handler(SIGINT); )

(Also, this is all assuming the python 3 logic on Windows. On Windows versions of python 2 I can't see any way to make time.sleep interruptible without significant rewriting. The problem is that on py2 the special Event used to signal the arrival of a control-C is a private variable that gets hooked up directly to Windows's low-level control-C logic, and there's effectively no way to get at it from outside. You might think GenerateConsoleCtrlEvent would help, but it is broken in too many ways for me to fit in this text box.)

Nathaniel's suggestion to update the implementation to work with raise(SIGINT) sounds like a good idea.

Fpr Windows, GenerateConsoleCtrlEvent is definitely a non-starter. PyErr_SetInterrupt shouldn't depend on having an attached console. IDLE generally runs without one. Even if the process has a console, there's no way to limit GenerateConsoleCtrlEvent to just the current process. It works with process groups, like POSIX killpg(). Typically a process is created in the session's Winlogon process group unless a new group was created by calling CreateProcess with the flag CREATE_NEW_PROCESS_GROUP. If you call GenerateConsoleCtrlEvent on a process ID that's not a group ID, the console behaves as if you passed group ID 0, which broadcasts to all processes attached to the console.

Even is IDLE is started from a console, the user execution process is not attached to one.

What I want, from an IDLE perspective, is for ^C in IDLE's shell to work as well as it does in a terminal. That means either fixing _thread.interrupt_main to always do what it usually does (from a user perspective), or to be given something else to use in .interrupt_the_server() that does so.

The current workaround for stopping runaway code in interactive mode, restarting the shell with cntl-F6 or the menu entry Shell => Restart Shell, is a) obscure and not known to most beginners, and b) does too much in that it throws away the workspace. It is much like closing a terminal Window with [X], and restarting the terminal and python, instead of using ^C.

Viktor, I am nosying you because of your posts to #66094. Do you have any comments on this issue?

Even is IDLE is started from a console, the user execution process
is not attached to one.

Actually, the execution process is attached to a console, but it doesn't use it for sys.std*. Try

open('con', 'w').write('spam\n')

It is much like closing a terminal Window with [X], and restarting
the terminal and python, instead of using ^C.

There's no default handler for SIGBREAK, so when working in the console it's typically better to use Ctrl+Break to kill Python. This preserves the console's arrow-key & F7 input history and scrollback buffer.

Hi Terry, this patch is what I imagined a fix would look like for Linux. I am not familiar with Idle (internally nor externally), so there may be improvements you can make.

It works as I expected for normal blocking functions and a tight “for” loop: it interrupts any blocking call, and triggers the usual SIGINT handler. If SIGINT has a Python handler (e.g. by default), that gets called which usually raises KeyboardInterrupt.

My change has a major flaw: it seems to deadlock something if you interrupt “input()” or “sys.stdin.readline()”. Perhaps you might have a better clue what the problem is. With the default SIGINT handler, this is what I see in the Idle window:

>>> input()  # Press Ctrl+C
Traceback (most recent call last):  <-- cursor flashing at end of line

If SIGINT is ignored, or the Python handler doesn’t raise an exception, Ctrl+C seems to have the effect of inputting a newline:

>>> input()  # Press Ctrl+C
''
>>> sys.stdin.readline()  # Press Ctrl+C
'\n'

BTW pthread_kill was only added to Python 3.3, so is not available for Python 2. I’m not sure what the best fix for 2.7 would be. Maybe it’s not worth it, or maybe you can find another way to a signal to the user process or its main thread without interfering with any background threads that Idle needs. But I don’t know enough about how Idle works to offer a solution.

A fix for 3.6+ will be better than nothing ;-)

Other operation which will block the process will do the same as time.sleep(), too.

    >>> import socket
    >>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    >>> s.bind(('', 25000))
    >>> s.recv(100)
    <Control-C>

IDLE will ignore this control-c, too.

Based on Martin's patch, I slightly changed the logic for the patch.

So, I add a finish flag to detect the code in Executive.runcode is done or not. interrupt_the_server will first SIGINT via interrupt_main, if this doesn't work after 0.2 seconds, it will then sent the signal by pthread_kill to main_thread.

This will work on Linux, and MacOS which I tested. And it will avoid the different behavior for input() (which I believe is because of the GNU readline been used.) and time.sleep() in only using interrupt_main or signal.pthread_kill.

On windows, s=input() is cleanly interrupted by ^-C, leaving s unbound, before and after the patch. time.sleep(1) is not interrupted, before and after the patch. Ditto for the socket test. Louie, what test are you using on *nix?

It still appears that for Windows a patch to signalmodule.c is still needed. Eryk and Nathaniel, can you agree on an actual patch?

Eryk: off topic a bit, but
>>> open('con', 'w').write('spam\n')
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    open('con', 'w').write('spam\n')
OSError: [WinError 6] The handle is invalid: 'con'

In terms of general design cleanliness, I think it would be better to make interrupt_main work reliably than to have IDLE add workarounds for its unreliability.

AFAICT the ideal, minimal redundancy solution would be:

interrupt_main just calls raise(SIGINT)
signal.pause() and signal handlers don't react to SIGCHLD in non-main thread #66094 is fixed by adding a few lines to Python's C-level signal handler on Unix-likes to check if it's running in the main thread, and if not then use pthread_kill to redirect the signal to the main thread.

If trying to fix #66094 is too controversial, then a minimal change would be:

in interrupt_main, if we're on Windows, call raise(SIGINT); otherwise, call pthread_kill(main_thread, SIGINT)

And in either case we'd also have to document that interrupt_main is now equivalent to the user having hit control-C. But AFAICT that's what all its users really want anyway.

interrupt_main is now equivalent to the user having hit control-C.

That's true on a POSIX system with pthread_kill, but not really for a Windows process that's attached to a console.

A real Ctrl+C executes the registered control handlers for the process. To emulate this, PyErr_SetInterrupt could try calling GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) to broadcast a Ctrl+C event. If the latter fails, it can fall back on raise(SIGINT), such as when the process isn't attached to a console.

One problem is that GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) doesn't cancel a blocking console read like Ctrl+C does. Python's C handler could call CancelSynchronousIo(hMainThread) to address this problem in general. Unfortunately, when a console read is canceled in the client, it isn't canceled in the console itself. The contents of the read will be discarded, but it's a bit clunky that the user has to press enter.

Terry, I assume you ran IDLE via pyw.exe or pythonw.exe, which won't inherit the console of its parent. You have to use py.exe or python.exe. In this case you can also use sys.__stdout__.write('spam\n').

If you run via pythonw.exe or pyw.exe from a command prompt, you could use ctypes to attach to the console, but it would involve searching back to find the cmd.exe or powershell.exe PID to call AttachConsole(pid). More simply you can call ctypes.WinDLL('kernel32').AllocConsole() to get a new one.

It appears you are right about open('con'...).

However, this issue is about the fact that 
>>> import time; time.sleep(10)
^C
...
KeyboardInterrupt

works in the console but does not work in the IDLE Shell in default mode. In default mode (-n not on the command line) entered code is executed in a separate process and and ^C causes a message to be sent to the separate process to simulate ^C keypress by calling 'something'.

The indirect process works for 'normal' code like
>>> s = input('Hi')
so input() is not part of this issue, except that it or its interruption should not be broken.  Ditto for
>>> while True: pass

A real Ctrl+C executes the registered control handlers for the process.

Right, but it's *extremely* unusual for a python 3 program to have a control handler registered directly via SetConsoleCtrlHandler. This isn't an API that the interpreter uses or exposes. The vast majority of programs receive control-C notification by letting the C runtime convert the low level console event into a SIGINT, and then receiving this via the signal module.

To emulate this, PyErr_SetInterrupt could try calling GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) to broadcast a Ctrl+C event.

But as mentioned up thread, this is really flaky - and even when it works it tends to kill random other processes, which is *certainly* not what anyone expects from calling interrupt_main. You can't really call it experimentally – even trying to call it can do stuff like cause appveyor tests to lock up.

Given these two issues, I think that emulating control-C at the signal level makes the best tradeoffs. Something that works 100% reliably for 99.99% of python programs is way better than something that is 80% reliable for 100% of programs.

One problem is that GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) doesn't cancel a blocking console read like Ctrl+C does. Python's C handler could call CancelSynchronousIo(hMainThread) to address this problem in general. Unfortunately, when a console read is canceled in the client, it isn't canceled in the console itself. The contents of the read will be discarded, but it's a bit clunky that the user has to press enter.

This might be something to address as a separate issue? I'm guessing this doesn't affect idle, and it doesn't affect time.sleep, which seem to be the main topics of this thread, plus it sounds like any solution would be mostly orthogonal.

+1 on having PyErr_SetInterrupt() call raise(SIGINT) or equivalent.

The strong claim that "interrupt_main is now equivalent to the user having hit control-C" is the reason I suggested broadcasting CTRL_C_EVENT via GenerateConsoleCtrlEvent. That operation isn't buggy on its own to my knowledge. There is a glitch due to the CRT. If a non-console process calls AllocConsole or AttachConsole, its list of registered handlers gets reset to the default handler that calls ExitProcess, and the CRT provides no way to set its handler again.

I mentioned the possibility of calling CancelSynchronousIo in order to cancel a console read like Ctrl+C does (but clunkier) -- again, because of the equivalence claim. ERROR_OPERATION_ABORTED would need to be handled like EINTR on Unix. This would entail small changes to a lot of code, so it does need a separate issue if there's any support for this idea.

Then maybe simplest solution is to scale back the claim :-).

The important semantic change would be that right now, interrupt_main() is documented to cause KeyboardInterrupt to be raised in the main thread. So if you register a custom SIGINT handler that *doesn't* raise KeyboardInterrupt, interrupt_main() doesn't care: it doesn't invoke your custom handler, it just raises KeyboardInterrupt. My proposal would make it so that interrupt_main() does cause custom SIGINT handlers to be called, as well as triggering all of the interpreters existing machinery for prompt signal delivery (waking up time.sleep etc.), so in that respect it would be much more similar to a "real" control-C than it is now. This is a non-trivial semantic change, so it would need to be documented, but I'm not attached to the "equivalent to control-C" phrasing.

Even on UNIX, a "real" control-C is delivered to all processes in the foreground process group (or something like that, ttys are weird), but we would want interrupt_main() to only be delivered to the current process, so the equivalence wouldn't be exact there either.

That operation isn't buggy on its own to my knowledge

I spent quite some time trying to make this work in order to test control-C handling in trio, and eventually gave up because I just could not make it work reliably. One notable effect is that if you call GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0) in an appveyor test run, then it breaks appveyor -- at the end of the run it freezes with "Terminate batch job (Y/N)?" and then sits until the test run times out. It's possible to work around this with some extremely obscure code (make sure you always spawn a subprocess with CREATE_NEW_CONSOLE or maybe CREATE_NEW_CONSOLE_GROUP etc.), but (a) we can't tell people that they need to do that before running code that uses interrupt_main(), and (b) even when I did that, then I still had bizarre issues I couldn't figure out, where sometimes the event just wouldn't be delivered unless I ran the interpreter with the -u option for unbuffered stdio (?!?!) (see [1]).

I just don't think this is something that CPython can use.

I eventually switched to simulating control-C events for testing by calling raise(SIGINT), and that's worked much much better.

I mentioned the possibility of calling CancelSynchronousIo in order to cancel a console read like Ctrl+C does (but clunkier) -- again, because of the equivalence claim. ERROR_OPERATION_ABORTED would need to be handled like EINTR on Unix. This would entail small changes to a lot of code, so it does need a separate issue if there's any support for this idea.

It still wouldn't be equivalent because control-C only cancels console reads, but CancelSynchronousIo would cancel *any* blocking I/O operation that the main thread was doing, right? So e.g. if the main thread was blocked doing socket.recv, then control-C wouldn't interrupt that, but interrupt_main() / PyErr_SetInterrupt() would. I can certainly see an argument that Python's C-level signal handler *should* call CancelSynchronousIo in general, and then raise(SIGINT) and control-C would be equivalent because they both called CancelSynchronousIo, but yeah, that's way beyond the scope of this issue.

[1] python-trio/trio@9584365#diff-345cfb6c136028f9514b67ee7bb8e035R11

Please step back a bit and read the implementation of interrupt_main(): it calls PyErr_SetInterrupt() (in signalmodule.c!), which merely sets an internal flag saying SIGINT was received.

So, there: PyErr_SetInterrupt() already behaves like SIGINT, *except* that it doesn't actually deliver a C signal, it merely sets a flag. Which is the reason that it fails waking up C syscalls like select().

Demonstration:

>>> def handler(signum, frame):
...     print("got signal %d!" % (signum,))
... 
>>> signal.signal(signal.SIGINT, handler)
<built-in function default_int_handler>
>>> _thread.interrupt_main()
got signal 2!

In the end, making interrupt_main() *actually* deliver a SIGINT instead of merely setting the internal flag for it will certainly be true to the original intent.

Note that, since interrupt_main() claims to interrupt the main thread, it would be ok to use pthread_kill() to make sure it's indeed the C syscall running in the main thread that gets interruped.

Also note that:

PyErr_SetInterrupt() has been setting the SIGINT since the beginning in 1995 (changeset 06d511d by Guido)
interrupt_main() has been calling PyErr_SetInterrupt() since the beginning in 2003 (changeset a11e846 by Kurt B Kaiser)

so it stands reasonable to keep the SIGINT behaviour. The documentation is simply wrong and/or imprecise.

I doubt the Gnu Readline library nor Python’s readline module are relevant. The input function only uses Readline if sys.stdin is the original stdin terminal; I suspect Idle monkey-patches sys.stdin. The readline method reads directly from the file object; it never uses the Readline library (despite the unfortunate clash in names :).

Louie seems to have added two workarounds: the finish flag, and using interrupt_main with a timeout. Both seem racy. What if the flag is altered in another thread between when the flag is checked and when it is acted upon? What has to happen in the 0.2 s between calling interrupt_main and raising SIGINT?

I made Louie's patch a draft pending experimenting with other changes.

I am not terribly concerned about accidental long sleeps. I may investigate shell restart leaving a zombie process and whether interrupt failure could result in a restart popup.

The two comments I deleted were redundant here on github.

#157662 sends a real SIGINT to the main thread of the user process (signal.pthread_kill(), or signal.raise_signal() on Windows) instead of calling _thread.interrupt_main(), so time.sleep() and other blocking calls are interrupted. It also fixes the deadlock in input() observed in #2466: the interrupted wait for a response now releases its lock. The signal is sent while holding a new lock which protects sending a message, otherwise the connection could be corrupted.

Merge branch 'main' into pythongh-74112-interrupt-signal

gh-74112: Make Ctrl-C in the IDLE Shell interrupt blocking calls (GH-…

[3.13] gh-74112: Make Ctrl-C in the IDLE Shell interrupt blocking cal…

[3.14] gh-74112: Make Ctrl-C in the IDLE Shell interrupt blocking cal…

3.15 backport must be generated after prior 3.15 backports are merged after 3.15 branch is unblocked.

The new test failed on the AMD64 Windows Server 2022 NoGIL buildbot: https://buildbot.python.org/#/builders/1241/builds/9341

@encukou

The new test failed on the AMD64 Windows Server 2022 NoGIL buildbot: https://buildbot.python.org/#/builders/1241/builds/9341

@encukou This should be fixed by gh-158914: the test started the SIGINT timer before entering assertRaises(), so on a loaded machine the interrupt could arrive outside the block.

gh-74112: Fix race in test_interrupt_blocking_call (GH-158914)

pythongh-74112: Fix race in test_interrupt_blocking_call (pythonGH-15…

@christianaurichzm

Thanks, @christianaurichzm.

[3.14] gh-74112: Fix race in test_interrupt_blocking_call (GH-158914) (…

[3.15] gh-74112: Make Ctrl-C in the IDLE Shell interrupt blocking cal…

Navigation Menu

Repository navigation

IDLE: in shell, time.sleep ignores _thread.interrupt_main() #74112

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

IDLE: in shell, time.sleep ignores _thread.interrupt_main() #74112

Description

Linked PRs

Activity

eryksun commented on Mar 28, 2017

njsmith commented on Mar 28, 2017

njsmith commented on Mar 28, 2017

eryksun commented on Mar 28, 2017

terryjreedy commented on Mar 28, 2017

eryksun commented on Mar 28, 2017

vadmium commented on Apr 3, 2017

vadmium commented on Apr 3, 2017

terryjreedy commented on Apr 3, 2017

mlouielu commented on Jun 28, 2017

mlouielu commented on Jun 28, 2017

terryjreedy commented on Jun 28, 2017

njsmith commented on Jun 28, 2017

eryksun commented on Jun 29, 2017

eryksun commented on Jun 29, 2017

terryjreedy commented on Jun 29, 2017

njsmith commented on Jun 29, 2017

pitrou commented on Jun 29, 2017

eryksun commented on Jun 30, 2017

njsmith commented on Jun 30, 2017

pitrou commented on Jun 30, 2017

pitrou commented on Jun 30, 2017

pitrou commented on Jun 30, 2017

vadmium commented on Jul 1, 2017

terryjreedy commented on Feb 4, 2023

serhiy-storchaka commented on Sep 17, 2026

terryjreedy commented on Sep 27, 2026

encukou commented on Oct 6, 2026

christianaurichzm commented on Oct 6, 2026

serhiy-storchaka commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions