From 9b22db377e0af5f40951334b44045cd90e333053 Mon Sep 17 00:00:00 2001 From: Larry Ruane Date: Mon, 19 Oct 2026 09:05:12 -0600 Subject: [PATCH 1/5] README: scaling notes, link Dunkels' site The broadcast-cost paragraph framed the cost as small because microcontrollers have few protothreads. The real point is how many waiters share one channel, which holds on any hardware. The PT_NWAIT entry only covered shrinking the table. With many more waiters than buckets each wakeup walks a long list (measured: 67 us per handoff with 1M waiters and 1024 buckets, 117 ns with 2^20), so say to raise it. Link Dunkels' protothreads site, which is friendlier and links the paper, instead of the PDF. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SR7v9BVsFrHiEhQBSqZmAs --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 6c67da7..0fc4f57 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![CI](https://github.com/LarryRuane/protothread/actions/workflows/ci.yml/badge.svg)](https://github.com/LarryRuane/protothread/actions/workflows/ci.yml) -[Protothreads](http://en.wikipedia.org/wiki/Protothreads) is a programming model invented by Adam Dunkels that combines the advantages of _event-driven_ (sometimes also called _state machine_) programming and _threaded_ programming. The trick underneath, jumping back into the middle of a function, goes back to [Duff's device](https://en.wikipedia.org/wiki/Duff%27s_device) (1983), Tom Duff's `switch` that jumps into the middle of a loop. Simon Tatham turned it into [coroutines in C](https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html), and Dunkels built protothreads on that. The main advantage of the event-driven model is efficiency, both speed and memory usage. The main advantage of the threaded model is [algorithm clarity](http://dunkels.com/adam/dunkels06protothreads.pdf). Protothreads gives you both. A protothread is a lightweight thread. As with event-driven programming, there is a single stack, shared by every protothread. But as with threaded programming, a function can (at least conceptually) block. (This is why protothreads are called "stackless". There is a stack, but no thread has one of its own.) This protothreads implementation: +[Protothreads](http://en.wikipedia.org/wiki/Protothreads) is a programming model invented by [Adam Dunkels](https://dunkels.com/adam/pt/) that combines the advantages of _event-driven_ (sometimes also called _state machine_) programming and _threaded_ programming. The trick underneath, jumping back into the middle of a function, goes back to [Duff's device](https://en.wikipedia.org/wiki/Duff%27s_device) (1983), Tom Duff's `switch` that jumps into the middle of a loop. Simon Tatham turned it into [coroutines in C](https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html), and Dunkels built protothreads on that. The main advantage of the event-driven model is efficiency, both speed and memory usage. The main advantage of the threaded model is algorithm clarity. Protothreads gives you both. A protothread is a lightweight thread. As with event-driven programming, there is a single stack, shared by every protothread. But as with threaded programming, a function can (at least conceptually) block. (This is why protothreads are called "stackless". There is a stack, but no thread has one of its own.) This protothreads implementation: * is not an implementation of POSIX threads, and isn't compatible with other versions of protothreads * does not require assembly-language code or use setjmp/longjmp * is independent of CPU architecture @@ -573,7 +573,7 @@ It's okay for multiple conditions to map to the same channel, as in producer/con **Use `pt_broadcast()` by default.** It's what the Unix kernel's `wakeup()` always did, and with the `while` loops above it's always correct. Every waiter re-tests, those that can proceed do, and the rest go back to sleep. `pt_signal()`, which wakes only the oldest waiter, is an optimization with conditions attached, and they're easy to break. It's safe only when three things hold. Every waiter on the channel waits for the same condition. The woken waiter is certain to run (it can't be `pt_kill()`ed). And that waiter, on every path, either leaves the condition false or, if it leaves it true, signals again to pass the wakeup on. Those are properties of the whole program, not of the line where `pt_signal()` is called. So a signal that's correct today can become a hang when someone later adds a different kind of waiter to the same channel. -Broadcasting wakes waiters that then go back to sleep, in proportion to how many are waiting on the one channel. On a microcontroller, with protothreads numbering in the tens and seldom more than a few sharing a channel, that's negligible. Where it does matter, consider splitting the channel (making it finer-grained). A channel is only an address, so each condition can have its own, `&s->a` and `&s->b` rather than `&s`. When a channel has exactly one waiter, signal and broadcast do the same thing. This library's lock, timers, and queues split their channels this way. +Broadcasting wakes waiters that then go back to sleep, in proportion to how many are waiting on the one channel. When only a few share a channel, which is the usual case, that's negligible. Where it does matter, consider splitting the channel (making it finer-grained). A channel is only an address, so each condition can have its own, `&s->a` and `&s->b` rather than `&s`. When a channel has exactly one waiter, signal and broadcast do the same thing. This library's lock, timers, and queues split their channels this way. The debug (testing) define `PT_SIGNAL_WAKES_ALL` checks the other direction. If set to `1`, every `pt_signal()` wakes everyone. Correct code still works, more slowly. This library's test suite and demos enable that in CI, and its own semaphores, locks, timers, and queues don't call `pt_signal()` at all. @@ -736,7 +736,7 @@ All configuration is by preprocessor macro. Because the library is header-only, > `PT_NWAIT` (default `1024`) > -> Number of wait queues, a power of 2. Waiting threads are hashed onto this table by channel address. Each entry is one pointer, so the default costs 8 kB per `protothread_t` on a 64-bit machine. **Set `PT_NWAIT=1` on a memory-constrained system.** With only a handful of waiters, one linear wait list is both smaller and faster than hashing. +> Number of wait queues, a power of 2. Waiting threads are hashed onto this table by channel address. Each entry is one pointer, so the default costs 8 kB per `protothread_t` on a 64-bit machine. **Set `PT_NWAIT=1` on a memory-constrained system.** With only a handful of waiters, one linear wait list is both smaller and faster than hashing. A wakeup searches the whole list its channel hashes to, so it slows down when waiters far outnumber wait queues. For a large number of protothreads, raise `PT_NWAIT` to about the number waiting at once. > `PT_NO_MALLOC` > @@ -967,7 +967,7 @@ The one exception is `FindPROTOTHREAD.cmake`, which is third-party code by Ryan Lawrence M. Ruane, [Process Synchronization in the UTS Kernel](https://static.usenix.org/publications/compsystems/1990/sum_ruane.pdf), Computing Systems 3(3), Summer 1990, pp. 387-421. Background for the wait-channel design, broadcast wakeups, and semaphore convoys. -I wish to gratefully acknowledge Adam Dunkels (with support from Oliver Schmidt) for inventing this brilliant idea. Please see his [web site](http://dunkels.com/adam/pt/). +I wish to gratefully acknowledge Adam Dunkels (with support from Oliver Schmidt) for inventing this brilliant idea. Please see his [web site](https://dunkels.com/adam/pt/). My thanks to Paul Soulier for introducing me to the concept of protothreads, and to Marshall McMullen and John Rockenfeller for reviewing drafts of this article. From f008ec0bc4e2b1a1468c42d9a0111c3f7c1de480 Mon Sep 17 00:00:00 2001 From: Larry Ruane Date: Mon, 19 Oct 2026 09:55:57 -0600 Subject: [PATCH 2/5] README: lead the Dunkels differences with wait channels Wait channels and nesting are the two main differences from Dunkels' protothreads. Put the scheduler first, since nesting's difference is explained in terms of it, and say that a blocked thread takes no CPU time however long it waits. Add a TODO entry for growing the wait table at runtime. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SR7v9BVsFrHiEhQBSqZmAs --- README.md | 4 ++-- TODO.md | 12 ++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0fc4f57..d468900 100644 --- a/README.md +++ b/README.md @@ -17,9 +17,9 @@ If protothreads never run two things at once, why threads at all? Because runnin I wrote this from scratch, and it is not compatible with other versions of protothreads. I came to it from a Unix kernel background. Dunkels' idea is brilliant, but I wanted a programming interface that felt more like the threaded code I was used to. These are the main differences: - * **Blocking functions nest with an ordinary call.** A protothread function can `pt_call()` another protothread function, passing it arguments, and the one it calls can block. This works to any depth, so you can factor blocking code into subroutines the way you would anywhere else. Dunkels' library nests too, with `PT_SPAWN()`. What differs is how a blocked thread gets resumed, the third point below. + * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. So it takes no CPU time, however long it waits. Dunkels' library leaves scheduling to the application. A blocked protothread is simply called again later, and re-tests its condition each time, re-running every level of any nesting to get there. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. + * **Blocking functions nest with an ordinary call.** A protothread function can `pt_call()` another protothread function, passing it arguments, and the one it calls can block. This works to any depth, so you can factor blocking code into subroutines the way you would anywhere else. Dunkels' library nests too, with `PT_SPAWN()`. What differs is how a blocked thread gets resumed, the point above. * **You can block anywhere**, including inside a `switch` statement. Implementations built on Duff's device cannot, because they're already using `switch`. - * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. Dunkels' library leaves scheduling to the application. A blocked protothread is simply called again later, and re-tests its condition each time, re-running every level of any nesting to get there. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. Nesting and arbitrary blocking require [gcc label variables](http://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html), which aren't standard C, restricting the compiler to **gcc or clang** (see [Compiler requirements](#compiler-requirements)). Dunkels' `switch`-based version is portable to any C compiler. diff --git a/TODO.md b/TODO.md index 3d80ac0..d83ae38 100644 --- a/TODO.md +++ b/TODO.md @@ -211,6 +211,18 @@ debugging into a one-line call, and costs nothing in a production build. new array has to happen with interrupts masked if `pt_timer_run()` is called from an interrupt handler. Each env would also store its index in the array, so cancel can find it. + * **Grow the wait table.** With many more waiters than `PT_NWAIT` buckets, + each wakeup walks a long list (67 us per handoff at 1M waiters and 1024 + buckets, 117 ns at 2^20). Without `PT_NO_MALLOC`, `protothread_t` could + hold a pointer and a mask, starting at the inline array, and `pt_wait()` + could double the table when waiters pass twice the bucket count. A + failed malloc keeps the old table. Rehash oldest-first to keep per-channel + FIFO order. The rehash is O(N) with interrupts masked, so turn growth off + when `PT_CRITICAL_*` is defined. Grow only tables from + `protothread_create()`, so `protothread_free()` can free them. The hash + takes the middle bits, which caps the table at 2^16 on 32-bit. No API + change. Not urgent, since 100k waiters still take under 1 us per wakeup at + the default size. * **Killing a sleeping protothread** leaves a dangling entry on the timer list unless `pt_timer_cancel()` is called first; same hazard the reader-writer lock has. Documented; a generated typed kill (see type-safe top-level From 6c03d880b42a2bbc5307f58217fc1fac54dd63d4 Mon Sep 17 00:00:00 2001 From: Larry Ruane Date: Mon, 19 Oct 2026 10:00:25 -0600 Subject: [PATCH 3/5] README: don't count nesting as a difference from Dunkels' Dunkels' PT_SPAWN() nests blocking functions the same way pt_call() does: the parent re-calls the child on resume, at any depth, with arguments. The difference is when the chain is re-entered, which is the scheduler's doing, so say that in the scheduler bullet and drop the separate nesting bullet. Nesting works with Dunkels' switch-based version too, so only blocking inside a switch needs labels-as-values. Likewise, the bytes per thread over Dunkels' buy the wait lists, not nesting. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SR7v9BVsFrHiEhQBSqZmAs --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d468900..2e16f8d 100644 --- a/README.md +++ b/README.md @@ -17,11 +17,10 @@ If protothreads never run two things at once, why threads at all? Because runnin I wrote this from scratch, and it is not compatible with other versions of protothreads. I came to it from a Unix kernel background. Dunkels' idea is brilliant, but I wanted a programming interface that felt more like the threaded code I was used to. These are the main differences: - * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. So it takes no CPU time, however long it waits. Dunkels' library leaves scheduling to the application. A blocked protothread is simply called again later, and re-tests its condition each time, re-running every level of any nesting to get there. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. - * **Blocking functions nest with an ordinary call.** A protothread function can `pt_call()` another protothread function, passing it arguments, and the one it calls can block. This works to any depth, so you can factor blocking code into subroutines the way you would anywhere else. Dunkels' library nests too, with `PT_SPAWN()`. What differs is how a blocked thread gets resumed, the point above. + * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. So it takes no CPU time, however long it waits. Dunkels' library leaves scheduling to the application. A blocked protothread is simply called again later, and re-tests its condition each time, re-running every level of any nesting to get there. Both libraries let blocking functions nest, with `pt_call()` here and `PT_SPAWN()` in Dunkels'. Here a blocked chain of calls is re-entered only once its channel is signaled. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. * **You can block anywhere**, including inside a `switch` statement. Implementations built on Duff's device cannot, because they're already using `switch`. -Nesting and arbitrary blocking require [gcc label variables](http://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html), which aren't standard C, restricting the compiler to **gcc or clang** (see [Compiler requirements](#compiler-requirements)). Dunkels' `switch`-based version is portable to any C compiler. +Allowing blocking within a `switch` requires [gcc label variables](http://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html), which aren't standard C, restricting the compiler to **gcc or clang** (see [Compiler requirements](#compiler-requirements)). Dunkels' `switch`-based version is portable to any C compiler. ### What you get ### @@ -643,7 +642,7 @@ But blocking is often *fine*. If the call is rare and short, like reading a conf ## Memory overhead and performance (benchmarks) ## -The best-known implementation of protothreads (by Adam Dunkels) uses just two bytes per protothread. This implementation is not quite so parsimonious, mainly because it includes a scheduler. Threads are on either the wait or the run list, and that costs pointers. In exchange you get nesting, wait channels, and synchronization primitives. +The best-known implementation of protothreads (by Adam Dunkels) uses just two bytes per protothread. This implementation is not quite so parsimonious, mainly because it includes a scheduler. Threads are on either the wait or the run list, and that costs pointers. In exchange you get wait channels, so a blocked thread isn't run until it's woken. Each protothread function context has a `pt_func_t` structure, which is 2 pointers. Each overall protothread requires a `pt_thread_t` structure, which is 5 pointers. Measured with `PT_DEBUG=0`: From 7c1980eaeb094a5aeb1f3c8afc761185e4909fd3 Mon Sep 17 00:00:00 2001 From: Larry Ruane Date: Mon, 19 Oct 2026 10:41:44 -0600 Subject: [PATCH 4/5] README: why pt_wait() takes no condition Dunkels' PT_WAIT_UNTIL() takes the condition as an argument, and a reader may ask why pt_wait() doesn't. A bare wait can sit deep inside nested control flow, with the code re-evaluating everything from the top after it wakes, which is how some UTS kernel code was written. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SR7v9BVsFrHiEhQBSqZmAs --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 2e16f8d..29e261b 100644 --- a/README.md +++ b/README.md @@ -570,6 +570,8 @@ You can reason about the second in the same way as the first. That is also why t It's okay for multiple conditions to map to the same channel, as in producer/consumer. But we could signal one channel when the mailbox becomes full, and a separate channel when the mailbox becomes empty. We could even use `c->mailbox + 1` for this additional channel, since this address isn't dereferenced (although this would look strange). +`pt_wait()` takes no condition, unlike Dunkels' `PT_WAIT_UNTIL()`. A wait doesn't have to sit in a loop around one expression. It can be deep inside nested `if` and `while` statements, and after it wakes, the code can go back to the top of the function and re-evaluate everything. I wrote kernel code in UTS that way, because the condition being waited for was the whole chain of tests rather than one expression. (`pt_wait_until()` takes a condition, for the interrupt case.) + **Use `pt_broadcast()` by default.** It's what the Unix kernel's `wakeup()` always did, and with the `while` loops above it's always correct. Every waiter re-tests, those that can proceed do, and the rest go back to sleep. `pt_signal()`, which wakes only the oldest waiter, is an optimization with conditions attached, and they're easy to break. It's safe only when three things hold. Every waiter on the channel waits for the same condition. The woken waiter is certain to run (it can't be `pt_kill()`ed). And that waiter, on every path, either leaves the condition false or, if it leaves it true, signals again to pass the wakeup on. Those are properties of the whole program, not of the line where `pt_signal()` is called. So a signal that's correct today can become a hang when someone later adds a different kind of waiter to the same channel. Broadcasting wakes waiters that then go back to sleep, in proportion to how many are waiting on the one channel. When only a few share a channel, which is the usual case, that's negligible. Where it does matter, consider splitting the channel (making it finer-grained). A channel is only an address, so each condition can have its own, `&s->a` and `&s->b` rather than `&s`. When a channel has exactly one waiter, signal and broadcast do the same thing. This library's lock, timers, and queues split their channels this way. From 5776f0e1272f1400b4b1364b515d6aaac9c83e01 Mon Sep 17 00:00:00 2001 From: Larry Ruane Date: Mon, 19 Oct 2026 10:53:57 -0600 Subject: [PATCH 5/5] README: say what Dunkels' scheduling looks like "Leaves scheduling to the application" didn't say what that means. His library has no list of threads; the application's loop calls each one in turn, which acts like a single run list holding every thread, blocked or not. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01SR7v9BVsFrHiEhQBSqZmAs --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 29e261b..1c2a059 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ If protothreads never run two things at once, why threads at all? Because runnin I wrote this from scratch, and it is not compatible with other versions of protothreads. I came to it from a Unix kernel background. Dunkels' idea is brilliant, but I wanted a programming interface that felt more like the threaded code I was used to. These are the main differences: - * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. So it takes no CPU time, however long it waits. Dunkels' library leaves scheduling to the application. A blocked protothread is simply called again later, and re-tests its condition each time, re-running every level of any nesting to get there. Both libraries let blocking functions nest, with `pt_call()` here and `PT_SPAWN()` in Dunkels'. Here a blocked chain of calls is re-entered only once its channel is signaled. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. + * **A real scheduler, with _wait channels_.** `pt_wait()`/`pt_signal()`/`pt_broadcast()` on an arbitrary address deliberately mirror condition variables (`pthread_cond_wait()`, except a `pthread_cond_t` variable isn't needed) and the classic Unix kernel [`sleep()`/`wakeup()`](https://www.ques10.com/p/47397/sleep-event-and-addresses-in-unix-memory-managem-1/). Each thread is on either the run list or a wait list, and a blocked thread does not run until its channel is signaled. So it takes no CPU time, however long it waits. Dunkels' library has no list of threads. The application's main loop calls each one in turn, so in effect every thread is always on one run list, blocked or not, and re-tests its condition each time it's called. Both libraries let blocking functions nest, with `pt_call()` here and `PT_SPAWN()` in Dunkels'. Here a blocked chain of calls is re-entered only once its channel is signaled. Semaphores, reader-writer locks, timers, and message queues are built on top of it, in optional headers of their own. * **You can block anywhere**, including inside a `switch` statement. Implementations built on Duff's device cannot, because they're already using `switch`. Allowing blocking within a `switch` requires [gcc label variables](http://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html), which aren't standard C, restricting the compiler to **gcc or clang** (see [Compiler requirements](#compiler-requirements)). Dunkels' `switch`-based version is portable to any C compiler.