capnp-RPC zero-copy integration: Improvement: `-> stream` flow control window based on available (arena?) memory.

[ygoldfeld]

In #121 we added zero-copy integration with capnp-RPC (https://capnproto.org/rpc.html). Super cool! In so doing we came across a detail: how vanilla (non-zero-copy) capnp-RPC handles https://capnproto.org/news/#multi-stream-flow-control.

I now copy-paste a comment, as it is pretty decent background.

template<typename Shm_lender_borrower_t, typename Shm_arena_t>
size_t Session_vat_network<Shm_lender_borrower_t, Shm_arena_t>::Rpc_msg_out_impl::sizeInWords()
{
  /* What to return here is not necessarily obvious (of course that is subjective).  The API contract is
   * "Get the total size of the message, for flow control purposes."  The question then is what "flow control"
   * in this particular context is.  Naively one could assume it has to do with controlling back-pressure
   * w/r/t simply bytes traveling over the transport (whether -- in vanilla TwoPartyVatNetwork -- over the network
   * or IPC; just the latter for us).  Without thinking about it too hard one would then simply
   *   return m_msg->sizeInWords(); // This would return the size of our little SHM handle-containing message.
   * All cool -- that's what we're still transmitting; done!  But no.  To explain:
   *
   * (Firstly let's eliminate the over-the-network case from consideration.  We are not doing that, so it'll be
   * easier and fully appropriate to ignore in the discussion.  So: just local IPC.)
   *
   * This sizeInWords() does *not* apply to general traffic going through this connection, in the first place.
   * It applies only to a specific capnp-RPC user-facing feature: *streaming*.  (This is completely independent of
   * AsyncIoStream or AsyncCapabilityStream... which is in fact the IPC-transport byte/FD-stream one could naively
   * guess per the paragraph above.  Not that!)  This is where in the schema an interface-method return type
   * is `-> stream`.  As of this writing this isn't covered in the main docs at the capnp web site, but it is
   * introduced nicely in https://capnproto.org/news/#multi-stream-flow-control.  Read that
   * as background, optionally, but the bottom line is: It allows one to set up, say, bulk uploads -- which will stream
   * at a controlled rate.  This feature consists of two parts; part 1 is the syntactic niceness of expressing this
   * without having to set up the logic in one's own code; and part 2 -- which concerns us here -- is that by
   * expressing it in that syntax, it causes capnp-RPC's guts to engage *multi-stream flow control*, wherein it
   * takes a good stab at automatically controlling the rate at which it'll allow the `stream` payload to
   * proceed.  So that's the flow control in question:
   *   - A window-size (controlled by newStream(); see that guy).
   *   - An accounting of how much of the window is currently in use. <- The relevant part here in sizeInWords().
   *
   * (Reminder: We're ignoring any networking aspect of this.)  Without networking -- with a tiny latency and
   * very high bandwidth -- what's left to control is how much RAM is taken by (collective) streams.
   * (Incidentally: There is at most one ongoing stream per capability a/k/a interface-implementing object, and
   * 2+ objects don't share streams.)  So, e.g., if I generate random bytes in chunks, stream them over capnp-RPC, and
   * the receiver then (say) writes them to tape as fast as it can, then I wouldn't want to keep generating
   * more random bytes until receiver-side has indicated it has written a bunch and can accept more bytes now.
   * Without back-pressure the accumulated-too-fast (and more and more so) bytes would exceed the available RAM.
   * So that's, ultimately, all it is.
   *
   * What we're returning here, then, is how much we'd contribute to that window.  Both the window (see
   * newStream()) and the window-used (here) are about RAM use.  For a vanilla TwoPartyVatNetwork the RAM use
   * is just the stuff traveling through the IPC-transport: first inside a MallocMessageBuilder around here (m_msg),
   * then in the kernel buffer(s), then in/near the MessageReader in the IncomingRpcMessage (a/k/a Rpc_msg_in).
   * So in that case `m_msg->sizeInWords()` is right.  In our case, we avoid (almost) all that: the use of
   * the transport is (basically) negligible (~only SHM-handles go through it); instead we just plop user
   * (streamed) data in SHM.  SHM is RAM; our area in it just doesn't get copied from one area to another to another
   * (zero-copy!).  Thus: */
  return m_capnp_msg_in_shm.sizeInWords();
} // Session_vat_network::Rpc_msg_out_impl::sizeInWords()

Now... what about the window then? Currently it default to 512Ki and can be overridden by the user via a few similar-looking knobs at different layers (VAT network layer, or client/server context layer, or even Ez_rpc layer).

The proposal here is to auto-determine some smart -- available-RAM-based? -- window. Specific are full-on TBD; but here is at least some food for thought.

• Calculation/policy could be based on which SHM-provider was chosen (compile-time decision: Session_vat_network<...decided here...>).
  • SHM-classic: A SHM-arena's size is limited and cannot be changed after arena is created. (Default 2Gi, knobs in ipc::session or on Pool_arena itself.) This value could be used as input into .
  • SHM-jemalloc: There is no arena size. There is general RAM and/or OS-dependent limits on total SHM use. Details TBD.
• Even if calculating based on some notion of "available space," it probably needs to be conservative. Needs a reserve policy, since the Returns (rpc::Message type) that free chunks are SHM messages too; and a real streaming workload against which to tune. TBD-galore.

Priority: By default, not high... but really it might be downstream of a profiling project for capnp-RPC integration specifically. That might be a separate ticket. Reminder: as I write this, capnp-RPC will be released soon (release 3) -- that is v1 of capnp-RPC integration.