[koenvanderveen]

What

The DS now reads the outbox of the DO through P2PConnection.receive() / mark_processed(), the same way the DO already reads its inbox. This is the step before a WebRTC P2P connection (over a fake channel): WebRTC has no Drive file ids or timestamps, so the DS could not read from it before.

• receive() takes an optional after: the msg_id of the newest message the receiver processed, and already_received: the ids of the ones it has. Drive keeps outbox messages (the DS rebuilds a lost cache from them), so it lists from a bit before after and skips already_received before downloading (see the look-back below). A connection that drops processed messages ignores both. They go away with send-and-forget.
• DriveP2PConnection.receive(FILE_EVENTS) lists the outbox once and downloads in parallel through the existing _download_all. mark_processed on file events does nothing on Drive.
• The DS cache keeps last_msg_id_per_peer (was last_event_timestamp_per_peer) and received_msg_ids, both rebuilt from saved messages on start. sync_down and sync_down_parallel are merged into one sync_down.
• Router: new watcher_receive_events_messages; owner_mark_all_processed is now mark_all_processed, used by both roles. Removed the Drive-only reads: watcher_get_events_messages, watcher_get_outbox_file_metadatas, outbox_messages, outbox_message_metadatas.

Outbox compaction is unchanged; it still uses the Drive-only methods and moves in a later PR.

Fixes from review

• Failed download: a message that fails to download ends the outbox list, so the bookmark never passes it and the next sync tries again. The inbox still skips the file.
• Unreadable message: one that can never be read (format, signature, decrypt, decode) comes back as None with a warning, and the DS moves past it instead of failing every sync. One from a peer that is not set up yet ends that peer's list and waits for the next sync; other peers continue.
• Late messages: a message can land after newer ones (a retried send, or the notebook and syft-bg writing the same outbox). The DS lists back OUTBOX_LOOKBACK_SECONDS (60s) before its bookmark and skips what it already received. The bookmark only moves forward. A message that lands more than 60s late is still missed.
• Bookmark on restart: keyed by the lowercased peer email, so it is found when the email differs in case. Restoring is split into _load_saved_messages, _restore_received and _restore_file_hashes.

Drive API calls: one listing per peer plus one download per new message, as before. The look-back can add a listing page when more than 100 messages fall in it; messages already received are not downloaded again.

Tests

• New: outbox messages after a given one come oldest first; processed outbox messages stay; no outbox returns nothing; a restarted DS with a cache on disk receives only the new messages.
• New for the fixes: a late message is received without downloading the newer one again; a message older than the look-back is not listed; the outbox list stops at a failed download; the router reads an unreadable message as None and stops at one not readable yet; a DS sync for each fix; a restored bookmark is found for the peer email in another case.
• Fast unit suite: 733 passed
• tests/migrations: 115 passed
• Integration tests in mock mode: 8 passed, 4 skipped (real Drive only), before the fixes
• Real-Drive integration tests are updated but not run (nightly).