diff --git a/CLAUDE.md b/CLAUDE.md index e89db90..0ff8841 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,7 +50,7 @@ swift run -c release pladder-cli polish-set docs/polish-set.json --model s1-mini | Learned corrections | After a paste the field is watched through Accessibility for up to 60 s; a word the user corrects that passes a token diff, a phonetic gate (Soundex or edit distance ≤ 2) and a yes/no review by the on-device model becomes one menu line, "Learned “x” → “y”? Add / Dismiss" | Nothing runs before Cmd+V is posted: the hook is in `AppModel.handle(.inserted)`, the watcher lives on its own thread and reads only the pasted range plus a margin, the review runs on a detached task. Present only with Accessibility and Apple Intelligence, absent otherwise, no setting, no change to the menu bar glyph. Dismissed pairs go to `dismissed-corrections.json`, not settings, so a bug there can never cost the dictionary. Pure case changes are never proposed. Terminals and TUIs expose a screen buffer, not a field, so nothing is learned there | | Toggle key | Another recordable chord, off by default. A chord of its own latches at release however long the press; equal to the push-to-talk chord it makes that key hybrid: a tap under 400 ms latches, a longer hold stops at release. The next press of any chord, Escape or the 10 min cap ends a latched recording. The overlay is the same as for a held recording, in and out; only the menu's status line says which key stops it | Two-minute dictations should not need a key held for two minutes. Handy and VoiceInk default to hybrid on one key; here it is opt-in, because a stray tap would otherwise leave the microphone open until the cap pastes two minutes of room noise. Hold, toggle and hybrid are decided in `HotkeyGestureTracker`, a clockless value type timed by the instant each monitor stamps on its events, so both monitors behave alike and a press that waits for the microphone cannot make the next release look longer. A same-chord press within 50 ms of its release is a bounce (some Bluetooth keyboards do this mid-hold): it never acts, and the first one seen turns on a 50 ms settle before every stopping release for the rest of the run, so only a keyboard that needs it pays for it and the release path is otherwise untouched. No separate press debounce: both monitors already report alternating presses and releases. Without Accessibility the toggle chord registers with Carbon like the key; one Carbon cannot register has no stand-in, except that a hybrid chord follows the key's. With a lone modifier as a hybrid key, the Command of a later Cmd+C ends a latched recording | | Escape | Discards a recording without transcribing and plays the stop sound; taken only while a recording is on | Never taken globally, so Escape keeps closing dialogs. On the tap `HotkeyChordSet` catches it before the chord trackers, so the interrupted-press rule never sees it, and Escape with the chord's own modifiers held still counts; Carbon registers the bare key around each recording, from the main queue so the release path never waits on it. Under Secure Event Input a modifier-only chord stays on the tap, where no key-down arrives, so Escape cannot cancel there | -| Output | Clipboard + simulated Cmd+V; the old clipboard is restored off the critical path | Universal, fast | +| Output | Clipboard + simulated Cmd+V; the transcript is a pasteboard promise and the old clipboard comes back off the critical path, 400 ms after Cmd+V if the target app has read it by then, otherwise 200 ms after it does, or after 8 s if nothing does | Universal, fast. Waiting for the read as well as the 400 ms: an app busy when Cmd+V arrives reads late, and the timer alone handed it the old clipboard (issue #40). A read never shortens the 400 ms, because Chromium sometimes reads once before the real paste and the pasteboard serves the second read itself. The promise is served on the main thread, so a busy Pladder main thread delays the target's read; the `paste` log line records when it came. It relies on the nspasteboard.org markers: without them Universal Clipboard reads every write within ~15 ms, which would pass for the paste | | Polish model | A picker under the toggle: Apple Intelligence (default), S1-mini by Superwhisper at full precision (f16, 1.5 GB) or 8-bit (Q8_0, 805 MB), the S1-mini files run by llama.cpp on the GPU. `PolishRouter` is the coordinator's one refiner and forwards to the chosen model | On the polish set (docs/BENCHMARKS.md) S1-mini is more accurate than Apple's model and three to five times faster, and translated nothing where Apple's model translated two dictations. It is Qwen3-0.6B fine-tuned for dictation cleanup, trained on English, and needs its own fixed prompt. llama.cpp comes as its prebuilt XCFramework, pinned by release and checksum, because MLX Swift needs Xcode to build its Metal shaders; `bundle.sh` embeds and signs the framework. The GPU leaves the Neural Engine to Parakeet. A file downloads only when polish is on and the model picked, is loaded at the first key-down with the fixed prompt prefix decoded once, and is freed on a switch or with polish off. The licence asks for the name as "S1-mini" by "Superwhisper" | | Post-processing | Filler remover, dictionary replacer, fuzzy custom-word corrector, whitespace normaliser, spoken punctuation, in that order | No latency, no network. Spoken punctuation takes only phrases that are never ordinary words ("period", "Punkt", "punto" stay words), runs last because the whitespace step folds line breaks, and needs Spanish evidence for "coma"; Parakeet already writes numbers as digits. An earlier Apple Intelligence step was removed from this path unmeasured; the model is back behind the experimental polish toggle only | | Mute while dictating | Off by default; `kAudioDevicePropertyMute` on the default output device 200 ms into a recording, restored off the release path | Music or a call otherwise goes into the microphone. The delay means a tap-and-release never toggles anything; a device the user had already muted is left alone, and the device that was muted is the one unmuted even if the default changed meanwhile | @@ -78,7 +78,7 @@ swift run -c release pladder-cli polish-set docs/polish-set.json --model s1-mini - **Long recordings.** FluidAudio's encoder window is 15 s. Longer audio is split into windows and stitched, and seams can drop or duplicate words. Those windows now run while the key is held rather than at release, so the wait is flat with length, but the seam risk is unchanged: it is the same layout and the same merge. The paced benchmark guards it by requiring the text to be byte-identical to transcribing the whole recording at once, and the 30 s to 10 min fixtures watch the word error rate. - **Polish latency and quality.** Apple's model takes about 1.5 to 2 seconds warm on an M1 for a typical dictation, S1-mini 0.3 to 0.5, and both are capped at eight; past the cap the text is pasted as dictated. A small model can still rewrite rather than clean; for Apple's the guided `cleanedText` field, greedy sampling, inline examples and naming the transcript's language keep that rare. S1-mini's known slips are in docs/BENCHMARKS.md, the worst a German number word read wrong, which real dictation rarely feeds it since Parakeet writes digits. `pladder-cli polish-set` is where model and prompt changes are judged. - **Polish model memory.** S1-mini stays loaded while chosen and polish is on: about 0.8 or 1.5 GB of weights plus a 235 MB key-value cache for its 2,048-token context. Long dictations are polished in chunks of about 250 words to fit. -- **Clipboard clobbering.** Output saves the pasteboard, pastes, and restores it after a short delay. +- **Clipboard clobbering.** Output saves the pasteboard, pastes, and restores it once the target app has read the transcript. Nothing read within 8 s leaves the transcript on the clipboard that long, never stale text in the target. - **Permissions.** Accessibility and Microphone grants are keyed to the code signature. `bundle.sh` signs with an Apple Development or Developer ID certificate when one is in the keychain; an ad-hoc signature changes on every build and resets both grants. - **Lost key-up.** When the 10 min watchdog fires, the coordinator transcribes and pastes as if the user had released. Discarding is probably the right behaviour; tracked separately. diff --git a/Sources/PladderSystem/PasteboardOutput.swift b/Sources/PladderSystem/PasteboardOutput.swift index 70800c0..78e275c 100644 --- a/Sources/PladderSystem/PasteboardOutput.swift +++ b/Sources/PladderSystem/PasteboardOutput.swift @@ -3,6 +3,7 @@ import ApplicationServices import CoreGraphics import Foundation import PladderCore +import os /// Inserts text by putting it on the general pasteboard and synthesising Cmd+V, /// then putting the user's clipboard back. @@ -17,6 +18,16 @@ import PladderCore /// cancelling a dictation must never yank the pasteboard out from under an app /// that has not read it yet. /// +/// The restore waits for the paste as well as for a clock. The transcript goes +/// on the pasteboard as a promise, so the target app's read comes back to us as +/// a call to `TranscriptPromise`. The old clipboard returns `restoreFloor` after +/// Cmd+V, as it always has, if the transcript has been read by then, and +/// otherwise `readSettle` after the read. An app that is busy when Cmd+V +/// arrives reads late, and the fixed delay alone handed it the user's old +/// clipboard instead of the transcript. If nothing reads it, `restoreCap` ends +/// the wait, which leaves the transcript on the clipboard a little longer and +/// never pastes stale text. Issue #40 has the measurements. +/// /// This is an actor because the pending restore is shared mutable state: a second /// `insert` may start while the previous restore is still waiting. /// @@ -24,13 +35,31 @@ import PladderCore /// managers that honour them keep dictations out of their history — the whole /// point of an app whose text never leaves the Mac. public actor PasteboardOutput: TextOutput { - /// How long to wait after Cmd+V before restoring the previous clipboard. + /// The earliest the previous clipboard comes back after Cmd+V, read or not. /// /// The paste is asynchronous from our point of view: the target app reads the - /// pasteboard on its own run loop some time after it receives the key event. - /// Restoring too early gives the app the *old* contents. 400 ms is generous - /// enough for slow Electron apps while still feeling instant to the user. - public let restoreDelay: Duration + /// pasteboard on its own run loop some time after it receives the key event, + /// 0 to 25 ms later for every app measured, a second or more for a web page + /// whose main thread is busy. Restoring before the read gives the app the + /// *old* contents. + /// + /// A read is not always the paste: Chromium sometimes reads once as Cmd+V + /// arrives and again when the page gets round to pasting, and the + /// pasteboard keeps the data after the first read, so the second never + /// reaches us. Nothing says which read is which, so a read never brings + /// the clipboard back sooner than this. 400 ms is the fixed delay this + /// replaced, which is enough for every app that is not busy. + public let restoreFloor: Duration + + /// How long after the target app's last read the previous clipboard comes + /// back, when that is later than `restoreFloor`: a busy app that read late. + public let readSettle: Duration + + /// How long after Cmd+V the previous clipboard comes back when nothing has + /// read the transcript: a paste into something that takes no text, or an + /// app that never got the key event. The transcript stays on the clipboard + /// until then, which is the safe way to be wrong. + public let restoreCap: Duration /// How long to wait between Cmd+V and the Return keystroke when the caller /// asked for submit. The target app handles the paste on its own run loop @@ -71,8 +100,16 @@ public actor PasteboardOutput: TextOutput { /// The change count our write produced, i.e. what the pasteboard must /// still be at for the restore to be safe. let changeCount: Int - /// The detached task waiting out `restoreDelay`. Nil while the paste is - /// still being posted. + /// The transcript on the pasteboard, which reports every read. Held + /// here so it outlives the paste whatever the pasteboard item does. + let promise: TranscriptPromise + /// When Cmd+V was posted. Nil while it is still being posted; a read + /// before then is not the paste and does not count. + var posted: ContinuousClock.Instant? + /// The last read of the transcript since Cmd+V. + var lastRead: ContinuousClock.Instant? + /// The detached task waiting for the restore to fall due. Nil while + /// the paste is still being posted. var task: Task? } @@ -84,12 +121,18 @@ public actor PasteboardOutput: TextOutput { /// on the release-to-paste path. private var prepared: (snapshot: Snapshot, changeCount: Int)? + private static let log = Logger(subsystem: "de.dinooo13.pladder", category: "paste") + public init( - restoreDelay: Duration = .milliseconds(400), + restoreFloor: Duration = .milliseconds(400), + readSettle: Duration = .milliseconds(200), + restoreCap: Duration = .seconds(8), submitDelay: Duration = .milliseconds(50), propagationDelay: Duration = .zero ) { - self.restoreDelay = restoreDelay + self.restoreFloor = restoreFloor + self.readSettle = readSettle + self.restoreCap = restoreCap self.submitDelay = submitDelay self.propagationDelay = propagationDelay } @@ -104,6 +147,14 @@ public actor PasteboardOutput: TextOutput { if let key = await MainActor.run(body: { KeyboardLayout.commandVKeyCode() }) { pasteKey = key } + // While our own transcript is still on the pasteboard the user's + // clipboard is the pending snapshot, which `insert` carries forward. + // Capturing would only read our own promise, from off the main + // thread, which AppKit warns against. + if let pending, pending.changeCount == NSPasteboard.general.changeCount { + prepared = nil + return + } prepared = (Snapshot.capture(), NSPasteboard.general.changeCount) } @@ -144,8 +195,11 @@ public actor PasteboardOutput: TextOutput { } } - let ourChangeCount = Snapshot.write(text) - pending = Pending(snapshot: snapshot, changeCount: ourChangeCount, task: nil) + let promise = TranscriptPromise(text) { [weak self] promise, instant in + Task { await self?.transcriptRead(promise, at: instant) } + } + let ourChangeCount = Snapshot.publish(promise) + pending = Pending(snapshot: snapshot, changeCount: ourChangeCount, promise: promise) do { if propagationDelay > .zero { @@ -172,29 +226,71 @@ public actor PasteboardOutput: TextOutput { // A concurrent `insert` may have superseded us across the sleep above; it // owns the snapshot now and will schedule its own restore. - guard pending?.changeCount == ourChangeCount else { return .pasted } - pending?.task = restoreTask(for: ourChangeCount) + guard pending?.promise === promise else { return .pasted } + pending?.posted = .now + scheduleRestore() return .pasted } - /// Waits out `restoreDelay` off the critical path, then hands back to the - /// actor to do the restore. Detached so the caller's cancellation — a + /// When the pending restore falls due: `settle` after the last read since + /// Cmd+V but no sooner than `floor` after Cmd+V, or `cap` after Cmd+V if + /// nothing has read it, and never later than that. + static func restoreDue( + posted: ContinuousClock.Instant, + lastRead: ContinuousClock.Instant?, + floor: Duration, + settle: Duration, + cap: Duration + ) -> ContinuousClock.Instant { + let latest = posted + cap + guard let lastRead else { return latest } + return min(max(posted + floor, lastRead + settle), latest) + } + + /// (Re)starts the pending restore's wait from what is known now. + private func scheduleRestore() { + guard let current = pending, let posted = current.posted else { return } + current.task?.cancel() + let due = Self.restoreDue( + posted: posted, lastRead: current.lastRead, floor: restoreFloor, settle: readSettle, cap: restoreCap) + pending?.task = restoreTask(for: current.promise, at: due) + } + + /// The target app read the transcript. Called from the main thread, where + /// AppKit serves promises, by way of a task. + private func transcriptRead(_ promise: TranscriptPromise, at instant: ContinuousClock.Instant) { + guard let current = pending, current.promise === promise, let posted = current.posted else { return } + if current.lastRead == nil { + let after = (instant - posted).components + let seconds = Double(after.seconds) + Double(after.attoseconds) / 1e18 + Self.log.notice("clipboard read \(seconds, format: .fixed(precision: 3)) s after Cmd+V") + } + pending?.lastRead = instant + scheduleRestore() + } + + /// Waits for the restore to fall due off the critical path, then hands back + /// to the actor to do it. Detached so the caller's cancellation — a /// cancelled dictation — cannot make the restore fire early, which would give /// the target app the user's old clipboard instead of the transcript. - private func restoreTask(for changeCount: Int) -> Task { - Task.detached(priority: .utility) { [restoreDelay] in - try? await Task.sleep(for: restoreDelay) - // Only a *newer* insert cancels this task, and it takes ownership of - // the snapshot when it does, so there is nothing left to restore. + private func restoreTask(for promise: TranscriptPromise, at due: ContinuousClock.Instant) -> Task { + Task.detached(priority: .utility) { + try? await Task.sleep(until: due, clock: .continuous) + // A read that moves the deadline, or a *newer* insert, cancels this + // task; the newer insert takes ownership of the snapshot, so there + // is nothing left to restore. guard !Task.isCancelled else { return } - await self.completeRestore(changeCount: changeCount) + await self.completeRestore(promise) } } - private func completeRestore(changeCount: Int) { - guard let pending, pending.changeCount == changeCount else { return } + private func completeRestore(_ promise: TranscriptPromise) { + guard let pending, pending.promise === promise else { return } self.pending = nil - pending.snapshot.restore(ifChangeCountIs: changeCount) + if pending.lastRead == nil { + Self.log.notice("clipboard not read within \(self.restoreCap.components.seconds) s of Cmd+V; restoring") + } + pending.snapshot.restore(ifChangeCountIs: pending.changeCount) } /// Sends one key down/up to the HID event tap, i.e. the same place a real @@ -257,21 +353,43 @@ public actor PasteboardOutput: TextOutput { /// Replaces the pasteboard with `text`, marked as a transcript nobody /// should archive, and returns the resulting change count so we can tell - /// later whether anybody else has written since. + /// later whether anybody else has written since. For the clipboard-only + /// path, where the transcript is the result and stays. /// /// One item carrying five types, so it is still a single write: the /// markers cost a few bytes of IPC, not a second round trip. static func write(_ text: String, to pasteboard: NSPasteboard = .general) -> Int { - let item = NSPasteboardItem() + let item = markedItem() item.setString(text, forType: .string) + pasteboard.clearContents() + pasteboard.writeObjects([item]) + return pasteboard.changeCount + } + + /// `write`, with the text served by `promise` when an app reads it, so + /// the read is reported. Still a single write. + /// + /// The markers matter twice here. Something in macOS reads every + /// unmarked write about 15 ms after it lands, paste or no paste + /// (Universal Clipboard, by the log); with the markers it does not. + /// Without them that read would count as the paste and bring the old + /// clipboard back before the target app had read the transcript. + static func publish(_ promise: TranscriptPromise, to pasteboard: NSPasteboard = .general) -> Int { + let item = markedItem() + item.setDataProvider(promise, forTypes: [.string]) + pasteboard.clearContents() + pasteboard.writeObjects([item]) + return pasteboard.changeCount + } + + private static func markedItem() -> NSPasteboardItem { + let item = NSPasteboardItem() for type in markerTypes { item.setData(Data(), forType: type) } // `Bundle.main.bundleIdentifier` is nil for the bare SwiftPM binary // and is the harness's id under `swift test`; the app's own id is // the fallback. item.setString(Bundle.main.bundleIdentifier ?? "de.dinooo13.pladder", forType: sourceType) - pasteboard.clearContents() - pasteboard.writeObjects([item]) - return pasteboard.changeCount + return item } /// Puts the snapshot back, unless the user copied something else while we @@ -291,3 +409,31 @@ public actor PasteboardOutput: TextOutput { } } } + +/// The transcript as a pasteboard promise: the text is handed over when an app +/// asks for it, and every ask is reported, which is how `PasteboardOutput` +/// knows the paste happened. +/// +/// AppKit calls the provider on the main thread, so an app reading the +/// transcript waits for Pladder's main thread to be free. The time from Cmd+V +/// to the first read is logged for that reason. +final class TranscriptPromise: NSObject, NSPasteboardItemDataProvider, Sendable { + let text: String + private let onRead: @Sendable (TranscriptPromise, ContinuousClock.Instant) -> Void + + init(_ text: String, onRead: @escaping @Sendable (TranscriptPromise, ContinuousClock.Instant) -> Void) { + self.text = text + self.onRead = onRead + } + + func pasteboard( + _ pasteboard: NSPasteboard?, + item: NSPasteboardItem, + provideDataForType type: NSPasteboard.PasteboardType + ) { + item.setString(text, forType: type) + onRead(self, .now) + } + + func pasteboardFinishedWithDataProvider(_ pasteboard: NSPasteboard) {} +} diff --git a/Tests/PladderSystemTests/PasteboardOutputTests.swift b/Tests/PladderSystemTests/PasteboardOutputTests.swift index 37509bb..2c99c23 100644 --- a/Tests/PladderSystemTests/PasteboardOutputTests.swift +++ b/Tests/PladderSystemTests/PasteboardOutputTests.swift @@ -76,4 +76,78 @@ import Testing #expect(pasteboard.string(forType: .string) == "something the user copied") } + + // MARK: The transcript as a promise + + /// Records every read the promise reports. + private final class Reads: @unchecked Sendable { + private let lock = NSLock() + private var _count = 0 + var count: Int { lock.withLock { _count } } + func record() { lock.withLock { _count += 1 } } + } + + /// Main actor: AppKit serves an in-process promise read synchronously and + /// warns when that happens off the main thread. + @MainActor @Test func publishedTranscriptIsServedMarkedAndReportsTheRead() throws { + let pasteboard = Self.namedPasteboard() + defer { pasteboard.releaseGlobally() } + let reads = Reads() + let promise = TranscriptPromise("hello") { _, _ in reads.record() } + + let changeCount = PasteboardOutput.Snapshot.publish(promise, to: pasteboard) + + #expect(changeCount == pasteboard.changeCount) + #expect(reads.count == 0) + let types = Set(try #require(pasteboard.pasteboardItems?.first).types) + #expect(types.isSuperset(of: [.string, Self.transient, Self.concealed, Self.autoGenerated, Self.source])) + #expect(pasteboard.string(forType: .string) == "hello") + #expect(reads.count == 1) + } + + @MainActor @Test func restoreRoundTripsAfterAPublishedTranscript() { + let pasteboard = Self.namedPasteboard() + defer { pasteboard.releaseGlobally() } + pasteboard.clearContents() + pasteboard.setString("user text", forType: .string) + let snapshot = PasteboardOutput.Snapshot.capture(from: pasteboard) + + let promise = TranscriptPromise("transcript") { _, _ in } + let ourChangeCount = PasteboardOutput.Snapshot.publish(promise, to: pasteboard) + #expect(pasteboard.string(forType: .string) == "transcript") + snapshot.restore(ifChangeCountIs: ourChangeCount, on: pasteboard) + + #expect(pasteboard.string(forType: .string) == "user text") + } + + // MARK: When the old clipboard comes back + + private static let posted = ContinuousClock.now + private static func due(read: Duration?) -> Duration { + PasteboardOutput.restoreDue( + posted: posted, lastRead: read.map { posted + $0 }, + floor: .milliseconds(400), settle: .milliseconds(200), cap: .seconds(8)) - posted + } + + @Test func nothingReadWaitsForTheCap() { + #expect(Self.due(read: nil) == .seconds(8)) + } + + @Test func aPromptReadKeepsTheOldDelay() { + #expect(Self.due(read: .milliseconds(12)) == .milliseconds(400)) + } + + @Test func aLateReadIsWaitedFor() { + // A busy page reads a second after Cmd+V; the old 400 ms delay alone + // had already put the user's clipboard back by then. + #expect(Self.due(read: .milliseconds(1_006)) == .milliseconds(1_206)) + } + + @Test func aReadNearTheCapStillEndsAtTheCap() { + #expect(Self.due(read: .milliseconds(7_900)) == .seconds(8)) + } + + @Test func aReadBeforeCmdVKeepsTheOldDelay() { + #expect(Self.due(read: .milliseconds(-5)) == .milliseconds(400)) + } } diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 960c1ec..fbee6df 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -308,3 +308,11 @@ reported a held key as released and pressed again, and every release since then has waited 50 ms before `recordingStopped`. That wait is felt but is not in the `release-to-paste` number. A latched recording's release-to-paste runs from the closing press. + +A `clipboard read 0.008 s after Cmd+V` line in the `paste` category follows +each paste: when the target app read the transcript, measured from Cmd+V, so it +comes on top of `release-to-paste`, which ends when Cmd+V is posted. The transcript is +served from Pladder's main thread, so a value well above the usual few +milliseconds in an app that normally reads fast means that thread was busy. +`clipboard not read within 8 s of Cmd+V; restoring` means nothing read it: +the paste went somewhere that takes no text. diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 68e0284..9ce7736 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -70,7 +70,9 @@ paste, it happens somewhere else. `AVAudioEngine.pause()` blocks until the current device buffer completes. The samples are already complete when it starts, so the orange indicator goes off a few milliseconds later and nobody waits for it. -- **The old clipboard is restored after the paste**, not before it. +- **The old clipboard is restored after the paste**, not before it, and + and not before the target app has read the transcript. The transcript is + a pasteboard promise, so the read reports itself. - **The Return for the send key is posted from a detached task**, 50 ms after Cmd+V.