Repository navigation
Record a flow in Chrome, narrated, straight into Intelligence with the OpenBot Recorder - #809
Closed
davidmckayv wants to merge 326 commits into
Closed
davidmckayv wants to merge 326 commits into
davidmckayv wants to merge 326 commits into
Conversation
…sonless sign-out as revoked
… keep the live retry when another recording drains
…ever fail the page on it, and hand disconnect listeners the wrapped port
…pairs NUL characters before sending
… label click, none for an IME confirm, and one typing step per field visit
…ropping steps or digests
…t, and export the meta type and key
…nd decodes like Jason
…ges copy patches/ before bun install (and into the production tree), and OpenBot Desktop copies it into a deployment, since bun install --frozen-lockfile stops with Couldn't find patch file without it; a test holds every patch package.json names to Desktop's copied tree
…d field's copies masked with its value, CopilotKit's capture batches never a save, OpenBot's own chat runs captured
…'s TrajectoryConnection asking its runtime for a capture pass (the REST trajectory/<id>/connect route, or the single route's trajectory/connect envelope) is the page setting up its report on itself, and its answer carries a join token
…ad.linked shape and place, the frontend tool lifecycle, no Intelligence metadata in agent.event, and a CopilotKit page's clicks carrying its Thread
…an agent.event over Intelligence's budget has only its bulky text cut, the longest first, so OpenBot's RUN_STARTED (its 39 tools and 5 context entries, about 95 KiB) still joins its Thread and run and keeps every tool name whole End-to-end run 3 stored that RUN_STARTED with every string cut to 23 characters, its threadId, runId and tool names among them, because the agent events' opt-in string trimming capped every string in the event to one common length. trimEvent's strings mode now takes the fields it must keep whole (ids, the names of a tool, a step or an activity, and the enum-like fields: types, roles, JSON Patch operations and paths, a schema's required names and enum values), and cuts only the rest. How every other event trims is unchanged. The regression test replays the run's real RUN_STARTED, as Intelligence's socket delivered it, through the agent events.
…t knows its page runs CopilotKit and its clicks carry the open Thread, threadId null before there is one
End-to-end run 3's clicks in OpenBot's chat carried no threadId at all. The page asks GET /api/copilotkit/info and OpenBot's runtime answers with CopilotKit's RuntimeInfo, but the debugger capture ignored that call with the rest of OpenBot's own API, so the agent events never learned the page runs CopilotKit. The capture now follows OpenBot's runtime info route (and the single route's { method: "info" } form, should the page's transport fall back to it) for that call alone: its answer is read only to say the page runs CopilotKit, never as AG-UI or a run, and every other call to OpenBot stays ignored. The page hook needed no change: it already reported this answer, and the page-hook parity test now replays OpenBot's chat to hold the two captures to the same observations.
…nBot's Send is a form's submit button, so its click now names the Thread of the run it sent In end-to-end run 3 the form's submit arrived a millisecond after the Send click and closed the click's span, so the run Send started fell in the submit's span and the click named no Thread. A submit that follows a click within 50 ms is the click's own default action, so what the submit started counts as the click's for its agent context. Only the agent context reads this; a save in the submit's span stays the submit's. The regression test records OpenBot's chat from its real CDP recording through the network capture, the agent events and the steps, with the content script's own capture of the editor click and of the Send click with the submit it makes.
…ry id and tool name whole, and OpenBot's own chat page is a CopilotKit page whose Send names its Thread
…now a one-time code field by a label that says the code was texted or sent to the phone In the full end-to-end run the admin password typed on /checkout reached Intelligence from /account, typed into "Note to support" and pasted into "Internal memo", because each content script masks with the digests of its own document only. The service worker already holds every digest of the recording, so it now masks each content event again with them (background/content-masking.ts): the edit's value and text, the attributes, label and name that copy it, its selectors and the rest of the step, whatever page, tab or frame it came from. Page-change addresses take the digests too, and a value pasted whole now matches its digest even when it holds spaces. The click's private agent target is left as the page wrote it. Two one-time codes leaked the same run: 582046 in a field labelled "Enter the code we texted you" and 315827 in one labelled "Enter what we sent to your phone", the second in its save's body and answer as well. Neither has autocomplete=one-time-code, a code-like name or a numeric shape. Chromium's autofill (ONE_TIME_CODE: otp, verification code, 2fa, six digit), Bitwarden's TotpFieldNames, KeePassXC-Browser's accepted OTP fields and Proton Pass's TWO_FA and OTP dictionaries all name the code, none the delivery, so the rule now also takes a field whose words say the site just sent what it takes (texted, emailed, sent to you or your phone, in the text message), names no ordinary thing sent (an order, tracking, invoice or booking number), and holds 4 to 8 digits, is numeric, takes 4 to 8 characters or sits on a sign-in or verification page. The content script judges it with the value typed, so the code's digest masks the save. The masking reads an edit's accessible name as its label, so the service worker judges the same words, and a value it redacts there has its digest kept for everything after. The regression tests drive the run's own checkout, account and security pages through the real content script in happy-dom and the real service worker, with its real network capture on the fake debugger, and pin the rule on the run's real rows.
… and the recording's start in time order, and fix the stale popup and page notices The end-to-end run lost steps whenever Chrome stopped the service worker or the browser quit: the Archive order click and its PATCH (run 5, worker stopped 600 ms after the click), the Send refund submit and its POST (run 4), and in Edge the last click and typing before quitting. The step queue lived only in the worker's memory. steps.ts now has snapshot() and restore(), with an onChange hook, and its rules are unchanged: a restored step waits out what is left of its window, a page change looks its page up again. recording.ts writes the snapshot to storage.local as it changes (at once, then at most every 100 ms), takes it up when a restarted worker loads, and after a browser quit hands the held steps to the recording's drain, so the document that recovers it puts them in the outbox ahead of the interrupted outcome. storage.session would not outlive the quit, and these are the same masked events the outbox already keeps on disk. Saves still in flight are not kept: the capture that would report their end went with the old worker. Every narrated trajectory had rows out of time order. A phrase's transcript is stamped with its start but exists only when the phrase ends, 4 to 9 s later in run 1b, and it went straight to the stream. The document now tells the service worker when a phrase begins, the steps from then on wait in the queue (15 s from the phrase's start at most), and the transcript is placed among them by its start. At Stop the document hands over the phrase being said before the held steps are sent. recording.started and the started outcome are stamped when Record began, so they no longer come after the recording's first page. Start over now ends the old trajectory with reason restarted and a recording.restarted event naming the new trajectory, whose recording.started names the one it replaced. The popup no longer says Streaming together with the retry's reason once Intelligence takes events again, and no longer keeps "kept in this browser until you connect again" after the person connected again. The 50-action notice still open shows again on the page the tab moves to. In Edge the voice sentences name Edge's speech service, not Chrome's. Regression tests run the real service worker and the real offscreen document over a fake Intelligence with rows from the end-to-end run: the run 5 restart, the Edge quit (with run 4's typing), run 1b's narrated timings, and Stop mid-phrase; each fails on the code before this change.
…here in a recording is masked everywhere after it, and one-time codes in fields that only say a code was sent are masked
…he minute In end-to-end run 6, Settings, Browsers listed three rows that all read "Google Chrome on macOS" (three Chrome profiles), and none said which was the browser the page was open in. This browser's own copy of the Recorder already tells the page its install id, and every connection carries one, so the page now shares it between the two sections and the matching row reads "Google Chrome on macOS · This browser", its Disconnect named the same way. A row now gives the date and time it connected rather than "2 hours ago", which two copies connected a minute apart both said; last used stays relative. The server already returned everything needed.
…r an Enter CopilotKit's click already records adds nothing, a list's choice is a select step and a box ticked a check step, CopilotKit's target.role stays its own, and selectors rank as Playwright's generator ranks them End-to-end run 1 turned single actions into two or three rows: a click on a submit button gave a click and a submit (the save on the submit), and Enter in a form with a button gave a key, the keyboard click Chrome makes on the default button, and a submit. The Recorder's own submit and key now appear only when CopilotKit's collector records nothing of the action: a submit in the same task as a click the collector sent is that click's default action, and an Enter in a field whose form has a default button is held until its keypress has run, then dropped if Chrome's click came (CopilotKit's keyboard click is the row) or sent if not (a page that stops the Enter, a form with no button, where the key is the row and its submit adds nothing). Every event CopilotKit sends is still sent exactly as it sends it: a box ticked stays its click and its input, and a double click its clicks. CopilotKit main sends every list and box edit with eventType input, so stepKindOf now reads the kind from the control: a select is a select step, a checkbox or radio a check step. A select step names its chosen options as the person read them (target.selectedLabels) beside CopilotKit's value, selectedValues and text, which stay as CopilotKit wrote them. The Recorder's computed role moves to target.ariaRole, so CopilotKit's target.role (the role attribute) is no longer overwritten. Selectors follow Playwright's selector generator: test attribute, role and name, text, a hand-made id, the name attribute, then a short CSS path, each that also finds another element ranked after every one that finds this alone. An id Playwright calls GUID-like (Wikipedia's mwGQ, which led run 2's link selectors) is never a selector or a path anchor. A link's stable address (no session, token, clock or random part) is a selector above its position, as Selenium IDE ranks its href locator, so OpenBot's unnamed new-chat link is found by a[href="/channel/new"]. A field in a cross-site iframe names that iframe by the name it gave the frame, else the address it loaded (with which one when two load it), else its position, where it was just "iframe". The regression tests play Chrome's own event sequences on the shop's and the sites' real markup, with a bare CopilotKit main collector on the same page held to the same events and fields.
…ng link clicks: a repeated write is a save, a one-time code verified, confirmed or redeemed is a save, a form posted as a page load and redirected is the save of the click that sent it, and a typed address is its own page change Run 5's second Archive order (the same PATCH and body) carried nothing because the repeat screen dropped it: it now screens reads only, as polling and refetches read, while a write is the person's each time. The shop's /api/mfa/verify, /api/phone/confirm and /api/backup/redeem carried nothing because their codes are redacted on both sides and none of verify, confirm or redeem was a word for a save; they now are. the-internet's POST /authenticate and POST /forgot_password are answered with 303, and the redirect's requestWillBeSent replaced the POST with the GET it led to, so the post was never a call. A write redirected to a read is now finished as its own call when the redirect comes, and a form the top frame posts as a page load is a save whatever its address. Link clicks waited 10 to 13 seconds whenever the page posted telemetry on the click: the write was counted in flight before it was screened, the page change cancelled it, and Chrome never reported it once the old document was gone. A write the cheap screens drop (telemetry, a prefetch) is never in flight, and the top frame's next document being answered ends every write the old page still had in flight. A navigation now carries Chrome's transitionType and transitionQualifiers beside CopilotKit's navigationType, so an address typed into the address bar says so, and a typed, bookmarked or picked address is never folded into the page change before it as a redirect. A redirect that is folded keeps how its chain began. The regression tests drive the network capture and the steps with Chrome's CDP sequences for each case (the 303 redirect's second requestWillBeSent, a cancelled write with no loadingFailed) and webNavigation's details.
…ge, and cut them before the conversation when a RUN_STARTED is still over budget End-to-end run 3: every RUN_STARTED carried OpenBot's same 39 tool schemas and 5 context entries (61,104 of the trajectory's 82,300 bytes), so each was cut to fit, every string to about 21 characters, and with them the only copy of each frontend tool's result: list_bots's answer left as "3 coworkers already e". A RUN_STARTED's input now leaves out tools and context that equal what the Thread's runs carried before, as its history already leaves out messages an event carried. When one must still be cut, the text of its tools and context goes first, to the longest one length that fits, so the person's messages and a frontend tool's result, which CopilotKit announces nowhere else, stay whole. A backend tool's result is unchanged: its agent.message carries the text whole.
…ick take the Thread of the run it sent even when the page runs it late End-to-end runs 1 and 2: after New thread, the Recorder linked the new Thread with reason "open" where CopilotKit's capture said "switch", gave the person's first message there runId null where CopilotKit gave the agent's last run on the Thread before, and the chat box and Send both named the Thread the chat had left. Run 3: OpenBot's first Send named no Thread, since its run came 2.18 s after the click (the composer routes the message first) and the click had left the steps a second after it. Each agent's chat in a page is now on one Thread at a time, as CopilotKit's open-Thread registration is: its first Thread in a page view is linked "open", a later one "switch", and the Thread it moved from is no longer open. The run id CopilotKit gives a message, a tool call or an error with no run of its own is the page agent's last run, shared by the agent's Threads in the tab, as LearningBridge's lastRunIds is keyed by the agent the page keeps across Threads; a new document starts it afresh, and a replayed run sets it as the page's agent applies the replay. A click with the open-Thread context takes the Thread of the first run the page asks for in its join: after the click, before the person's next step in the tab (its form's submit is its own), and within 4 s (the steps' horizon and settle, as long as a step nothing follows is already held, against run 3's 2.18 s). The steps hold such a click until its run comes or its join closes. A click whose Thread the chat left after it, by no doing of the click's, names none: New thread calls nothing, so the traffic cannot say when the chat left it, and a stale Thread is never given.
…d: held steps survive a restart, narration and the start in time order, notices that match the state, Start over linked, this browser marked
…s own submit and key fold into CopilotKit's events, select and check steps, repeated writes, code submissions and form posts saved, no long holds, steadier selectors, and typed addresses kept # Conflicts: # extension/src/background/navigation.ts # extension/src/background/steps.ts # extension/tests/steps.test.ts
…t's move to a new Thread followed as CopilotKit does, a late run still naming its Send, a Thread's tools and context carried once, and CopilotKit's run id carried across Threads # Conflicts: # extension/src/background/steps.ts # extension/tests/agent-events-page.test.ts
The sensitive-field rule ran on every clicked element, so a control that only names a secret ("Retrieve password", "Show password", "Reset password link") lost CopilotKit's own copy of its text, and its words became a digest masked everywhere after it. The rule now applies only to an element that holds what the person enters (an input not drawn as a button or a choice, a text area, a select, an editable region or a text-entry role), as CopilotKit's capture tells one, in the masking and in the content script's tracker. Password values, a password field's own copies after Show password turns it to text, cards, codes and digests of typed secrets stay masked.
A step left for the document and Intelligence a moment before the browser quit, while the held steps in storage still listed it (their next write comes after the step is handed on), so the restarted browser sent it again: the typed name in the Edge restart reached Intelligence twice with one timestamp. Each held item now has a key, kept in the snapshot and sent with it; the document notes the key in the same IndexedDB transaction that stores the event, and does not store a key its recording's trajectories already took. A step still held at the quit has no note and is sent once, as before. The same holds when Chrome stops only the service worker.
…each step's tab A tab opened from a followed tab took its first page change's `from` from its own address (the new tab's URL, which onCreatedNavigationTarget gives), so the account opened from /support read from=/account to=/account. Its first recorded page change now comes from the opener tab's top page at the moment Chrome reported the new tab. Steps from different tabs could be told apart only by address: every step and page change the Recorder sends now carries `tab`, the tab's number in the recording (1 for the tab it started in, then 2, 3 in the order tabs join), never Chrome's tab id. It is stamped as the step enters the held steps, so it survives a restart or a quit, and CopilotKit's own fields stay as the page sent them.
The agent events kept their per-tab state in the worker's memory only, so after Chrome stopped the worker (run 4, 300 ms after a click) the New thread, chat box and Send clicks carried no threadId at all, and the Send not the Thread its run started. What a click's context and the links depend on (that the page runs CopilotKit, the Thread each chat has open and the Threads it left, the Thread linked last, the latest run requests, each agent's last run, and the runs recorded per Thread) is now kept with the held steps, in the same storage write, and taken up before anything reaches the restarted worker's agent events. Ids and times only: the AG-UI streams and conversations are taken up again from the page's next traffic. A stale context must not send a link twice or lose one: a link takes its place in the kept context only as it goes to the steps, and when the context changed, the held steps are written before anything leaves, with the leaving items still in them, so the keys stop a second send.
…d keeps its text, a step that left is never sent again after a quit, a new tab's first page comes from its opener and every step names its tab, and a click's Thread context survives a restart
On the shop's checkout the person typed only the admin password and the card, then clicked Send refund. The page posted the refund 4 ms after the click, but the click carried no request: the masked body held both values only as [redacted], so it no longer carried anything the person typed, and "refunds" is no write word. A card-only payment or a login posted by fetch missed its save the same way. The service worker now keeps, per followed tab, the digests of what the person typed into sensitive fields there (the latest 64, in memory and storage.session as the recording's digests are, cleared at Stop). When the network capture masks a write, it reads the unmasked body for a value matching one of that tab's typed secrets, where the masking itself looks for one, and marks the exchange when it finds one. Nothing is unmasked: only the mark goes on to the steps, and it never reaches the request a step carries. saveQuality counts a marked POST as carrying the person's values, so it is the click's save. Everything else is unchanged: reads, GraphQL queries and the page's own chatter are still never a save. The regression test replays the final run's checkout rows through the real content script on the shop's checkout markup, the real worker and the real network capture, with the refund POST and telemetry the shop logged at that click. It failed before (Send refund carried no request) and passes now, with the password, the card and the shop's tokens masked throughout. A rule test pins the new check against the run's real bodies.
…rets is still the step's save
# Conflicts: # desktop/src-tauri/src/deployment.rs
…the page receives, in the shape the CopilotKit SDK emits, clicks and typing from CopilotKit main's capture code carried as a patch, and the gateway setting capture needs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds the OpenBot Recorder, a Manifest V3 extension for Chrome and Edge that captures a task from any site. A person clicks Record, does the task in their own browser while talking it through, and clicks Stop. The recording goes straight from the browser into CopilotKit Intelligence as a Product Trajectory, in that person's own Learning space, where Learning can turn it into a Skill.
A recording is what the person did and what their agent did, never the page's traffic. It is the data Intelligence needs to train Skills, evals and Insights, and nothing else:
transcriptevent with its start time, and no audio leaves the extension or is stored.agent.run,agent.message(the person's own messages too),tool.call(with a page tool's executing and completed),thread.linked, and a click inside a chat message with its Thread and message. What those leave out goes in asagent.event: a tool call's whole arguments and its result, state, reasoning bounds, errors, the agent's own custom events. Never one row per streamed token; a Thread's tools and context only when they change; CopilotKit's suggestion runs left out as its own capture leaves them out.OpenBot receives nothing that was recorded. It does three things:
trajectory/<id>/connectanswers only the Recorder's bearer.user-plus a hash of the OpenBot user id) to the runtime's Learning selector. It creates the space when the browser connects, and again on a capture pass if it is still missing.Learning, per person, shared by choice:
Permissions are asked at install, so the first Record records. Website access is a required host permission, and a fresh install opens one setup tab that asks for the microphone. The install dialog shows the same warnings the debugger permission already implies.
Also in this branch:
@copilotkit/runtimemoves to 1.79.1 (AG-UI 1.0.2), for server-side Learning space assignment.@copilotkit/learning0.0.1 is patched with CopilotKit main's build (patchedDependencies,patches/), because npm's 0.0.1 predates main's click and typing capture. Both images and OpenBot Desktop copypatches/. The patch goes once main is published.shared/.Intelligence needs settings only, no code:
FF_LEARNING_TRAJECTORIESon, and a realtime gateway that can ask Intelligence whether an organization may capture;INTELLIGENCE_GATEWAY_WS_URLthat people's browsers can reach;INTELLIGENCE_SERVICE_TOKENwithINTELLIGENCE_PROJECT_IDon a local evaluation install, or a CopilotKit CLI login on managed Intelligence. The runtime key alone cannot create a person's space.Where it runs
New state that outlives a request? Postgres, in migration 0057:
recorder_connections: hashed bearer, label, revoked flag.recorder_connect_codes: hashed one-time codes, deleted on use or expiry.recorder_trajectory_bindings: trajectory to person.learning_person_spaces: each person's Learning space.learning_skill_sharing: each person's sharing mode.learning_open_skills: the individual Skills a person chose to share.Four per-process caches have TTLs and hold nothing that must be shared:
What happens on the second replica?
Anything serialised?
trajectory_idprimary key withON CONFLICT DO NOTHING, followed by an owner check.Anything fanned out to a browser? No. The extension streams to Intelligence's realtime gateway directly.
New listener, port, or schedule? No new port and no schedule.
/api/recorder/*and/api/settings/learning, behind the same ingress as the API.Boundary and audit
recorder.connected,recorder.revokedandrecorder.trajectory_refused;learning.sharing_changed,learning.skill_openedandlearning.skill_closed.externally_connectablemessages.Changelog
CHANGELOG.mdunderUnreleased. It covers:OPENBOT_RECORDER_EXTRA_EXTENSION_IDS(a bad value stops the server at boot);openai/*models behind a non-OpenAIOPENAI_BASE_URL(COPILOTKIT_OPENAI_API=responseskeeps the Responses API).Proof
Run against a real local Intelligence, in real Chrome 156 and real Edge 155. Nothing mocked: real sites, real CopilotKit apps on a real model, real OpenBot.
durationMs, a chat-box click before its first run (null where CopilotKit names the open Thread), and wherethread.linkedsits. OpenBot's chat links its trajectory to its Thread, its tool results land whole, and its Send names its Thread.transcriptevents in time order, and a spoken card number was stored as[redacted].failed:disconnected;restartedand links the new one;format:check,lint,typecheck,build,drizzle-kit checkand drift: all pass.test:ci: 8,606 tests, 0 failures.abandoned_project_picker_times_out_and_reaps_helper(passes on its own) andworker_exhaustion_does_not_adopt_healthy_survivors(needs port 3001, which is in use here).Known limits: