Skip to content

feat(im): Feishu and DingTalk chat Bots, proactive notifications, web-confirmed writes and data retention - #1114

Open
newexploreor wants to merge 34 commits into
mainfrom
feature/enterprise-im
Open

newexploreor wants to merge 34 commits into
mainfrom
feature/enterprise-im

Conversation

@newexploreor

Copy link
Copy Markdown
Collaborator

What changed

Chat Bots now cover three enterprise IM channels instead of WeCom only, following docs/proposals/enterprise-im.md (v3, stages A–H).

  • Feishu (self-built app) and DingTalk (enterprise internal app, Stream mode) private chats: account binding (two-phase, platform login plus a code sent back by the same sender), project queries answered as the bound user, enterprise lock (Feishu: admin confirms; DingTalk: admin enters the CorpId, first matching event locks it).
  • Writes from chat through web confirmation (网页确认, all three channels): the turn freezes write drafts and replies with a summary and a /bot-approve.html#draft=… link; only the bound user can open it, confirmation re-checks binding, project permission, expiry and parameters, and a draft runs at most once. Automatic writes stay WeCom only. New WeCom Bots default to read-only; existing ones keep automatic execution.
  • Proactive SOP notifications on Feishu and DingTalk (completed / failed / waiting) to a bound member, accepted only after a trial send is delivered; works for notification-only Bots. Feishu retries an unknown outcome with the same uuid; DingTalk stops at “delivery unknown” for manual confirm or resend. Expired DingTalk replies can be resent as a proactive private message.
  • Delivery reliability: database-clock leases with an epoch per owner, two-phase sends with fixed keys per generation, reply_unknown instead of guessing, manual confirm/resend on the record page, maintenance switch AGENTFLOW_BOT_WORKERS_ENABLED.
  • Secrets encrypted with AES-256-GCM (AGENTFLOW_CREDENTIAL_KEYS, -credential-action status|reencrypt); Compose generates a key.
  • Data retention: hourly cleanup removes text of finished turns, notifications and drafts plus the native CLI logs of Bot turns after 90 days and deletes finished records after 180 days (configurable, 0 keeps).

Manual: docs/im-channels.md. Final test record: docs/testing/im-final-test-record.md.

Scope

  • Platform core / API / runner
  • Public adapter / deployment
  • Classic
  • Portable
  • Shared behavior contract
  • Installer / CLI
  • Documentation only

Compatibility

User-visible changes (also in CHANGELOG.md):

  1. New WeCom Bots default to read-only; existing Bots keep automatic execution.
  2. WeCom fallback send: a failed stream is resent as a proactive message only when it provably did not go out; otherwise the reply shows “送达待核对”.
  3. Disabling a Bot cancels queued and running turns, keeps generated replies until re-enabled, and marks in-flight sends for review.
  4. Retention applies to existing WeCom records on the first start with workers enabled. Set AGENTFLOW_BOT_BODY_RETENTION_DAYS / AGENTFLOW_BOT_RECORD_RETENTION_DAYS first if longer retention is needed.

Upgrade needs a maintenance window (manual section 2): stop all old replicas → start the new version once with AGENTFLOW_BOT_WORKERS_ENABLED=false (startup DDL adds columns and tables) → run migrations/036_enterprise_im_channels.sql and 037_bot_binding_ownership.sql (idempotent) → configure keys and run -credential-action reencrypt → confirm retention settings → restart with workers enabled. After reencrypt, rolling back to a pre-IM build requires restoring the backup (rollback matrix in manual section 10).

New dependencies: github.com/larksuite/oapi-sdk-go/v3 v3.13.0 (Feishu long connection) and github.com/open-dingtalk/dingtalk-stream-sdk-go v0.9.1 (only its payload / chatbot data types; the connection loop is an in-repo Stream client because the SDK client can crash the process, reproduction in the separate probe module scripts/im-sdk-probe). Go notices regenerated.

Validation

Run locally on the merged code (origin/main@6f606cf merged in, no conflicts left):

go build ./... && go vet ./...                                   pass
bash scripts/build-chatbot.sh && go test -p=2 ./... -count=1    pass except environment-only failures that also fail on origin/main:
                                                                  cmd/agentflow TestSDNotifySendsReadyAndMainPID (socket path too long),
                                                                  internal/runtime 7 Docker tests (no Docker)
internal/repo MySQL tests on MySQL 8.4.13 and MariaDB 10.11      pass except TestMySQLReadsObserveExternalChangesWithoutReload (fails on base too)
Feishu fault tests F1–F6, DingTalk D1–D5 (MySQL 8.4.13)          pass
npm run coverage (webtest)                                        222 pass, 87.89% lines
python3 scripts/test-install.py                                   pass (50)
python3 scripts/check-public.py                                   pass
python3 scripts/collect-go-notices.py                             pass
go run golang.org/x/vuln/cmd/govulncheck@v1.9.0 ./...             no reachable vulnerabilities
scripts/im-sdk-probe: go test + govulncheck                       pass

Upgrade and rollback rehearsal on MySQL 8.4.13 from origin/main@6f606cf to this branch, including the first retention run and restore-backup rollback: see the final test record, section 3.

Real tenants: Feishu private chat (R1) and DingTalk private chat (R2) signed off on a staging deployment; stage G signed off without per-item evidence. Feishu/DingTalk web confirmation (W-01..W-04) not yet run in a real tenant.

Not run locally (left to CI): Go coverage gate, real-browser tests, scan-secrets.sh, Playwright e2e, local/scripts/*.

Safety

  • I did not include credentials, internal endpoints, production data, or organization-specific infrastructure.
  • I preserved user files and unrelated changes.
  • I updated generated Classic host bundles when applicable.
  • I documented third-party content and licensing changes when applicable.

yowucai added 30 commits November 18, 2026 03:28
Single-branch plan for Feishu and DingTalk on top of the WeCom Bot stack.
Version 3 rebases on origin/main@e89cda9 (migrations now 036/037) and fixes
the implementation contracts: database-time leases with a dialect helper,
physical enabled/version columns, lock order config -> connection ->
interaction/delivery, retry terminal states, disable semantics, separate
interaction and notification status enums (send_unknown), shared tenant
confirmation for Feishu and DingTalk, and the maintenance window order.
Introduce the Channel/Conn/PushSender contract and route the existing WeCom
long connection through it without changing user-visible behavior.

- channel.go: inbound event, reply route, outbound message and SendResult
  types; optional progress and card capabilities.
- channel_wecom.go: WeCom adapter wrapping dial/stream/cards; send errors
  are classified by evidence (not written / written without receipt /
  platform rejection).
- inbound.go: common validation and enqueue shared by all adapters.
- socket.request records the send phase while keeping error messages and
  context unwrapping.
- notification.New injects CLIProvider.Verify (signed bootstrap only) and a
  classified wecom-cli push into the WeCom channel.
- Fake channel test covers receive, duplicate redelivery, app mismatch,
  rejection, unknown outcome with reconnect, and reconnect after drop.
Separate module exercising larksuite/oapi-sdk-go v3.13.0 and
dingtalk-stream-sdk-go v0.9.1 under Go 1.26.11. Feishu Start returns on
context cancel; the DingTalk StreamClient panics with 'send on closed
channel' after a pong timeout followed by a late frame, which is why the
DingTalk adapter will run its own stream loop.
TestConnectionReconnectsAfterLocalClose incremented a plain int from
httptest handler goroutines, which go test -race reported on main as well.
Use atomic.Int64; the every-second-attempt rejection and the two-reconnect
assertion are unchanged.
Document the probe steps, the panic stack (client.go:165 'send on closed
channel' in a processLoop goroutine), the source lines that close the
channels while reader and pong goroutines still send, the other rejected
StreamClient behaviors, and the requirements for the stage F connection
layer. The probe stays a separate diagnostics module and fails once the
defect stops reproducing.
Stages B-D of docs/proposals/enterprise-im.md; they change the same repo
and service code paths and land together.

Stage B - configuration scope and credentials
- NotificationBot gains channel, write approval, config/credential
  versions and tenant state; IdentityScope/ReceiveSlot/LeaseKey keep WeCom
  keys byte-identical and scope new channels by configuration.
- notification_bots physical columns (channel, external_app_id,
  tenant_key, unique receive_slot, enabled, interaction_enabled, versions);
  saves are CAS on config_version, retry MySQL deadlock victims, and keep
  one receiving configuration per application.
- internal/credcrypt: AES-256-GCM keyring from AGENTFLOW_CREDENTIAL_KEYS,
  lf1:<kid>: envelope bound to configuration and channel; legacy plaintext
  stays readable; -credential-action status|reencrypt (re-entrant).
- dbNowUS dialect clock; leases and send windows are *_us columns written
  and compared on the database clock.
- bot_native_sessions mapping written before the CLI starts; a failed
  write stops the turn.
- AGENTFLOW_BOT_WORKERS_ENABLED maintenance switch; migrations 036/037
  backfill physical columns, project snapshots and WeCom binding owners
  (BINARY identifier comparisons; verified twice on MariaDB 10.11).

Stage C - fencing and delivery state machine
- bot_connections epoch per owner; every interaction write locks
  config -> connection -> interaction and checks owner, epoch and live
  lease; zero rows is ErrBotLeaseLost.
- ready -> sending -> replied | reply_failed | reply_unknown with a fixed
  send key, short send lease, 15s lease headroom and 10s call timeout.
  Not-delivered retries 3x (5s/30s/2m) then fails; unknown retries with
  the same key inside first send + 55 minutes only for idempotent
  channels, else reply_unknown. Expired routes fail with route_expired.
- Disable keeps ready replies, cancels queued/running and marks sending
  unknown; delete cancels unsent work, marks in-flight unknown and revokes
  the configuration's bindings. Notifications gain send_unknown.
- Manual confirm/resend APIs with action-key de-duplication; resend is a
  new generation and key. WeCom fallback send only after proven non-delivery.

Stage D - bindings and read-only policy
- Challenges carry config, channel and masked sender; confirmation must
  match the configuration; link issuance is rate limited.
- botExecutionUser re-reads configuration, binding and project role.
- Read-only exec mode: platform writes hidden and refused, external MCP
  writes and workspace tools refused, drafts never executed; new Bots
  default to read-only, legacy WeCom keeps auto.
- Natural-language Bot answers no longer need a terminal tool call.
Stage E of docs/proposals/enterprise-im.md (R1 Feishu private-chat loop).

- channel_feishu.go: Verify obtains a tenant access token (enterprise
  query is informational); Connect runs the official long connection
  (larksuite/oapi-sdk-go v3.13.0, auto reconnect off, service reconnects)
  and acknowledges an event only after persistence within 2.5s, returning
  500 for storage errors so Feishu redelivers; Reply uses Message Reply
  with the persisted send key as uuid. Transport and platform codes are
  classified (dial failure / rate limit / token: not delivered; other
  transport: unknown; other codes: rejected).
- Tenant confirmation: the first trusted event records a candidate without
  bumping config_version and answers with a fixed notice; a Bot manager
  locks it (CAS on configVersion) via /tenant/confirm; locking does not
  reconnect; other tenants are dropped; conflicts can be rediscovered.
- Inbound: private text only, group events counted and ignored, non-text
  and over-8000-character input get a notice without reaching the model.
- Bot management UI: channel picker (DingTalk shown as in development),
  write approval for WeCom, credential migration hint, enterprise
  confirmation step with the platform query, record-page confirm/resend
  actions. Bind page shows channel, Bot, project and masked sender.
- OpenAPI for Bot create/update/delete, tenant and manual actions;
  docs/im-channels.md (setup, minimum Feishu permissions, maintenance
  window, R1 acceptance checklist); CHANGELOG notes the WeCom read-only
  default, fallback-send and disable behavior changes; Go notices bundle.
…migration

Found by the MySQL 8.4.13 rehearsal and the multi-process fault test:

- Feishu ack: repository calls take no context, so a blocked database held
  the event callback past Feishu's 3s window. The sink now runs in its own
  goroutine and the callback answers 500 at 2.5s; the redelivery is stored
  once and processed once (fault test F2).
- 037: the WeCom owner backfill and report selected a non-aggregated
  column, rejected under ONLY_FULL_GROUP_BY (MySQL 8.4 default, error 1055).
- Bot natural answers: only plain text from a successful run is accepted.
  Protocol JSON whose drafts were filtered during parsing (for example
  "将创建需求") previously passed as the reply; it now reruns the terminal
  step like any draft attempt. Matrix test covers read-only and legacy auto
  Bots, progress-only, empty, protocol JSON, failed CLI and redaction.
- internal/botinteraction/feishu_fault_mysql_test.go: real MySQL, receiver
  child processes killed with SIGKILL, simulated Feishu platform with
  one-hour uuid de-duplication. F1 duplicate delivery, F2 blocked storage,
  F3 long-connection drop, F4 crash during the model turn (lease takeover),
  F5 crash with the reply in flight, F6 lost reply response.
- docs/im-channels.md: rollback, fault recovery and the natural-answer
  behavior change; CHANGELOG entry.
Fixed acceptance commit 57c5450; automated results; MySQL 8.4.13 upgrade and
rollback rehearsal with legacy data written by origin/main; natural-answer
behavior matrix with baseline comparison; fault tests F1-F6; R1-01..R1-13
status by real-environment criteria with the evidence collected so far and
what each item still needs.
The page heading, command palette entry and English copy still described
WeCom-only notification channels, which read as if Feishu were missing.
The channel picker itself is inside Add channel (step 1).
compose.yaml lists the platform environment explicitly, so
AGENTFLOW_CREDENTIAL_KEYS never reached the container and saving a Bot
secret failed with credential_key_missing even when .env set it.
AGENTFLOW_PUBLIC_URL (chat binding links) and AGENTFLOW_BOT_WORKERS_ENABLED
were missing as well.

- compose.yaml passes the three variables; the public URL falls back to
  LOOPFORGE_PUBLIC_URL.
- scripts/configure.sh generates a 32-byte key for new .env files and adds
  one to existing .env files that lack it; an existing key is never
  replaced.
- test-install.py covers generation, upgrade of an old .env, idempotence
  and the compose wiring.
R1 accepted by yowucai on 2026-11-18. Real-tenant pass: R1-01, R1-02,
R1-05; rehearsal pass: R1-13; remaining items not run individually and
waived by the acceptance owner, with automated and fault-test evidence.
scripts/check-public.py rejects internal identifiers in tracked files.
Stage F of docs/proposals/enterprise-im.md (R2 DingTalk private-chat loop).

- channel_dingtalk.go: own Stream connection layer on gorilla/websocket;
  only payload frames and the chatbot callback model of
  dingtalk-stream-sdk-go v0.9.1 are used. One reader, serialized writes,
  no channel closed under a sender; WebSocket ping every 60s with a read
  deadline; context-aware dial; no reconnect inside the adapter (the
  service reconnects with backoff); SYSTEM ping answered, disconnect
  reconnects; bot callbacks acknowledged after persistence within 2.5s,
  500 on storage errors, bounded concurrency.
- Verify via /v1.0/oauth2/accessToken; endpoint via connections/open with
  a no-redirect, bounded HTTP client.
- Events: senderStaffId identity, sender/bot corp must match, robotCode
  checked when present; session webhook accepted only for
  https://oapi.dingtalk.com/robot/sendBySession.
- Session webhook sealed with the credential keyring, bound to the
  interaction (AAD), cleared on replied, route_expired and Bot delete;
  expiry in ms -> route_expires_at_us. No idempotency key: unknown
  outcomes stop at reply_unknown.
- Enterprise scope: administrator-entered CorpId; the first matching
  trusted event locks it (config_version+1, attributed to that admin),
  another enterprise is a sticky conflict until corrected or rediscovered.
- API tenantKey (tenant_required, immutable once locked); UI CorpId field
  and automatic confirmation step; channel page copy.
- Tests: event mapping, stream ping/ack/timeout/disconnect/late-frame/
  close/cancel (-race), webhook classification and no redirects, service
  loop, repo tenant/seal, HTTP save, webtest; multi-process MySQL fault
  test D1-D5 (duplicate, blocked storage, silent gateway + late frame,
  crash in send + manual resend, expired webhook).
Fixed acceptance commit f9c94e0, automated results, fault tests D1-D5 and
the R2-01..R2-12 real-environment checklist (not run: no DingTalk tenant
yet), including the facts to record during the pilot.
DingTalk and Feishu replies are plain text, so the Markdown link syntax
was shown verbatim and the auto-linker kept the closing bracket in the
URL, which made the binding page reject the token. Put the URL on its
own line for these channels and make the page extract the 64-hex token
tolerantly. Page title no longer says WeCom.
Stage G, part 1. Feishu (im/v1/messages to open_id, send key as uuid)
and DingTalk (robot oToMessages/batchSend, cached app token) implement
PushSender; notification delivery routes both through it, also for
notification-only configurations.

Targets are members bound in a private chat with the Bot. Saving a
target first sends a trial message and only a confirmed delivery
records target_verified_at on the binding; revocation or re-binding
clears it. Deliveries persist send_key, generation and the database
start time before the network call: Feishu retries an unknown outcome
with the same key inside 55 minutes, DingTalk stops at send_unknown.
Admins can confirm receipt or resend (new generation and key), and a
target switch with queued notifications requires an explicit cancel
or migrate choice.
Stage G, part 2. When a DingTalk session webhook expired before the
reply went out (reply_failed / route_expired), an administrator can
switch the saved reply to a proactive private message to the original
sender. The transaction re-checks the configuration, the failed state
and that the sender's binding still belongs to the same platform user
with a successful trial send; the HTTP layer also re-checks project
permission. The reply goes out as a new generation with a new key via
PushSender and the switch, actor and reason are recorded.

Without a verified sender the records page offers a trial send to that
member instead (POST /api/notification-bots/:id/targets/verify), which
leaves the notification recipient unchanged.
Stage G, part 3. WeCom Bots get an optional confirm write mode. The turn
runs with platform writes blocked; drafts are frozen in
bot_pending_drafts (random ID, parameter hash, 30-minute expiry) and the
reply carries a summary plus a /bot-approve.html#draft=<id> link.

Only the bound platform user can open or confirm a draft. Confirmation
re-checks the configuration, the sender's binding, project permission,
expiry and the frozen parameters, then moves the draft from pending to
executing in one locked statement, so a repeated click or a second tab
returns the current state instead of executing again. Execution reuses
resolveAssistantDrafts (preflight warnings stop a draft). A draft left
in executing is reported as needs_review and never re-run. Unbinding,
deleting the Bot, disabling chat or leaving the confirm mode cancel
pending drafts. Feishu and DingTalk stay read-only.

Also adds a MySQL/MariaDB integration test for the stage G statements
(push targets, two-phase pushes, target switch, push resend, draft
confirmation under concurrency).
…ation

Manual section 8 (recipients and trial sends, delivery rules, DingTalk
push resend, WeCom web confirmation, upgrade), the stage G acceptance
record with automated results and the real-environment checklist,
CHANGELOG, OpenAPI entries and the implementation notes in the
proposal (verification stored on bindings, no business idempotency
key).
web/embed.go lists static files explicitly; add bot-approve.html and
bot-approve.js and cover both Bot pages in the static route test.
Stage H, part 1. An hourly cleanup (with the binding-request cleanup,
batches of 1000, at most ten per category and run) applies two horizons
to terminal records, counted from when they finished:

- body (default 90 days): message, reply and sealed route of
  interactions, rendered notification content, frozen write drafts and
  the native CLI logs of Bot turns are removed; rows stay and can no
  longer be sent or executed again (body_purged_at, content_source
  purged);
- record (default 180 days): the rows are deleted.

Records waiting for someone (queued, running, ready, sending,
reply_unknown, pending, awaiting_authorization, send_unknown, executing,
needs_review) are never touched. Pending drafts past expiry become
expired. AGENTFLOW_BOT_BODY_RETENTION_DAYS and
AGENTFLOW_BOT_RECORD_RETENTION_DAYS adjust the horizons; 0 disables one.

Native logs are removed only where a finished, due bot_native_sessions
mapping points: file names equal to the requested or reported session
ID in the user's own CLI home (CodeBuddy, Claude Code; Codex rollouts by
thread ID), without following symbolic links and skipping sessions still
shared with a running or newer mapping. Each mapping records removed,
missing, error or unsupported with the relative paths; mappings are
deleted after the record horizon only once their log is settled.
compose.yaml passes AGENTFLOW_BOT_BODY_RETENTION_DAYS and
AGENTFLOW_BOT_RECORD_RETENTION_DAYS (90/180 by default); the install
test checks every Bot variable is wired; deployment.md lists the chat
Bot settings.
Conflicts: go.mod/go.sum (both dependency sets, go mod tidy),
internal/repo/models.go (Bot tables plus gitHTTPCredentialRow),
web/embed.go (bot-approve files plus kimi-code.png) and the Go notice
bundle (regenerated with scripts/collect-go-notices.py --write).
Manual: retention settings, upgrade step to confirm them before the
first cleanup, section 9 (retention rules, native log cleanup and
monitoring) and section 10 (rollback matrix). Final test record with the
automated results on the merged code, the MySQL 8.4 upgrade and rollback
rehearsal of the final build (first retention run included) and the
IM-01..IM-12 traceability per channel. OpenAPI gains the remaining Bot
notification routes and the purge markers; CHANGELOG and the proposal
note the retention behaviour. The stage G record now carries the MySQL
8.4 rerun.
Feishu and DingTalk Bots can now choose the 网页确认 write mode, so
members can create requirements and dispatch or start SOPs from a
private chat: the turn returns an operation summary and a confirmation
link, and the bound user confirms on the platform page, where the draft
runs at most once with the same re-checks as on WeCom. The link stands
on its own line in these plain-text channels. Automatic writes stay
WeCom only; saving auto on Feishu or DingTalk is refused.
yowucai added 4 commits November 20, 2026 02:53
Requirements created through web confirmation used to get the project
default SOP silently. The chat summary now names the SOP each new
requirement will be bound to, and the confirmation page lists the
project's published SOPs so the user can switch it and choose to start
the SOP right after creation. Only SOPs offered on the page are
accepted; the frozen draft is unchanged. The confirm-mode prompt tells
the model to set workflowId when the user names an SOP.
Waiting and failed node notifications of Bots with chat enabled are now
sent as action cards on Feishu (interactive message, buttons carry the
action and delivery) and DingTalk (createAndDeliver with the template
ID an administrator configures, STREAM callbacks, IM_ROBOT space).
Without a DingTalk template the notification stays plain text.

Clicks arrive on the receiving connection (Feishu card.action.trigger,
DingTalk /v1.0/card/instances/callback), are acknowledged inside the
callback window and queued through the common inbound path. They run
handleBotCardAction with the WeCom card checks (bound sender, project
permission, node still in the notified state and attempt, action still
offered); only the recipient's click is placed in the delivery's
conversation. The result goes back to the clicker as a private message.

Manual section 8.5 documents the Feishu callback and the DingTalk
template contract; acceptance items C-01..C-05 and W-05..W-06 added.
…odel

- Feishu replies with write drafts are now an interactive card (the
  Bot's explanation, the operations with their SOP, and 确认执行 /
  确认并启动 SOP / 取消 buttons plus a link to the confirmation page).
  Clicks come back as card.action.trigger and run the same re-checks and
  pending -> executing transition as the page (shared executeBotDraft);
  only the requester can use them, the result is a private message.
- Feishu and DingTalk answer group messages that @ the Bot (Feishu
  mentions with mentioned_type=bot, DingTalk isInAtList); other group
  messages are still ignored and counted. @ placeholders are removed
  from the text. Each group keeps its own conversation; every message
  runs as the sender's bound account.
- The Bot form calls the model field 回答模型 and always offers 默认.
- The confirm-mode prompt keeps internal field names out of replies.
The SOP dropdown of new requirements now starts with 项目默认 SOP (name),
selected when the model named none, and lists the same SOPs as the
platform requirement form: the project's own, shared SOPs and the generic
templates (archived ones excluded). Choosing the default removes any
workflowId the model set, so the requirement is bound the usual way. The
chat summary marks unnamed SOPs as 项目默认.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant