Skip to content

feat(tools/discourse): add a read-only Discourse topic bridge - #14

Draft
MakerFlyFly wants to merge 5 commits into
audit/discourse-basefrom
feat/discourse-topic-view
Draft

MakerFlyFly wants to merge 5 commits into
audit/discourse-basefrom
feat/discourse-topic-view

Conversation

@MakerFlyFly

@MakerFlyFly MakerFlyFly commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add a stdlib-only Discourse bridge with magpie-discourse topic view <id|url> [--max-posts N], so projects can read forum discussions as structured thread/message data.
  • Complete the requested post stream in batches of at most 20, cap output at 500 posts by default, and fail explicitly on missing requested posts or malformed responses. Optional API credentials stay within the configured HTTPS origin and forum path.
  • Integrate the package with the workspace, inventories and adapter specification. Declare contract:mail-archive with Coverage: partial; the complete contract and additional commands remain follow-up work.

Upstream submission

The first implementation is submitted as apache/magpie#1755, targeting upstream main.
This fork draft preserves the fixed audit baseline and review history.
The test results below describe that audit baseline; the upstream PR records validation on its newer baseline.

Type of change

  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Documentation

Test plan

  • Package: 213 mocked tests (54 additional cases in this revision; existing coverage retained and the exact message assertion updated for body_hidden); ruff check, ruff format check and mypy pass. No package skips or live HTTP in tests.
  • Repository integration suites: vendor-neutrality-score 42, skill-and-tool-validator 493 (repository-root invocation), dev 263, spec-validator 87 and spec-inventory 31 pass, with no skips.
  • prek run --all-files --verbose passes, including lychee and all four workspace hooks.
  • The workspace run covers 51 members: 9,311 passed, 361 conditional skips, 4 deselected and 26 passing subtests. Skips and deselections are not counted as passes.
  • The optional native Claude Code check is skipped because its CLI is absent; its successful hook wrapper does not establish native-runtime coverage.
  • Magpie final-commit checklist: A–E pass; 0 blocking, 0 advisory items.
  • Anonymous live CLI read on meta.discourse.org topic 183261: 3 GET requests, 45 unique posts with HTML bodies, boolean body_hidden flags and truncated: true. No hidden-body positive case was observed live; those cases use mocked responses. No forum writes.
  • New regressions distinguish explicit empty bodies from missing/null/invalid bodies on all response posts, including posts outside the output cap; check optional boolean body markers without inferring them from hidden; stop at the first incomplete batch; reject duplicate stream IDs; and reject raw Unicode/space base paths before HTTP while preserving percent-encoded bases.
  • Existing regressions cover missing/malformed streams, input validation, HTTP errors, ASCII username and latin-1 API key validation before requests, cp1252/cp936 stdout with Unicode round trips, empty base URL markers, mega-topic errors, moved-post ordering, configured redirect boundaries, GET-only transport behavior, raw/decoded path boundaries, early base-path validation, and the real urllib redirect chain.

RFC-AI-0004 compliance

  • Sandbox — the adapter declares its network reach; no sandbox or allowlist changes. Adopters allow their configured forum host.
  • Vendor neutrality — vendor-specific behavior stays in the adapter, with partial coverage excluded from complete-backend counts.
  • Write-access discipline — only GET requests are permitted, including redirects.
  • Privacy LLM — the CLI returns data and makes no LLM calls. Output does not classify visibility; callers must establish visibility and apply existing privacy approval before sending content to a model.

Linked issues

Refs apache/magpie#307.

Refs MakerFlyFly/magpie#13 (fork audit).

Notes for reviewers

This draft targets the fork's fixed audit baseline at upstream commit cb6bb8b8.
It covers only topic viewing; no skill integration or complete/selectable mail-archive backend is claimed.

  • Every response post must include a string cooked body, even outside the output cap. Explicit empty strings, whitespace and placeholders are preserved; missing/null/non-string bodies are response errors. An empty string only states what the API returned.
  • body_hidden relays the server's optional boolean cooked_hidden marker, defaulting to false. It is independent of hidden; false proves neither original text nor public visibility. Staff may see a hidden post's content without a replacement marker.
  • Bodies remain cooked HTML with body_format: "html". JSON uses ASCII escapes; parsing restores the original Unicode.
  • Mega topics without an ordinary post ID stream are explicitly unsupported, even when the initial posts would satisfy the limit. No dedicated mega-topic pagination is implemented.
  • truncated describes only the configured post limit. A missing requested post fails immediately after its batch, before later batches; HTTP errors also fail the whole read. Concurrent deletion or batch rate limiting can therefore fail a read; no partial results or automatic retries are provided. Duplicate stream IDs are rejected.
  • --max-posts limits post count, not response bytes or total command duration. Byte and whole-command budgets remain follow-up design work.
  • Messages follow API post_stream.stream order. Moved/imported posts can have older timestamps; chronological archive-contract ordering is not promised.
  • Authorized reads may include private categories or personal messages. The JSON is not evidence that content is public.
  • Initial requests and redirects must match both the raw and once-decoded configured forum path, as well as the origin. Default HTTPS port 443 remains equivalent to an omitted port; encoded separators, traversal and non-GET requests are refused.
  • URL paths are decoded once as UTF-8; a remaining percent character is refused. Literal-percent paths (%25) and nested escapes (%2541) are unsupported. Normal UTF-8 slugs and query parameters remain supported.
  • API usernames must be ASCII; API keys retain latin-1 encoding validation. Unsupported usernames and invalid base paths fail during configuration without making a request or echoing credential values.
  • Configured base paths must percent-encode non-ASCII characters and spaces. Raw Unicode/space base paths are rejected with a configuration hint; accepted URL spelling is preserved.
  • A self-consistent empty topic remains a valid empty result. Missing collections/streams and an empty stream paired with a non-empty topic remain errors.

At the latest publication check, GitHub reports no check runs or Actions runs for this head.
The results above are local validation; no cloud-CI success is claimed.

AI disclosure

AI-generated text for the fork audit record. The author has completed review of the implementation.
Claude Code assisted the initial local prototype.
Codex (GPT-6) assisted the revisions, regression tests and verification.

Projects discussing design and governance on Discourse need structured
topic context alongside their existing mail and forge sources.

Read a topic and its bounded post stream with a stdlib-only CLI. Return
mail-archive-shaped thread data while declaring partial coverage, and
fail explicitly on missing requested posts or malformed responses.

Keep optional API credentials on the configured HTTPS origin and wire
the package into the workspace, adapter inventory and specifications.

Generated-by: Codex (GPT-6)
An HTTP error can arrive before its response body has been fully read.
Keep the HTTP status in the CLI error channel when that body is cut off,
instead of letting the error-reporting helper raise an unhandled exception.

Generated-by: Codex (GPT-6)
Validate API header encoding before requesting and escape JSON for legacy
stdout encodings. Reject empty base URL query and fragment markers.

Bind GET requests and standard-library redirects to the configured forum
origin and path, rejecting encoded traversal and other ambiguous paths.
Report unsupported mega topics explicitly without returning partial data.

Document unknown visibility, fail-closed reads and forum stream ordering.
Keep the existing fields, partial archive coverage and read-only scope.

Generated-by: Codex (GPT-6)
Decoded-only path checks can admit a different raw routing prefix.
Require both raw and decoded forum boundaries, decode paths once and
reject encoded separators or remaining percent escapes.

Reject invalid base paths during configuration and require ASCII API
usernames while retaining the API key's latin-1 encoding check.

Exercise the real urllib redirect chain with fake HTTP transports and
document the deliberate URL and authentication compatibility limits.

Generated-by: Codex (GPT-6)
Require cooked bodies on every response post while preserving explicit
empty strings. Expose the server's cooked_hidden marker as body_hidden
without inferring original content or visibility.

Stop after the first incomplete post batch, reject duplicate stream IDs
and preserve participant order using set membership for deduplication.

Reject raw non-ASCII or space-containing base paths during configuration
and document post-count, response-size and total-duration boundaries.
Keep the adapter-spec addition applicable to current upstream context.

Generated-by: Codex (GPT-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant