Skip to content

feat(website)!: WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) - #428

Open
williamloo-scale wants to merge 11 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk
Open

williamloo-scale wants to merge 11 commits into
wloo/website-harness-gapsfrom
wloo/website-environment-sdk

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #427.

agentenv_protocol.website: the easy path to a synthetic website that follows website/v1 (urn:agentenv:website/v1).

@environment_card(name="notes")
@website(domains=["notes.test"], static="web/dist")
class Notes(WebsiteEnvironment):
    @get_data
    def export(self) -> list:
        return [DataPart(data={"notes": self.db.all_notes()})]

    @api.get("/api/notes")
    def list_notes(self, ctx: ApiContext) -> list:
        return self.db.notes(owner=ctx.user)
  • @api.get/post/put/patch/delete(path, name=, roles=, status=): path, query and body are bound from type hints, errors are exceptions mapped to statuses, and an OpenAPI 3.1 document is generated.
  • ApiContext: the signed-in user (else the acting user, so the agent starts signed in), the caller's role, and the public URL from X-Forwarded-Host/-Proto.
  • Roles: the gateway forwards whatever AgentEnv-Role a browser sends, so a site believes a non-default role only when the request also carries AgentEnv-Role-Key equal to the site's WEBSITE_ROLE_KEY (compared in constant time). Every other caller, and every caller of a site with no key set, is default, so roles= routes are closed until a key is set.
  • Serves the built pages with an index.html fallback, the data plane and a card declaring website/v1 with its routes and openapi/ready/session operations.
  • Readiness is website/v1's ready alone, as a computer app's is computer-app/v1's: @ready_check methods run at once, each within 10 s, a plain def in a worker thread and one call at a time, each problem saying what the check returned, raised or how long it ran. The answer carries waited_s and the timeout_s from @website(ready_timeout_s=...) (300 s by default), which is how long a deploy waits for the site. There is no health= path; a server whose card has no ready is still read at its health path, which is deprecated. /agentenv… routes are matched before the site's own.
  • self.now() reads the universe clock (clock/v1 sync_time), read once per request from the gateway as MCP servers read it, so a frozen clock is exactly frozen. An unarmed clock is wall time, and an unreachable one answers 503, except on the data plane and the card. An empty env_get_time_url goes back to wall time. Every response carries X-Universe-Now; clock_patch=True patches the page's Date. sync_time also takes main's optional timezone (feat(sync_env_clock): an optional timezone reads virtual_time in the zone and tells every synced server (FD-5003) #442), which the site reads as self.timezone; now() stays UTC.
  • A site keeps its own data, usually in its schema of the env's database (database_url), and serves it through the data plane's hooks (@reset_data, @add_data, @get_data) as any AgentEnvEnvironment does. The SDK has no store of its own. A site's data is read and written only through the data plane, and the SDK serves no /go and no website/v1 state op.
  • Sessions: the signed agentenv_session cookie lasts its Max-Age (7 days), never an absolute Expires, so a browser on another clock judges it the same. A deploy gives each site its own WEBSITE_SESSION_SECRET, so sessions outlive a restart or a snapshot reload; a site run without one (local serve()) draws its own. website/v1's session {user?} answers a storage state, {cookies, local_storage}, each cookie with max_age.
  • Plain def @api handlers run in a worker thread, as FastAPI's do. A route whose type hints can't be resolved fails the app's build, and session cookies go on successful answers only. The static fallback never serves a dot-named path.
  • website_state_verifier, which feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged and which read a site's /go?sid=, is removed. Grading a website from data/get against the task's initial state moves to agentenvhub-sdk's cua_gym_verifier, a separate PR in scaleapi2.
  • python -m agentenv_protocol.website.conformance URL checks any server over HTTP: card, any host name, health, universe time, data plane, clock, forwarded headers, cookies. It sends nothing to a server other than the one under check, and its clock check syncs the server back to wall time at the end. A cookie with an absolute Expires and no Max-Age warns without failing the run.

Breaking

Known gaps

Each is fixed later in the stack, as noted.

Test

packages/agentenv-protocol/tests: 1,121 passed after the readiness edits (b5dbda6), with the provider and env unit tests; lint clean. Full unit suite on CI. Before them: unit suite and protocol tests 12,659 passed.
End to end on a local Docker gateway (f22dff3): an SDK site sharing the name slack with an MCP server, beside the website browser. The deploy waited on ready (waited_s 21.4, timeout_s 120) and never read a health path; the compose carries a 64-hex WEBSITE_SESSION_SECRET, masked in the log. Conformance passed 8/8 through /svc/slack/. After sync_env_clock a frozen clock read exactly frozen, and a set-time showed on the next answer with no re-sync. The session cookie has Max-Age=604800 and no Expires, and a signed-in cookie still read as its user after a snapshot reload recreated the site's container. The universe's slack loaded into the MCP server.
packages/agentenv-protocol/tests: 946 passed; the website tests (59, with the Notes ones) pass after the role-key fix. Deployed locally behind the gateway, a spoofed AgentEnv-Role: admin now gets 403 on both /website/ and /svc/ (200 before).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The latest change appears safe to merge; it makes the worker-thread test reliable without changing site behavior.

Summary

Adds WebsiteEnvironment with typed API routes, sessions, universe time, readiness checks, and an HTTP conformance checker.

  • WebsiteEnvironment serves pages, typed routes, and AgentEnv data.
  • A command checks whether a server follows the website/v1 contract.
  • Deployments use a site's website/v1 readiness answer.
  • Browser routes keep site data and AgentEnv endpoints private.
  • Environment callers can tell websites and MCP servers apart.
  • The website state verifier is removed from task grading.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Browser --> Gateway
  Gateway -->|Site requests| WebsiteEnvironment
  Framework -->|Data and control requests| WebsiteEnvironment
  WebsiteEnvironment --> API[Typed API handlers]
  WebsiteEnvironment --> Pages[Static pages or mounted app]
  WebsiteEnvironment --> Clock[Universe clock]
  Checker[Conformance checker] -->|HTTP checks| WebsiteEnvironment
Loading

Reviews (19) · Last reviewed commit: "test(website): compare a plain def route..." · Reviewed by Greptile

williamloo-scale added a commit that referenced this pull request Nov 3, 2026
Brings in #428's review fixes beside this branch's: the clock's sync
reads under the refresh lock and keeps the answered rate; conformance
checks X-Universe-Now last, on every answer, at the recommended level,
with the reserved-path probes counted; the handwritten test server
stamps its 404s and 405s like every other answer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tract, and a conformance check tests any server (FD-4683)

This PR's change, rebased as one commit onto #427.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/agent_env/env/env.py Resolved
@williamloo-scale
williamloo-scale force-pushed the wloo/website-harness-gaps branch from 7fc14cd to b18049d Compare November 3, 2026 21:50
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 3579c2a to 76145d2 Compare November 3, 2026 21:50
Comment thread src/agent_env/env/gateway/gateway.py Resolved
williamloo-scale and others added 2 commits November 3, 2026 14:20
… keep older sites and shared names working (FD-4683)

- The gateway answers 503 on a site's route while its card can't be read, rather than forwarding with nothing reserved.
- A plugin's MultiEnv record looks each child up by its kind, so a website and an MCP server may share a name.
- The state verifier reads a site its definition names but its card doesn't list at /svc/<name>/go, deprecated.
- The clock script is put in streamed HTML holding only what comes before <head>.
- Conformance fails a website/v1 ready that answers an HTTP error, whatever its body says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ver HTTP client the test client uses (FD-4683)

After a sign-out the client's jar keeps the site's signed-out cookie under testserver.local, and setting the
forged one adds a second cookie of the same name. httpx2, which Starlette's TestClient uses when it is
installed (as on CodeBuild), sends the forged one first, so the site reads the valid signed-out cookie and
the test fails; httpx sends them the other way round. Clearing the jar first sends the forged cookie alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

This is a huge red flag: it leaks CUA-Gym into agentenv-framework. /go?sid= is CUA-Gym's verifier endpoint, and this PR builds it into the SDK and the website/v1 contract rather than keeping it outside the framework:

  • The SDK: agentenv_protocol.website defines and exports STATE_PATH = "/go" (environment.py:51, __init__.py:15). Every store-backed SDK site serves its state there (environment.py:173) and advertises it as website/v1's params.state.path (environment.py:198).
  • Conformance: the checker reads state at params.state.path, else at /go (conformance.py:200-208), so it judges any server by CUA-Gym's convention.
  • Verifier and gateway: website_state_verifier falls back to params.state.path, else /go?sid= (website_state_verifier.py:37, :279-283). The gateway's guard is built around params.state.path (gateway.py:1556). The /go fallback predates this PR, but this PR extends it.

website/v1's state op should be the only state contract: an SDK site answers its state only through it. Please drop /go and params.state.path from the SDK, the conformance check and website/v1. If CUA-Gym sites still need to be read at /go, that compatibility belongs in a CUA-Gym adapter outside the framework, not in agentenv_protocol or the core verifier.

@earakely-scale

Copy link
Copy Markdown
Collaborator

Following up on my /go comment: please remove CUA-Gym's verifier model from the framework altogether. That model is a site serving {initial_state, current_state, state_diff} per sid, and a verifier reading it. This goes further than my earlier note, so drop the website/v1 state op too, since it's the same model under another name. agentenv already has the primitives for this:

What the verifier needs agentenv primitive
current state the data plane's data/get, which SDK sites already serve from the same store (_store_get)
initial state the universe the task loaded with data/add, or a snapshot_env taken before the agent starts
diff computed once, in the verifier, from those two, the same for every env instead of each site writing its own (that also removes the store-id diff problem)
sid not needed: each deploy gets its own site, and the SDK already ignores it
keeping state from the agent the data plane is framework-only and already refused on the browsing route

Concretely:

  • SDK: remove the /go route, STATE_PATH, params.state.path, the state op, and the store's site-side state and diff (Collections.state() / diff()). A site exposes its data through data/get only.
  • Verifier: grade by reading data/get against the initial state, keeping the check model (path, op, value, reduce, weight). It then works for any env, MCP servers included, not only websites.
  • Conformance and gateway: check data/get rather than /go or state. Drop the params.state.path reservation.
  • Main: website_state_verifier as feat(website): a website env is one server image, checked at its health path (FD-4683) #390 merged it reads /go?sid=. Replace it with the above rather than extending it. CUA-Gym images that serve only /go get an adapter outside the framework.

@williamloo-scale

williamloo-scale commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Agreed, and done in this stack rather than deferred. /go?sid= and the per-site {initial_state, current_state, state_diff} model were CUA-Gym's, and they're gone from the framework:

Each PR removes what it had added, so the stack never introduces any of this. Grading a website now belongs outside the framework: cua_gym_verifier in agentenvhub-sdk, in a separate scaleapi2 PR. It reads data/get, diffs it against the task's initial state (the universe it loaded or a snapshot_env), and keeps the check model: path, op, value, reduce and weight.

@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 1ffec4c to 05da818 Compare November 4, 2026 03:37
…op /go, the state op and website_state_verifier (FD-4683)

- The SDK serves no `/go`, no `params.state.path` and no `website/v1` `state` op; `STATE_PATH`, `WebsiteState`,
  `website_state`, `website_sites` and `website/state.py` are gone, and the default store keeps no initial state or diff.
- Conformance runs the data plane's round trip on every server, with no skip for a state that shows changes.
- The gateway reserves the card's `url` and every advertised endpoint, no longer a state path.
- `website_state_verifier` is removed from the framework; grading against a site's state belongs outside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale
williamloo-scale force-pushed the wloo/website-environment-sdk branch from 05da818 to 33ad3b8 Compare November 4, 2026 03:56
… items in OpenAPI, and find the clock's sync_time by name (FD-4683)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale

Copy link
Copy Markdown
Contributor Author

@greptile-apps all three remaining review comments are fixed in d4efdbb, each with a test that fails without its fix:

  • Partial HTML downloads change (environment.py): _Universe no longer injects the clock script into a partial answer. A 206, or any answer with Content-Range, passes through unchanged, so its bytes still match the promised range.
  • Query lists lose item types (routes.py): a list[T] query parameter is documented as {"type": "array", "items": <T's schema>}, and it still binds as repeated values.
  • Working clocks can fail (conformance.py): _clock() looks up sync_time by name instead of taking the first operation the card lists.

The full unit suite passes on this commit (12,908) and lint is clean. All three threads are replied to and resolved.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026

Copy link
Copy Markdown

Confirmed: all three findings are fixed in d4efdbbf and have regression coverage:

  • Partial HTML downloads: _Universe skips clock-script injection for status 206 and responses with Content-Range; the test verifies a range response remains unchanged.
  • Query-list item types: OpenAPI now includes the item schema for list query parameters, while binding still uses repeated query values; the test checks list[int] documentation and binding.
  • Clock operation selection: the checker finds sync_time by name; the test passes when another clock operation is listed first.

These comments can remain resolved. I verified the code and tests in the commit; I did not independently run the suite.

williamloo-scale added a commit that referenced this pull request Nov 4, 2026
…ses the conformance check (FD-4683)

This PR's change, rebased as one commit onto #428.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
earakely-scale and others added 2 commits November 4, 2026 09:43
… per request, and conformance stays on the server it checks (FD-4683)

- The SDK no longer keeps a store: `self.store`, `Collections`, `@website(data=...)` and the default data hooks are
  gone. A site serves its data through `@reset_data`, `@add_data` and `@get_data` as any AgentEnvEnvironment does,
  usually over its `database_url`; `acting_user()` is nobody unless a site says else.
- The clock is read once per request, as MCP servers read it: no extrapolation, rate inference or refresh lock, so a
  frozen gateway clock is exactly frozen and a slow one holds up only its own request. An unarmed clock (404) is wall
  time; an unreachable one answers 503, but the data plane and the card answer without the header. `sync_time` with
  an empty URL goes back to wall time.
- Conformance sends nothing off the server under check: a card's path or URL naming another scheme, host or port
  fails that check. The clock check ends by syncing the server back to wall time.
- Plain `def` `@api` handlers run in a worker thread; a route whose type hints can't be resolved fails the app's build;
  session cookies go on successful answers only.
- The static fallback serves no dot-named path and treats a path that can't be a file's as not found; a time parses
  with any fraction of a second on Python 3.10; the clock script goes after `<head>` (not `<header>`), else after the
  doctype and `<html>` tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title feat(website): WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) feat(website)!: WebsiteEnvironment serves a site on the website/v1 contract, and a conformance check tests any server (FD-4683) Nov 4, 2026
…session answers a storage state (FD-4932)

- The session cookie carries Max-Age, never an absolute Expires, so a browser on another clock judges it the same.
- website/v1's session answers {cookies, local_storage}, each cookie with max_age, in browser/v1's shape.
- The gateway provider gives each site its own WEBSITE_SESSION_SECRET at deploy, so its sessions outlive a restart
  or a snapshot reload, which keeps the compose file it recreates from.
- Conformance warns, without failing the run, on a cookie with an absolute Expires and no Max-Age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 4, 2026 11:41
…ready, its checks bounded, within the time it declares (FD-4683)

- WebsiteEnvironment serves no health path, and @website takes no health=; readiness is website/v1's ready alone,
  as a computer app's is computer-app/v1's.
- @ready_check methods run at once, each within CHECK_TIMEOUT_S, a plain def in a worker thread and one call at a
  time, so a slow check never holds up the site's answers; each problem says what the check returned, raised or how
  long it ran.
- @website(ready_timeout_s=...), 300 s by default, says how long a site may take; ready answers waited_s and
  timeout_s, and a deploy waits that long and a margin, saying whether a site is past its time.
- A server whose card has no ready is still read at its health path, deprecated; conformance needs a health path
  only then, and its clock check reads /.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bare string was read letter by letter, and a URL, a port, a path or an empty name was kept. @website now takes a
list of host names, each like example.com or *.example.com for its subdomains, as host_problem says, the rule live
sites' allow_hosts use too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…one client session

Each request of a TestClient outside a with block runs on a loop thread of its own, so the second request's could
reuse the id of the first's ended worker thread. Holding one session keeps the loop's thread alive for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants