[earakely-scale]

What

Train on a bundle's tasks with verl, through verl's own agent loop. verl 0.9 runs multi-turn tool rollouts in verl.experimental.agent_loop: per row, an agent loop generates on verl's vLLM, parses the model's tool calls, adds their answers to the context and returns the tokens, the loss mask and, optionally, the rollout's reward_score, in which case verl skips its reward function. A custom loop is registered through actor_rollout_ref.rollout.agent.agent_loop_config_path, and each row picks it by agent_name.

agent_env.training.verl, behind a new verl extra (openenv, pyarrow, transformers; verl, vLLM and torch stay the trainer's to pin).

The writer

python -m agent_env.training.verl URL OUT writes the tasks of a bundle's OpenEnv server (agent_env.openenv.build_app):

• OUT/<split>.parquet: a row per task, with an explicit schema. A row has:
  • the task's system prompt and first prompt;
  • agent_name: agentenv and data_source: agentenv/<env>;
  • extra_info with the split, index, task id and the row's tools as JSON.
• OUT/agent_loop.yaml: registers the loop with the server's URL, the deadlines, attempts and the token's variable.

A row's tools come from playing one task of each toolset, as the NeMo Gym writer does; both writers share _openenv.tasks_and_tools.

Its options:

• --token-var NAME names the environment variable holding the token a server asks for. The writer reads it now and the loop in every verl worker, so the token is in no file.
• --tokenizer MODEL counts each row's prompt with its tools as the model's chat template renders it. Pass --chat-template-kwargs with verl's data.apply_chat_template_kwargs so the count matches verl's rendering.
• --max-prompt-length N refuses rows over N tokens before anything is written. verl would cut such a prompt from the left, losing the system prompt and the tools, and its own overlong filter doesn't count a row's tools.

The loop

agent_env.training.verl.loop.AgentEnvAgentLoop is verl's ToolAgentLoop with the row's tools in place of verl's configured ones, so verl's tokenization, tool parsers, turn caps and truncation are its own.

• Tools: each parsed call is a call on the row's episode, through the Session layer the TRL helper and the NeMo Gym server use. verl's parsers get typed tool schemas (qwen3_coder converts arguments by them), and the chat template renders the row's full JSON Schema.
• Reply and reward: the model's last reply, without special tokens, ends the turn it left open. The episode's reward is the rollout's reward_score.
• A rollout the env fails (a deploy, a call past its deadline, a server that went away) is played again from the start on a new episode, a fresh sample of the same policy.
  • It's replayed up to attempts times, but none starts once reset_timeout has passed since the rollout began.
  • One that fails every time raises RolloutFailed. verl's V1 trainer (trainer.use_v1, the default on 0.9.1 and main) records a session that raised as failed and leaves its trajectory out of the loss, its group's GRPO baseline (padding gets its own uid) and the eval reward.
  • With trainer.v1.sampler.sync_refill_failed_groups=True, a group left with no trajectories is replaced by a fresh prompt instead of stopping the run.
• A restarted env server delays rollouts rather than failing them. Opening its session, a rollout waits out a server at its limit or refusing connections (Shared.waits_for_server) within reset_timeout. The writer read the server first, so a refusal then is a server coming back; a server that refuses the token fails at once.
• Configuration errors raise, and the V1 trainer logs each one and leaves it out:
  • a first prompt over rollout.prompt_length (refused rather than cut);
  • an episode that ends unscored without failing;
  • a loop instance run twice.
• A cut env answer is warned of once a tool, when it's over multi_turn.max_tool_response_length, which verl truncates silently (256 characters by default).
• verl internals it leans on are named in the class docstring, and CI checks them on verl 0.9.1 and main.

The env server

build_app(token=...) requires Authorization: Bearer <token> on every request and WebSocket but /health: a request without it gets 401, and a WebSocket is closed before it opens. That lets the env server run where other machines, verl's workers on any node, reach it. Without a token, anyone who reaches the server deploys and drives its envs. The session layer sends the token (Shared.token, opened, read_tasks, probe).

Test infrastructure

tst/unit/test_optional_extras_stay_optional.py: verl's distributions are all trl's (pyarrow comes with datasets, and transformers), so the per-extra check has nothing to block for it. _COVERED records that and a test asserts it; the no-extras check covers the module.

Tests

• Unit tier:
  • tst/unit/training/verl_test.py, the writer on the fake bundle: the rows of each split and the loop config; tools per toolset, each probed once; an empty split keeping a row's columns; a server that asks for a token, read with the one its variable holds, with the token in no file and a wrong one refused; and what's refused before the server is read.
  • tst/unit/openenv/app_test.py: a server given a token refuses HTTP and WebSocket without it, but answers /health.
  • tst/unit/training/sessions_test.py: a session waits out a restarting server only when asked, and never one that refuses the token.
• verl tier (tst/verl/loop_test.py, make verl-test): AgentEnvAgentLoop.run on verl itself, with Qwen3's tokenizer and a stand-in rollout server, on the fake bundle. Env failures come from the fake env and a broken server, not from patching Session. It covers:
  • a played row's reward and reply, through hermes and qwen3_coder;
  • typed parameter conversion;
  • a replay on a new episode, and none once the time is up;
  • RolloutFailed when every attempt fails;
  • a prompt that doesn't fit, an unscored episode, and a loop run twice;
  • a token server played with and without its token;
  • the cut-answer warning;
  • the writer's prompt count with and without enable_thinking, and its refusal over the limit;
  • the config naming the loop.
• .github/workflows/verl.yml runs the verl tier on verl 0.9.1 and main with CPU torch; it allows no skips and isn't a required check.

GPU run

GRPO on the GridSim v1 bundle through AgentEnvAgentLoop, on the branch, with the TRL, NeMo RL and prime-rl runs' settings where verl has them:

• Box: one L40S (AWS g6e.4xlarge). verl's FSDP actor and its vLLM rollout shared the GPU, and the bundle's envs ran in Docker on the same machine (build_app, 64 episodes at once).
• Software: verl main in its verlai/verl:uv.cu130.dev5 image (torch 2.13, vLLM 0.29), with agentenv-framework[verl] from this branch installed into its venv.
• Model: the Qwen3-4B SFT warm start, LoRA r32/α64 on every linear layer, lr 2e-5 cosine with 3 warmup steps, thinking off, the hermes parser.
• Rollouts: 32 a step (4 days × 8, temperature 1.1), at most 4 assistant turns. Evals on the 200 eval days at steps 0, 20 and 40, one sample a day.

Qwen3-4B on GridSim, through verl	Step 0	Step 20	Step 40
Eval reward, 200 eval days	0.524	0.586	0.601

• The train reward rose from 0.585 over the first 10 steps to 0.697 over the last 20.
• None of the 1,280 training rollouts or 600 eval rollouts failed in the env, so none was played again or masked.
• 40 steps with the three evals took 67 minutes, about 75 s a step.

What it took to run, none of it in the code:

• verl 0.9.1's and main's uv.lock point flash-attn at a wheel URL that now 404s, so uv sync --extra fsdp fails; verl's image, whose uv cache holds the wheel, works.
• On one GPU, LoRA's rollout.layered_summon=True fails (FSDP refuses CPU offload with NO_SHARD); it ran with False.
• multi_turn.max_tool_response_length must cover the env's answers; verl's default of 256 characters would cut them.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues remain from this review.

Summary

Adds a verl dataset writer and an agent loop that plays AgentEnv episodes through OpenEnv.

• verl trains on OpenEnv tasks through an agent loop.
• OpenEnv servers can require a bearer token for requests and WebSockets.
• The NeMo Gym writer gets task and tool details from the shared helper.

No new actionable issues were found. Tests were inspected, not run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[OpenEnv task server] --> B[Dataset writer]
  B --> C[Parquet rows and loop config]
  C --> D[verl agent loop]
  D --> E[Model generates replies and tool calls]
  E --> F[AgentEnv episode]
  F --> D
  D --> G[Episode reward returned to verl]

Loading

_{Reviews (7) · Last reviewed commit: "fix(training): the verl writer counts pr..." · Reviewed by Greptile}