[earakely-scale]

What

Train on a bundle's tasks with Unsloth's GRPO, through the TRL helper (agent_env.training.trl). Unsloth's GRPOTrainer is TRL's, rewritten when unsloth is imported first. It takes AgentEnv(url).dataset and AgentEnv.factory as environment_factory as they are. That covers:

• with and without Unsloth's colocated vLLM (fast_inference);
• in 16-bit LoRA and 4-bit QLoRA;
• one- and two-turn tasks;
• evals during training.

The GPU proof below (Unsloth 2026.10.11 on TRL 1.15.0, transformers 5.17.0, vLLM 0.29.0) found one thing to change in the helper, and Unsloth's own constraints add a CI job.

A restarting env server no longer costs every rollout it spans

A TRL environment holds its session across rollouts. When the server dropped it, the helper reopened it with a 4-second wait for room, and didn't wait at all for a server refusing connections, as one restarting does. So the rollouts of every step a restart spanned were left unscored, including those that started while the server was coming back.

reopened (_openenv) now waits for a server that refuses connections as long as a reset may take, max(patience, reset_timeout), when the session asks it to (waits_for_server), and the TRL helper's sessions do.

• Unchanged: the short wait for room, so a dropped environment whose slot another took still gives up in 4 s rather than stalling a step. The first session still raises when the server refuses it, as one at capacity does.
• verl and SkyRL: they already wait with patience=reset_timeout, so they behave as before.

CI at Unsloth's ceiling

Unsloth 2026.10.11 caps TRL at 1.15.0 and transformers at 5.17.0. TRL 1.15 can't train on CPU (its log-prob path is a Triton kernel), so trl.yml adds a grpo (trl==1.16.*, transformers==5.17.*) job: the helper on the newest transformers Unsloth allows.

Tests

• sessions_test.py: a session tried again past its patience while the server restarts, and given up while it's only busy.
• trl_test.py: a session dropped by a restarting server opens again once the server is back, and the rollout scores. It fails without the change.
• The TRL tier passes on TRL 1.16.0 with transformers 5.17.0, locally and in CI's new job.

GPU runs

Unsloth 2026.10.11 with TRL 1.15.0, transformers 5.17.0 and vLLM 0.29.0 (vllm-bnb-plugin 0.0.3 for 4-bit), on AWS in a temporary VPC, torn down after:

• L40S (g6e.4xlarge): run A in 16-bit LoRA.
• 24 GB L4 (g6.4xlarge): everything in 4-bit QLoRA.

Bundles are served by AgentEnv.local, or by a separate build_app server for the outage runs. The framework is this branch (4370874, before the rebase onto #425's merge; the rebase only renames the session loop).

What runs under Unsloth with no change to the helper:

Run	Setup	Result
s1	GridSim, base Qwen3-4B, 4-bit, Unsloth's vLLM, 2 steps of 16	trains; 2 tool calls a rollout, 82 s a step
s3	the same without vLLM (fast_inference=False, transformers generate)	trains
s2	the two-turn inbox task, end_turn, evals at every step, reward= the mean	trains and evaluates; 3.2 tool calls a rollout in training, 4.5 in eval
s6	Qwen3.5-2B, 16-bit, the inbox task	trains and evaluates; TRL parses its calls
A	GridSim, the Qwen3-4B SFT warm start, 16-bit LoRA on the L40S, 40 steps of 32 with the TRL page's settings and Unsloth's changed defaults set back	eval on the 200 eval days 0.551 → 0.520 → 0.595 (TRL 0.57/0.56/0.61, SkyRL 0.54/0.55/0.61); train reward 0.66 throughout; 104 s a step; 103 min with three evals; no failed rollout
L	the same SFT model in 4-bit on the 24 GB L4, about 22 GB used, 10 steps	eval 0.506 → 0.446; 269 s a step; 31 min an eval
d	the two-turn inbox task, 4-bit Qwen3-4B, 5 steps of 32	train 0.16 → 0.25, eval 0.125 → 0.25, tool calls 3.4 → 4.6 a rollout
s7b	multi-environment mode: GridSim's and the inbox's rows in one dataset	env.factory a dict of two classes (gridsim, mail); trains
s8b, s8c	thinking on	inbox: reasons in about 500 tokens and calls its tools; GridSim: reasoning ran past 6,144 tokens before any call, so it scored 0 (a budget limit, on the page)
s9	Unsloth's AsyncGRPOTrainer on one GPU, generating on the model's own fast_inference engine, 3 steps of 16	trains; 2 tool calls a rollout

Found on GPU, and fixed here: a restarting env server cost every rollout of the steps it spanned. Two 4-step runs on the L4, 16 rollouts a step, with the server stopped 10 s into step 2 and back 30 s later:

	before (s5b, main)	after (s5c, this branch)
Step 2's rollouts, mid-episode	16 unscored	16 unscored
Step 3's, started while it came back	16 unscored	waited for it; scored (0.2 each, a 47 s step)
Step 4	not run	scored

Not code, for the docs:

• Unsloth replaces any GRPOConfig argument left unset: optim adamw_8bit, weight_decay 0.001, seed 3407, learning_rate 5e-5, a 10% warmup, per_device_eval_batch_size 4.
• With loss_type="dr_grpo", Unsloth turns scale_rewards off even when it's set to "group" or True; only None keeps TRL's scaling.
• An answer written in plain text, not passed to end_turn, reaches the task empty. That's the TRL helper's design, which TRL's page documents. Qwen3.5-2B scored 0 on the inbox task for it, where SkyRL's env reads the reply.
• The env-owned reward can't log per-score metrics: TRL's get_reward wrapper doesn't pass log_metric. Upstream follow-up.

Spend: about $40 of the $40, plus about $1.70. A first round's boxes ran unattended to their 7-hour shutdown, which deleted run A's and run L's results, so both ran again.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

The TRL helper now waits longer when reopening a session against a server that refuses connections.

• TRL sessions can reopen after an OpenEnv server restarts.
• Unsloth's GRPOTrainer can train from AgentEnv task data.

The earlier, unnumbered timeout finding is fixed: _call now includes the reopen budget. No new actionable issues were found.

_{Reviews (3) · Last reviewed commit: "fix(training): a TRL environment's call ..." · Reviewed by Greptile}