A more general setting for agent training
Multi-Session Agentic Reinforcement Learning
LARK, HKUST(GZ) | Foundation Model Lab, Huawei | City University of Hong Kong | HKUST
An agent’s work can span many sessions.
Its training should, too.
01 / The motivation
One rollout. More than one session.
Most existing agentic reinforcement learning builds on a simple assumption: one rollout produces a single session, with reasoning, tool calls, and observations accumulating in one continuous context window.
However, advanced agent inference techniques go beyond a single session: (1) context compaction, typically triggered when the model’s context window is nearly full, summarizes an accumulated history and replaces the working context, allowing the agent to continue beyond its context budget; (2) subagent forking delegates subtasks to independent agents, each operating in its own context window. Integrated into modern agent harnesses (Claude Code, Codex, etc.), these techniques are becoming routine tools for long-horizon tasks.
These workflows expose a mismatch between single-session training and multi-session inference. To align training with how agents actually execute tasks, we introduce Multi-Session Agentic Reinforcement Learning.
Definition
Multi-Session Agentic Reinforcement Learning
An agentic reinforcement learning setting in which each rollout may contain an arbitrary number of sessions. Each session consists of agent–environment interactions within one continuously accumulated context. Each session ultimately corresponds to one training segment with its own assigned advantage.
02 / Where current practice falls short
A shared outcome is a coarse signal.
A straightforward solution is broadcasting: compute an advantage for the whole trajectory and apply that same value to every session and every trainable token within it.
This makes multi-session trajectories trainable, but it leaves two questions unresolved. How many sessions should the agent use? And which sessions actually helped? A useful search, a lossy compaction, and an unproductive delegation all inherit the same global signal.
Learning across sessions requires both a global sense of success and a local sense of contribution.
03 / Our solution
Signals from sessions. Credit for each.
We propose Mosaic, a general method for multi-session agentic reinforcement learning that combines session-aware trajectory-level advantage estimation with session-level advantage shaping. The first accounts for how the trajectory uses sessions. The second differentiates the contribution of each session, without an additional critic.
Use enough sessions. Make them count.
Session count is a useful signal, but it is not a reward on its own. For a successful trajectory, fewer sessions indicate more efficient execution. For an unsuccessful trajectory, using too few sessions can signal insufficient exploration.
Mosaic ranks session counts within groups sharing the same outcome, then uses a bounded factor to adjust the original trajectory advantage. In mixed-outcome groups, the adjustment preserves its sign: using more sessions cannot turn a failed trajectory into a successful one.
Give each session its own credit.
At the local level, Mosaic treats a session as a transition between two context states. It estimates marginal information gain from the change in the policy’s average log-likelihood of the correct answer. Useful evidence can raise that likelihood; discarding important facts or adding misleading information can lower it. Furthermore, normal, compaction, and subagent sessions have different marginal gain distributions. Mosaic therefore normalizes the gain values separately by session type within the prompt group.
Finally, the trajectory-level and session-level signals are combined through a weighted sum to produce the final advantage for each session:
04 / The evidence
Better answers.
Fewer sessions.
We evaluate on BrowseComp, a benchmark of challenging web-research questions, using Qwen3.5-4B in a ReAct-style agent loop. Training uses 1,000 deep-search tasks from REDSearcher. Training and evaluation use both the compaction or forking execution loop.
with context compaction
over broadcasting
down from 3.17
| Method | Single session | Context compaction | Subagent forking | |||
|---|---|---|---|---|---|---|
| Accuracy | Sessions | Accuracy | Sessions | Accuracy | Sessions | |
| Base model | 23.22% | 1.00 | 24.80% | 1.70 | 21.64% | 2.98 |
| Trained on single-session trajectories | ||||||
| GRPO | 31.99% | 1.00 | 34.68% | 1.24 | 31.91% | 2.09 |
| IGPO | 28.52% | 1.00 | 31.36% | 4.12 | 24.64% | 3.26 |
| Trained on multi-session trajectories | ||||||
| Broadcasting | — | — | 35.31% | 3.17 | 31.44% | 4.43 |
| Mosaic ours | — | — | 38.86% | 2.35 | 32.46% | 3.77 |
Training with multi-session trajectories matters: single-session training does not automatically yield the full benefit of multi-session inference. Broadcasting already improves performance. Mosaic improves it further in both settings, with a lower average session count. More experimental results and analyses are available in the paper.
05 / Discussion & outlook
Train for the way agents actually work.
What the results suggest
Both trajectory-level estimation and session-level shaping contribute to Mosaic’s gains. The central lesson is that effective multi-session optimization depends on session utility and allocation—not simply on using more sessions. The smaller gains with subagents also suggest that credit assignment across branching sessions remains an open challenge.
We see four promising directions for future research in Multi-Session Agentic RL.
Infrastructure for reproducible research.
Multi-session trajectories arise naturally in complex, long-horizon tasks. While some benchmarks can elicit them, suitable public training data remain scarce, limiting the scale and reproducibility of multi-session agentic RL research. Open training datasets and corresponding agent execution environments are key to a sustainable research ecosystem.
Cross-session state transfer as a central learning objective.
Compaction must preserve working state for continued execution; forking must provide sufficient information for a subtask; merging must integrate the results into a state the main agent can use. Together, they raise a shared question: under a limited context budget, what information should be carried forward to support subsequent decisions? An important direction is to learn state transfer across these operations jointly, while studying information loss and error accumulation over repeated transfers.
Asynchronous training for multi-session rollouts.
Multi-session execution can amplify differences in rollout lengths and completion times, complicating group comparisons and sample reuse under policy staleness. Prioritizing completed rollouts may favor shorter trajectories, introducing selection bias and influencing learned execution strategies. Scaling to asynchronous settings—particularly for critic-free, group-relative optimization—requires further study of how groups are formed, stale samples are handled, and trajectories are reused.
Toward RL within (black-boxed) agent harnesses.
Multi-session RL could provide a foundation for training agents within harnesses whose orchestration is partially opaque, such as Claude Code. Beyond compaction and forking, other mechanisms—including dropping reasoning tokens or tool responses—can also disrupt context continuity. Future work should systematically examine these mechanisms and adapt optimization accordingly. When discontinuities occur, the choice between introducing a new session boundary and lossily concatenating the modified context with the preceding session may also affect training effectiveness.
More capable agents do not just need more room to work.
They need to learn how to use it.
Citation
@article{xu2026mosaic,
title={Mosaic: Multi-Session Agentic Reinforcement Learning},
author={Xu, Yilong and Xu, Minrui and Zhou, Xinyu and Long, Xiang and Tan, Haochen and Guo Zhijiang},
journal={arXiv preprint},
year={2026}
}