On-Policy Distillation · Rollout Efficiency同策略蒸馏 · Rollout 效率

Flash-OPD

Fast on-policy distillation that stops every rollout at its own reliability boundary.快速同策略蒸馏：
让每条 rollout 都停在自己的可靠性边界上。

Shared rollout horizons cut some trajectories too early and over-serve others. Flash-OPD interleaves cached generation with teacher verification. It retires each trajectory once its exact count of low-compatibility events reaches a budget, which delivers 2.2×–7.5× speedups over full OPD at matched or better accuracy.共享的 rollout horizon 会让一部分轨迹截断过早，又让另一部分被过度解码。Flash-OPD 把带缓存的生成与教师验证交替进行：一旦某条轨迹的低兼容事件精确计数达到预算，就立即让它停止。在准确率持平甚至更高的前提下，相比完整 OPD 实现 2.2×–7.5× 加速。

Wei Chen^1,2,3, Junle Chen³, Yitong Yang¹, Zhaoyang Xu¹, Jiaxin Lin¹, Yuxuan Liang², Xiaofang Zhou³, Kai Wang¹, Rui Chen¹ 1

Tencent Hy 2

HKUST(GZ) 3

HKUST

Try the playground在线交互演示 BibTeX

Paper · soon论文 · 即将上线 Code · soon代码 · 即将上线

One prompt, eight rollouts同一 prompt 的八条 rollout

0 / 8rollouts with supervision cut early监督被提前截断的 rollout

0 / 8rollouts decoded far past E越过 E 后仍大量解码的 rollout

reliable prefix可靠前缀 supervision cut early被提前截断的监督 decoded past boundary越过边界的解码 boundary E边界 E illustrative示意

7.52×

faster training on Qwen3-4B-Base ← Qwen3-4B (33.09 h → 4.40 h)Qwen3-4B-Base ← Qwen3-4B 训练提速（33.09 h → 4.40 h）

21.9×

fewer response tokens per rollout on Qwen3-1.7B-Base ← Qwen3-4BQwen3-1.7B-Base ← Qwen3-4B 每条 rollout 保留的 token 更少

+3.35

avg. in-domain accuracy vs. OPD (19.20 vs 15.85) on Qwen3-4B-BaseQwen3-4B-Base 域内平均准确率高于 OPD（19.20 vs 15.85）

8.71×

cross-generation speedup, R1-1.5B ← Qwen3-4B, no pair-specific tuning跨代组合 R1-1.5B ← Qwen3-4B 的加速，无需针对性调参

\hat{E} \geq E

provably never stops before the defined reliability boundary可证明：停止点永远不会早于所定义的可靠性边界

01 · The problem01 · 问题

One horizon per batch can't fit every trajectory.整个 batch 共用一个 horizon，适配不了每条轨迹。

OPD accelerators like TOPD (fixed truncation) and Prune-OPD (feedback-adapted horizon) choose a single rollout length M per batch. Reliable lengths vary across trajectories, so any M both cuts useful supervision on some rollouts and wastes decoding on others. Drag the horizon and see for yourself.TOPD（固定截断）和 Prune-OPD（按反馈自适应 horizon）这类 OPD 加速方法，都给整个 batch 选同一个 rollout 长度 M。可不同轨迹的可靠长度差别很大：任何一个 M 都会在一些 rollout 上截掉有用的监督，又在另一些上白白浪费解码。拖动下面的 horizon 亲自试试。

six rollouts of one prompt, illustrative boundaries同一 prompt 的六条 rollout，边界为示意

Shared horizon M共享 horizon M500

reliable tokens cut early (supervision lost)被提前截断的可靠 token（监督损失）

tokens decoded past the boundary (wasted)越过边界后解码的 token（计算浪费）

What the paper measured论文实测

Tokens decoded past the reliable length E per trajectory (Qwen3-1.7B-Base ← Qwen3-4B, Fig. 2 right). No method had any steps cut.每条轨迹越过可靠长度 E 之后多解码的 token 数（Qwen3-1.7B-Base ← Qwen3-4B，Fig. 2 右）。所有方法都没有被截断的步。

Retained length Ê within one training step, min–max averaged over training (Table 7)同一训练步内保留长度 Ê 的 min–max，取全程平均（Table 7）

G1830–3,509

G2335–1,052

G3126–672

G4293–2,117

Why it happens为什么会这样

Reliability boundaries are heterogeneous within a batch and drift over training. On R1-1.5B ← JustRL-1.5B, TOPD's fixed 4,096 first exceeds the whole reliability range and later falls below its mean. On Qwen3-1.7B-Base, both TOPD and Prune-OPD stay far above E_max. A more sensitive horizon controller can't fix this, because the decision is made at the wrong granularity.可靠性边界在 batch 内高度异质，并且随训练不断漂移。在 R1-1.5B ← JustRL-1.5B 上，TOPD 固定的 4,096 起初超过整个可靠范围，后来又低于其均值；在 Qwen3-1.7B-Base 上，TOPD 和 Prune-OPD 都远高于 E_max。换一个更灵敏的 horizon 控制器也解决不了，因为决策的粒度本身就错了。

02 · The idea02 · 核心思想

Don't predict the boundary. Verify it.不去预测边界，而是验证边界。

Flash-OPD counts top-k disagreements between student and teacher. Reliability is defined as a first-passage time of that count. The boundary is unknown before sampling, but whether a prefix has crossed it can be checked exactly.Flash-OPD 统计学生与教师之间 top-k 不一致的次数，并把可靠性定义为这个计数的首达时间（first-passage time）。边界在采样前无从得知，但一个前缀是否已经越过边界，可以被精确判定。

Low-compatibility event at position τ (Eq. 3)位置 τ 上的低兼容事件（Eq. 3）

φ_{τ} = 𝟙 [\frac{| K_{τ}^{S} \cap K_{τ}^{T} |}{k} < η]

Reliability boundary = first passage of the count to the allowance B (Eq. 4)可靠性边界 = 计数首次达到容忍预算 B 的位置（Eq. 4）

c_{τ} = \sum_{i = 1}^{τ} φ_{i}, E = min {τ : c_{τ} \geq B}

Next verification point from the recent event rate (Eq. 5)由近期事件率决定下一个验证点（Eq. 5）

k \leftarrow k + max (k_{min}, ⌈ \frac{ρ (B - c)}{{\hat{λ}}_{k}} ⌉), {\hat{λ}}_{k} = \frac{c_{k} - c_{k - W}}{W}

Trajectory-specific轨迹特异

Each rollout has its own E, so the length decision has to be made per sequence.每条 rollout 都有自己的 E，所以长度决策必须逐条序列来做。

Exact first-passage精确首达

E is the hitting time of an integer counter. It is checked on the observed prefix and never estimated.E 是一个整数计数器的首达时间，直接在已观测的前缀上判定，从不估计。

Posterior后验性

E is unknown until y is sampled. So stopping has to move inside generation.在 y 被采样出来之前，E 是未知的，所以停止判断必须挪到生成过程内部。

STEP i步骤 i

Generate to k生成到 k

Resume from the cached prefix and decode only new positions. Each token is decoded once.从缓存的前缀继续，只解码新位置，每个 token 只解码一次。

STEP ii步骤 ii

Verify the count验证计数

Run the teacher and student on the new block and update the exact count c_k. Locality and permanence keep earlier rewards unchanged.在新块上运行教师和学生，更新精确计数 c_k。局部性与持久性保证之前的 reward 不会改变。

STEP iii步骤 iii

Stop or advance停止或推进

If c ≥ B, retire at the first crossing. Otherwise jump a fraction ρ of the estimated distance to the boundary.若 c ≥ B，在首次越界处停止；否则按估计的剩余距离向前跳 ρ 的比例。

The estimate decides when to look.估计只决定何时去看。

The trailing-window rate λ̂ only places the next checkpoint: big jumps far from the boundary, small steps near it. Under a stable rate this takes O(log B) rounds, and always at most 1 + ⌈(L_max − k₀)/k_min⌉.滑动窗口事件率 λ̂ 只用来放置下一个检查点：离边界远时大步跳，靠近时小步逼近。事件率平稳时只需 O(log B) 轮，最坏也不超过 1 + ⌈(L_max − k₀)/k_min⌉ 轮。

The count decides whether to stop.计数决定是否停止。

Proposition 1: $\hat{E} \geq E$ for any ρ ∈ (0, 1], any W, and any realization of λ̂. A bad estimate costs a few extra tokens (4.5%–10.9% overshoot in practice) and never any supervision.Proposition 1：对任意 ρ ∈ (0, 1]、任意 W 以及 λ̂ 的任意取值，都有 $\hat{E} \geq E$ 。估计不准只会多解码几个 token（实测 overshoot 为 4.5%–10.9%），永远不会损失监督。

03 · Interactive playground03 · 交互演示

Watch one trajectory verify itself.看一条轨迹如何边生成、边验证。

This runs the batched stopping rule from the paper's Algorithm 2 on one trajectory. Paper case replays Figure 3 exactly. Simulate draws events from a synthetic drift profile so you can stress-test the controller. Note that the count curve is only drawn for positions that have actually been verified.这里在单条轨迹上运行论文 Algorithm 2 的 batched 停止规则。论文案例精确复现 Figure 3；模拟模式从合成的漂移曲线中采样事件，可以拿来压力测试控制器。注意：计数曲线只画到真正被验证过的位置。

verified已验证 newly verified (this round)本轮新验证 low-compatibility event φ=1低兼容事件 φ=1 decoded, dropped (overshoot)已解码但丢弃（overshoot） verification point k验证点 k stop Ê停止点 Ê λ̂ extrapolationλ̂ 外推

#	decode解码区间	new events新事件	count c计数 c	rate λ̂事件率 λ̂	decision决策

04 · Results04 · 主要结果

Fastest in every pair, without giving up accuracy.每组都最快，而且不牺牲准确率。

The setup trains on DAPO-Math-17K for 203 steps (64 prompts × 4 samples) on 8× H20 and evaluates mean@16 on five competition-math benchmarks with a 31,744-token budget. All runs use one fixed Flash-OPD configuration: η = 0.7, B = 200, ρ = 1.0, k₀ = W = 128.在 DAPO-Math-17K 上训练 203 步（每步 64 个 prompt × 4 个样本），使用 8× H20；在五个竞赛数学 benchmark 上报告 mean@16，解码预算 31,744 token。所有实验使用同一组 Flash-OPD 配置：η = 0.7，B = 200，ρ = 1.0，k₀ = W = 128。

Accuracy by benchmark各 benchmark 准确率

mean@16, %

Training time训练时间

hours, 203 steps小时，203 步

The accuracy–cost frontier准确率–成本前沿

Each point is one accelerator on one teacher–student pair, showing speedup over vanilla OPD against the change in average accuracy. Flash-OPD sits farthest right in every pair. Its largest accuracy gain is on Qwen3-4B-Base ← Qwen3-4B, where students are prone to length runaway, and cutting off unreliable supervision appears to act as an implicit curriculum and regularizer.每个点代表一种加速方法在一组教师–学生组合上的结果：横轴是相对 vanilla OPD 的加速比，纵轴是平均准确率的变化。Flash-OPD 在每一组都位于最右侧。准确率提升最大的是 Qwen3-4B-Base ← Qwen3-4B：这组学生容易出现 length runaway，及时截掉不可靠的监督，似乎起到了隐式课程与正则化的作用。

It isn't a free lunch everywhere. On R1-1.5B ← R1-7B Flash-OPD is 5.32× faster but 0.61 points lower on average, the largest drop among the four pairs.它也不是处处免费。在 R1-1.5B ← R1-7B 上，Flash-OPD 快 5.32×，但平均准确率低 0.61 分，是四组里下降最多的。

Full Table 2 (all pairs, all methods)完整 Table 2（全部组合与方法）

05 · Where the time goes05 · 时间花在哪里

Shorter rollouts, no separate log-prob pass, cheaper updates.更短的 rollout、省掉单独的 log-prob 计算、更轻的参数更新。

Per-step wall-clock decomposition (Table 6). Verification already computes student and teacher outputs, and these are reused in the loss, so the log-prob stage disappears. Fewer retained tokens also shrink the policy update.每步 wall-clock 时间分解（Table 6）。验证阶段已经算出学生和教师的输出，并在 loss 中直接复用，所以 log-prob 阶段整个消失；保留的 token 变少，policy update 也随之变轻。

Seconds per training step每个训练步耗时（秒）

Retained length vs. OPD response length保留长度 vs. OPD 响应长度

OPD response lengthOPD 响应长度Flash-OPD mean ÊFlash-OPD 平均 ÊFlash-OPD min–maxFlash-OPD min–max

Mechanism counters机制统计

Table 7 · per-step averages over 256 trajectories and all 203 stepsTable 7 · 每步 256 条轨迹、全部 203 步的平均值

Pair组合	Ê min / mean / max	Stop: allowance / L_max / EOS停止原因：预算 / L_max / EOS	Verify rounds验证轮数	Verify amplification验证放大	Overshoot
G1 R1-1.5B ← JustRL-1.5B	830 / 2,708 / 3,509	130 / 96 / 30	2.64	1.54×	231 (8.5%)
G2 R1-1.5B ← R1-7B	335 / 585 / 1,052	256 / 0 / 0	6.23	2.41×	26 (4.5%)
G3 Qwen3-1.7B-Base ← Qwen3-4B	126 / 304 / 672	243 / 0 / 13	4.98	2.23×	14 (4.5%)
G4 Qwen3-4B-Base ← Qwen3-4B	293 / 569 / 2,117	250 / 0 / 6	6.23	2.19×	62 (10.9%)

The allowance decides most stops on G2–G4. G1 uses k₀ = W = 1024, k_min = 512 and L_max = 4400 because its trusted prefix is typically longer, so many of its trajectories reach the context limit.G2–G4 上绝大多数停止由容忍预算触发。G1 的可信前缀通常更长，因此使用 k₀ = W = 1024、k_min = 512、L_max = 4400，不少轨迹会触达上下文上限。

06 · Transfer, scale, robustness06 · 迁移、扩展与鲁棒性

The gains hold up off-distribution and at scale.在分布外和更大规模下，收益依然成立。

Out-of-domain transfer分布外迁移

Flash-OPD minus OPD accuracy (points), mean@4. GPQA is not mathematics. Hover a cell for the raw values.Flash-OPD 减 OPD 的准确率差（分），mean@4。GPQA 为非数学任务。悬停可查看原始数值。

Macro differences stay within 1.4 points on G1–G3. On G4 every held-out benchmark improves, for a +5.84 macro gain.G1–G3 上宏平均差异都在 1.4 分以内；G4 上所有 held-out benchmark 都有提升，宏平均 +5.84。

Across model generations跨模型代际

R1-1.5B student, Qwen3-4B teacher, same Flash-OPD config (Table 9)学生 R1-1.5B，教师 Qwen3-4B，Flash-OPD 配置不变（Table 9）

Matches OPD (28.45 vs 28.34 avg.) while cutting training from 12.02 h to 1.38 h.平均准确率与 OPD 持平（28.45 vs 28.34），训练时间从 12.02 h 降到 1.38 h。

Scaling the teacher: 4B → 14B → 32B扩大教师规模：4B → 14B → 32B

Qwen3-1.7B-Base student (Fig. 7)学生固定为 Qwen3-1.7B-Base（Fig. 7）

From 4B to 32B, step time rises by 26.5 s for Flash-OPD and 100.6 s for OPD, which is 73.7% less added time. Flash-OPD accuracy stays at 8.5–8.9% while OPD falls to 7.38%, and the Flash-OPD rollout length barely moves (304 → 319).从 4B 到 32B，Flash-OPD 每步时间只增加 26.5 s，OPD 增加 100.6 s，即新增耗时少 73.7%。Flash-OPD 准确率稳定在 8.5–8.9%，OPD 则降到 7.38%；Flash-OPD 的 rollout 长度几乎不变（304 → 319）。

Robustness to η, B and ρ对 η、B、ρ 的鲁棒性

Cell: avg. accuracy / retained tokens. The outlined cell is the default (η = 0.7, B = 200). Color shows the change from the default.格内为平均准确率 / 保留 token 数；描边格为默认配置（η = 0.7，B = 200），颜色表示相对默认值的变化。

allowance B → | threshold η ↓容忍预算 B → | 阈值 η ↓

Step discount ρ on R1-1.5B ← R1-7BR1-1.5B ← R1-7B 上的步长折扣 ρ

ρ	0.50	0.70	0.85	1.00
Avg. (%)平均（%）	29.86	29.67	30.11	29.75
Verify rounds验证轮数	11.08	8.51	7.23	6.23
Token overshoot	8	10	15	26

Performance only degrades at extreme thresholds, and a larger B partly offsets a high η. ρ trades rounds against overshoot, while accuracy stays within a 0.44-point band, as Proposition 1 predicts.只有阈值取极端值时性能才明显下降，且更大的 B 能部分抵消过高的 η。ρ 在验证轮数和 overshoot 之间权衡，准确率始终落在 0.44 分的区间内，与 Proposition 1 的预期一致。

07 · Limitations07 · 局限

What the guarantee does and doesn't cover这个保证覆盖什么、不覆盖什么

Compatibility proxy. Top-k overlap can be evaluated online on partial prefixes, but the stopping guarantee covers the boundary this criterion defines. It doesn't directly guarantee downstream accuracy. Richer signals that can still be computed online are an open direction.兼容性代理指标。Top-k overlap 可以在部分前缀上在线计算，但停止保证针对的是该准则所定义的边界，并不直接保证下游准确率。更丰富、且仍能在线计算的信号是一个开放方向。

Token-level only. Sequence-level objectives such as outcome rewards need the finished response, so applying this stopping rule directly could remove information the reward depends on. Future work targets multi-teacher OPD and agent tasks.仅适用于 token 级监督。outcome reward 这类序列级目标依赖完整回答，直接套用该停止规则可能丢掉 reward 所需的信息。后续工作将扩展到多教师 OPD 和 agent 任务。

08 · Citation08 · 引用

BibTeX

@article{chen2026flashopd,
  title   = {Flash-OPD: Fast On-Policy Distillation},
  author  = {Chen, Wei and Chen, Junle and Yang, Yitong and
             Xu, Zhaoyang and Lin, Jiaxin and Liang, Yuxuan and
             Zhou, Xiaofang and Wang, Kai and Chen, Rui},
  journal = {Preprint},
  year    = {2026}
}