ThinkPrior Read paper
Public preprint RLVR · Prompt selection · 2026

Before rollout history exists

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

ThinkPrior uses an external-anchor zero-rollout difficulty prior for cold-start prompt selection, reducing silent groups before RLVR history is available.

Download preprint Inspect the evidence

Tommy Sha Skylar Zhai Siqi Zhao

Research question

How do you select useful prompts before the policy has generated the history that existing selectors need?

Use a small external anchor to estimate pool difficulty once, before training. Convert its verifier-scored pass rate into a finite Beta initialization, rank prompts by expected learnability, then let target-policy outcomes update the posterior online.

A named bottleneck

Silent group /ˈsaɪ.lənt ɡruːp/

An all-correct or all-wrong rollout group whose group-relative advantages are identically zero. Under the studied KL-free objective, it contributes no reward-advantage gradient.

39% of rollouts under uniform sampling were spent on silent groups across the studied run. This is a measurement on the reported pool, not a universal RLVR constant.

The method

A prior that arrives before policy history.

ThinkPrior changes prompt selection and posterior updates. It does not change the training loss or optimizer.

ThinkPrior framework: a pool of 250 prompts is scored once by a 3-billion-parameter external anchor with 16 samples per prompt. The verifier-scored pass rate initializes expected learnability before training. Unlike an uninformative online posterior at step zero, ThinkPrior can rank prompts before any target-policy rollout.
Figure 1 An offline anchor pass supplies a zero-rollout difficulty prior. The training policy is not queried to build this initialization.
  1. 1

    Probe once

    For every prompt x, sample the external anchor k = 16 times and use the verifier-scored pass rate φ̂(x) as a step-zero difficulty estimate.

  2. 2

    Initialize softly

    Encode φ̂(x) as finite Beta pseudo-counts with κ = 4. The prior biases the cold start without pinning the later trajectory.

  3. 3

    Select, then adapt

    Rank candidates by posterior expected learnability. After selection, target-policy outcomes update the same posterior online.

Supported findings

Waste separates. Final accuracy does not.

Qwen2.5-Math-7B, GRPO with LoRA, a 250-prompt MATH pool, and 16 seeds per arm for the prior-isolating ThinkPrior versus online-NP comparison.

Two-panel result figure. ThinkPrior lowers silent@10 from 23.8 percent without the prior to 10.6 percent, a 13.1-point reduction. Final MATH500 accuracy trajectories overlap across training.
Prior-isolating evidence Across 16 seeds per arm, ThinkPrior reduced silent@10 from 23.8% to 10.6% (-13.1 points; 95% CI [-16.4,-9.9]). Final MATH500 was +0.68 points with 95% CI [-2.2,3.5]; because that interval crosses zero, no final-accuracy advantage is established. The evidence therefore supports less early waste, not a final-accuracy gain.

The evidence supports a cold-start benefit from the prior. It does not establish a final-accuracy advantage.

Headline comparison: ThinkPrior versus the same posterior and selection rule initialized without a prior.
Measure ThinkPrior Online-NP Observed difference Interpretation boundary
silent@10 10.6% ± 3.6 23.8% ± 5.1 −13.1 points; −55% relative 80 candidate groups per seed in steps 1–10; 95% CI [−16.4, −9.9] points.
waste@30 266 ± 30 329 ± 77 −63 rollouts; −19% Rollouts in silent candidate groups through step 30; 95% CI [−106, −20].
Final MATH500 56.9% ± 3.7 56.2% ± 4.2 +0.68 points 95% CI [−2.2, 3.5] points; we detect no difference in final accuracy.

Measured composition

ThinkPrior + DAPO uses less generation in the reported comparison.

Over 60 steps and three seeds per arm, ranking DAPO candidates with ThinkPrior reduces generated rollouts from 8,256 to 7,381 (−10.6%). Both arms retain the same 3,840-rollout update budget and the same observed mean MATH500 accuracy of 60.5%.

The composition changes the candidate pool. This supports compatibility, not isolation of the initialization as the sole cause.

Generated rollouts
7,381 vs. 8,256
Silent generated rollouts
2,131 vs. 3,325
Update rollouts
3,840 vs. 3,840
Seeds
3 per arm
Training curves for silent-group fraction and cumulative wasted rollouts. ThinkPrior starts below the no-prior baseline, while the margin narrows by step 40 on the finite 250-prompt pool.
Figure 3 The cold-start margin is front-loaded and closes by step 40 on the finite 250-prompt pool; the figure does not identify why it closes.

Limits & availability

A narrow claim, stated plainly.

ThinkPrior is presented as a cold-start selection prior—not a universal RLVR solution, a guaranteed accuracy improvement, or a fully off-policy selector.

01

Fixed-budget accounting

On the reported 250-prompt pool, the ThinkPrior versus online-NP result reallocates a fixed 3,840-rollout budget; it is not a net generation saving.

02

The anchor is not free

The one-time probe uses 4,000 small-model generations. The paper does not claim a measured full-cost break-even point.

03

Online after initialization

No target-policy rollout is needed before the first selection. Thereafter, the posterior uses target-policy outcomes.

04

Artifact status

Code, complete data, and training trajectories are not currently public. This repository publishes the preprint, selected explanatory figures, and citation surfaces only.

Citation

Cite the public preprint.

No DOI or arXiv identifier has been assigned on this public surface. Use the canonical project URL and update the record after a scholarly identifier exists.

BibTeX file CITATION.cff Paper PDF
BibTeX 
@misc{sha2026thinkprior,
  title  = {ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR},
  author = {Sha, Tommy and Zhai, Skylar and Zhao, Siqi},
  year   = {2026},
  note   = {Preprint},
  url    = {https://shatianming5.github.io/thinkprior/}
}