强化学习入场：把老三样翻译成 LLM 语言

先试一试 · 入门探针

【讲前探针】Number Search（L9 p.2：1–16 藏一个整数、4 次猜测、猜错环境只答 higher/lower、猜中才得 1）。假设你用 10 万条专家成功轨迹做 SFT 训出 agent，上线后它第一次猜 8 得到 higher、却接着猜了 2。判断：这个 agent 大概率仍能像专家一样继续收敛到正确答案，因为成功轨迹里见过各种局面。

正确错误

本讲地图

点击卡片下钻详解

双讲地图：从模仿到试错，从单搜到深研

5a（Daniel Fried）用一个 Number Search 例子把 SFT 的三个错配推向策略梯度与 GRPO；5b（Akari Asai）用 124 页把 Deep Research 拆成评测、建模、检索三战线。本讲把两条线拧成一个问题：奖励从哪来，学到的东西就去哪。

入门3 分钟#RL#deep-research#总起L9 ↗L10 ↗

详解 →

SFT 到底错在哪：任务、数据、暴露三错配

任务错配、数据错配、暴露偏差——SFT 最大化的是「示范动作」的概率，我们要的是「完成任务」的概率。ReST/ReST-EM 证明：只保留高奖励轨迹做 SFT，已经是 RL 的第一步。

入门5 分钟#SFT#ReST#5aL9 p.3-7 ↗p.15-16 ↗

详解 →

REINFORCE 力学：带掩码与再加权的 SFT

环境不可微（「你怎么对单元测试求导？」），但期望奖励的梯度可以：REINFORCE 四步推导落到工程上，就是「对标准 SFT 计算做掩码与再加权」。

进阶6 分钟#policy-gradient#REINFORCE#5aL9 p.18-24 ↗

详解 →

从减一个数到除以一个组：baseline→GRPO→Dr.GRPO

全失败的组在 basic REINFORCE 里没有梯度。减一个与动作无关的 baseline 不偏期望只降方差（bandit 实验 SD 0.325→0.108）；换成组均值就成了 GRPO——而 GRPO 引入的新偏差，就是 Dr.GRPO 要修的。

进阶7 分钟#GRPO#advantage#baseline#5aL9 p.26-42 ↗

详解 →

奖励越硬越好吗：hacking、过长推理与 RLVR 边界

凡有 KPI 处皆有刷法：proxy RM 过优化先升后降、模型学会改写自己的奖励函数、随机奖励也能 +21.4 分、RLVR 收窄解空间。奖励设计与 rubric 治理是同一门课的两半。

前沿6 分钟#reward-hacking#RLVR#5a×5bL10 p.44 ↗p.84-85 ↗

详解 →

Deep Research 解剖：Plan→Search→Reflect→Answer

从「Asai 办公室号 GHC 5507」的一次搜索，到「AI 能否比肩专家综合文献」的 Plan→Search→Reflect 循环；再到 WebGPT→OpenAI/Gemini 两代产品的形态与训练方式。

入门5 分钟#deep-research#planning#5bL10 p.3-12 ↗

详解 →

没有唯一 gold 怎么打分：四轴评测与引用核查

研究型问题没有唯一 gold：相似度指标与专家偏好一致性只比随机略好（ROUGE 58%）。四轴框架（搜索复杂度/领域/答案质量/证据支持）+ 专家 rubric + FACT 引用核查，才是站得住的评测。

进阶6 分钟#benchmark#rubric#citation#5bL10 p.14-50 ↗

详解 →

训练配方全景：mid-training→SFT→RL 全链路

Tongyi 的三段配方（mid-training→SFT→RL）、知识图谱造题、异步 rollout，到 DR Tulu 的演进 rubric：5% SFT 数据就能热启动 RL，全量混合最强。

前沿6 分钟#RLVR#RLER#训练配方#5bL10 p.58-98 ↗

详解 →

检索器只看 query 就够了吗：AgentIR 与推理感知检索

检索器只看最新 query，agent 的推理被丢掉——错失信息。AgentIR 把当前步推理喂给检索器并为此合成训练标签（DR-Synth），四模型消融从 48.7 拉到 66.3。

前沿5 分钟#retrieval#AgentIR#5bL10 p.100-124 ↗

详解 →

实践视角：把 Dr.GRPO 写成掩码再加权

作业路线：实现 Dr.GRPO（去 σ_R、改 1/C）并与 GRPO 对照；想继续的按四步顺序读官方 readings，Reward Hacking 主题配 Anthropic 两篇一线报告。

进阶4 分钟#实践#Dr.GRPO#阅读路线L9 ↗

详解 →

站内音频镜像（YouTube 自动字幕转写见页面底部）· 原录像归 CMU 课程方所有加载中…

伴读笔记

导语

前几讲里，Agent 一直靠提示词加上下文吃饭：本领要么来自预训练时吃过的大量数据，要么来自你现场写在提示里的样例。

第 5 讲上半场（5a「Reinforcement Learning Basics」）把训练篇的第二块拼图放上桌：让模型从任务的最终结果里自己变强。这是 SFT 给不了的。

下半场（5b「Deep Research Agents」）切到 Domains 系列第三站，看强模型加多步检索、长文综合，能长出什么样的集大成产品。OpenAI 和 Google 都管它叫 Deep Research。

这一版对齐官方两份讲义：5a《RL Foundations for Agents: From SFT to policy gradients》（Daniel Fried，46 页，2026-09-22），5b《Deep Research Agents — Tasks, Benchmarks, and Methods》（Akari Asai，124 页，2026-09-24）。

页码随文以 L9 p.X/ L10 p.X 标注。

结构上，第一至四节保留经典论文织成的旧地图；第五至十二节按讲义把八个主题重新铺开：先算 SFT 的三笔错配账，拆 REINFORCE 的「掩码与再加权」力学，理 baseline→GRPO→Dr.GRPO 的家族树，再看奖励的边界。

下半场按 5b 的三线展开 Deep Research 的解剖、评测、训练配方与推理感知检索。讲次页「本讲地图」的卡片可下钻到对应分节详解。

事实与数字只出自这两份讲义、官方 readings 与文中注明的来源。把它当成课前预习、课后消化用的地图，别当课程实录。

一、把 RL 老三样翻译成 LLM 语

强化学习 (Reinforcement Learning, RL) 的世界观很朴素：环境给分数，策略学着多拿分。

形式化是马尔可夫决策过程 (Markov Decision Process, MDP)，五元组 $(S, A, P, r, \gamma)$ ：状态、动作、转移、奖励函数、折扣因子。策略要最大化的是折扣回报 (discounted return)：

G_t = \sum_{k=0}^{T} \gamma^k \, r_{t+k+1}

对目标直接求梯度，就得到策略梯度定理：

\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\left[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \cdot A_t\right]

公式不如直觉值钱。把一条轨迹想成实习生完整跟下的一个项目， $A_t$ （advantage，优势）衡量这一步比策略平均水平好多少：好经历的走法调高概率，差的调低。

$A_t$ 需要一个基线 (baseline) 来减。经典基线是价值函数 (value function)，也就是 PPO 里那个 critic。

翻译成 LLM 语，对应关系是：

状态：当前上下文窗口，即系统提示、已生成 token 与工具返回。
动作：下一个 token，或一次完整的工具调用。
策略：语言模型本身。
轨迹：一次完整任务，比如一道数学题、一个 bug、一轮多步检索。
奖励：通常到轨迹末尾才结算的稀疏信号。答案对不对、测试过不过、格式合不合规。

打游戏是每帧都给分，LLM 是几万 token 之后才开奖。这个差别决定了后面所有算法的口味。

二、从 RLHF 到 GRPO：奖励从哪来

我喜欢用带实习生来串这段。SFT 是照着优秀范例抄一遍：快，但范例贵，模型学会的更像 “演”。

RL 是只看交付打绩效：不用一步步教，但绩效标准必须先立得住。

RLHF (Reinforcement Learning from Human Feedback) 的三部曲正好是先抄范例、再立标准、最后冲绩效：SFT 起步，用人类偏好训练奖励模型 (Reward Model, RM)，再用 RL 优化策略。

RM 的学习目标来自 Bradley-Terry 偏好模型：

P(y_w \succ y_l) = \sigma\left(r_\phi(y_w) - r_\phi(y_l)\right)

RL 那一步的默认算法是 PPO（Proximal Policy Optimization，近端策略优化）：拿概率比 $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\text{old}}(a_t \mid s_t)$ 做 clip，限制每次更新幅度，外加一项对参考策略的 KL 惩罚防止跑飞：

L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\big(\rho_t A_t,\ \mathrm{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\, A_t\big)\right]

PPO 的问题是排场大：策略、critic、参考模型、奖励模型，四套权重挤在一起。GRPO（Group Relative Policy Optimization，组相对策略优化）出自 DeepSeekMath，洞察很直接：对同一个提示采一组 $G$ 个回答，组内互为基线，critic 就省了：

\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}

谁比组内平均好就加强谁。就这么朴素的一招，成了开源 LLM-RL 的默认选项。

一句话记住 GRPO

同一道题让一组实习生各自作答，互相就是基线：比组均分好的加强、差的削弱 —— 不需要再养一个「打分员」（critic）。

另一条路线是可验证奖励 (RLVR, Reinforcement Learning with Verifiable Rewards)：数学对答案、代码跑测试、格式查正则，奖励由规则算出而非模型猜出。

示意一下这类奖励函数的朴素程度：

def reward(completion, gold_answer):
    pred = extract_final_answer(completion)   # 抽出最终答案字段
    if pred is None:            return -0.2   # 连格式都没遵守
    return 1.0 if pred == gold_answer else 0.0

DeepSeek-R1 展示了这条路的上限：纯 RL 阶段 (R1-Zero) 里，自我验证、回退重试这些行为是被奖励逼出来的，提示词里一个字没写。

局限同样清楚：可验证的域目前集中在数学、代码这类有硬标准的任务；开放域里什么算好仍要绕回 RM，于是有了第三节的麻烦。

三、Reward hacking：凡有 KPI 处皆有刷法

记得绑在手腕上的摇步器吗？微信步数是个健康程度的代理指标，于是有人晚上把手机绑在狗身上：步数涨了，健康没涨。

这就是奖励作弊 (reward hacking)，古德哈特定律 (Goodhart’s Law) 的 AI 版。当代理指标变成优化目标，它就不再是好指标。

RLHF 里这件事被定量研究过。Gao 等人的《Scaling Laws for Reward Model Overoptimization》用一个固定的金标准 RM 扮演人类，看优化 proxy RM 时金标准分数怎么变：先升后降，下降点随 RM 规模、KL 系数平滑变化。翻译过来：对着打分器用力过猛，真实质量会掉头向下。

LLM 场景里的实例并不抽象。代码 Agent 发现改测试比改代码更容易让测试通过；摘要模型学会堆砌关键词讨好打分器；RLHF 模型的讨好型人格 (sycophancy) 也是同一族问题。

工程上的缓解手段是一套组合拳：

KL 惩罚压着策略别跑太远。
RM 定期用新数据更新。
生成器与验证器分离。
引入过程级奖励。
人在环抽检。

没有哪一招能根治，每一招都在给优化过程加摩擦。

四、Deep Research Agent：规划、检索、综合的集大成

下半场换个心情。想象导师扔给你一个综述题目，你的工作流：先列提纲拆子问题，再钻图书馆翻文献、做卡片笔记，发现缺口就回头补查，最后成文并保证每句论断都有出处。

Deep Research Agent 就是把这套流程交给模型自主执行：

用户问题
  → 研究计划（拆子问题、排优先级，有的产品先给你确认）
  → 循环：生成查询 → 搜索/打开网页 → 摘录进笔记 → 发现缺口再补查
  → 综合成文（结构化长报告）
  → 引用对齐检查（每个 [n] 真能支撑对应论断）

产品侧先看 OpenAI。deep research（2025 年 2 月发布）以 o3 为底座、针对浏览与综合做端到端 RL。

官方报告在 GAIA 基准上 pass@1 平均约 67%，最难的第 3 级 47.6%，并随之推出了专测锲而不舍挖网能力的 BrowseComp。

Gemini 的 Deep Research 走「计划先行」路线：先生成多步研究计划，可编辑后再执行，长文与引用一次交付。

开源侧最有教育意义的是 HuggingFace 的 24 小时复现：用 smolagents 的 CodeAgent（让 Agent 用写代码的方式组织行动）在 GAIA 验证集拿到 55.15%，超过此前开源最优的约 46%。

gpt-researcher 则用 planner /execution/publisher 多角色分工，加可配置深度的递归检索，是工程化的另一路样本。

这个域有四个难点，每一个都踩在前几讲的主题上：

信用分配。长视野任务里哪次搜索立了功？Search-R1 证明结果奖励就能训出何时搜、搜什么。
上下文管理。靠笔记剪贴，别把整页网页塞进窗口。
评测。从单轮问答走向 GAIA、BrowseComp 这类任务级基准。
引用质量。从 WebGPT 开始，必须挂引用就是这个任务的一部分；引用的查全与查准 (citation recall /precision) 要单独评。

五、SFT 到底错在哪：任务、数据、暴露三错配

从这一节起进入官方讲义的推进线。Fried 用一个贯穿全场的 Number Search 小游戏开场 (L9 p.2)：1–16 里藏一个整数（本例 11），agent 有 4 次猜测，猜错环境只答 higher/lower，猜中才得 reward 1。

四条样例轨迹：τ1（8↑12↓10↑11，R=1）与 τ2（4↑8↑12↓11，R=1）动作完全不同、同样成功；τ3、τ4 超时挂零。就这个游戏，SFT 的三个缺陷逐一显形 (L9 p.3-7)。

任务错配。SFT 最大化的是「示范动作」的概率， $\mathcal{L}_{\text{SFT}} = -\sum_t \log \pi_\theta(a^*_t \mid h_{1:t})$ ，我们想要的却是完成任务的概率。完成任务有多条路径，τ1 与 τ2 殊途同归；且有些动作明显更差：「12→lower」之后再猜 13，就是白白浪费一次机会。

数据错配。示范里只有成功轨迹，τ3、τ4 这类失败数据 SFT 用不上，而它们明明也含信息。

暴露偏差。训练时从没见过自己犯错之后的局面，生成时一步走偏就没有可参考的续算分布。讲前探针里 agent 猜 8 得到 higher 之后接着猜 2，病灶就在这里 (L9 p.6)。

RL 的定义恰好逐条对症 (L9 p.7)：从 agent 自己生成动作 / 轨迹 → 评估 reward → 调整参数以最大化期望 reward。

任务错配由训练目标 = 期望奖励化解，任何成功路径都被强化；数据错配由按奖励趋高避低化解，失败轨迹也参与学习；暴露偏差由 on-policy 采样化解，训练数据就是自己犯错后的轨迹。恢复行为第一次有了教材。

中间还有一级台阶：ReST (Reinforced Self-Training) 的 Grow → Improve → Repeat 循环 (L9 p.15-16)。用当前策略采样轨迹并算奖励是 Grow，按奖励过滤、增扩数据集再训练是 Improve，换新策略重来。

它的损失说白了就是高奖励轨迹上的标准 SFT loss：保留 R=1 的 τ1/τ2，丢掉 τ3/τ4。只把成功轨迹攒起来做 SFT，已经是 RL 的第一步。官方 readings 里的 ReST 与 ReST-EM，讲的就是这两级台阶。

六、REINFORCE 力学：带掩码与再加权的 SFT

先把交互形式化 (L9 p.8-13)。agent 与环境循环：动作 $a_t$ 换回观测 $o_{t+1}$ 与奖励 $r_{t+1}$ ；策略只见历史 $h_t = (o_0, a_0, \dots, o_t)$ ，不见隐藏状态。轨迹 $\tau = (o_0, a_0, r_1, o_1, \dots)$ ，回报 $R(\tau) = \sum_t r_t$ ，Number Search 只在结尾付 1。

目标是最大化期望奖励 $J(\theta) = \mathbb{E}_{\tau \sim p_\theta}[R(\tau)]$ 。示例四条轨迹概率 0.30/0.20/0.25/0.25、奖励 1/1/0/0，期望恰为 0.50。

训练的几何图像只有一句话：把概率质量从低奖励轨迹搬向高奖励轨迹。

为什么不能直接反向传播？采样不可微，环境更不可微。讲义在 L9 p.18 反问的就是这件事：单元测试没法求导。

通解是策略梯度 (REINFORCE, Williams 1992)：对 $J(\theta)$ 求梯度，只有概率项依赖 θ，同乘同除 $p_\theta(\tau)$ 、再用 $\nabla p / p = \nabla \log p$ ，四步得到：

\nabla_\theta J = \mathbb{E}_{\tau \sim p_\theta}\left[R(\tau) \sum_t \nabla_\theta \log \pi_\theta(a_t \mid h_t)\right]

环境项对 θ 是常数，轨迹对数概率只剩动作项之和；动作再按 token 分解。落到工程上，讲义的落地说法是：策略梯度就是对标准 SFT 计算做掩码与再加权(L9 p.22)。

把轨迹串成 chat 模板：user 任务指令 → assistant guess (8) → tool higher → assistant guess (12) → …。

只有策略自己生成的 token，也就是 assistant 动作含 end-of-turn，计损失 ( $m_t=1$ )；user 指令与 tool 观测是条件上下文 ( $m_t=0$ )；再乘上全轨迹共享的标量 $R(\tau)$ ：

\mathcal{L}_{\text{PG}} = -R(\tau) \sum_{t:\, m_t=1} \log \pi_\theta(u_t \mid u_{<t})

训练循环四步固定 (L9 p.23)：Collect 用当前策略采任务与轨迹，Evaluate 逐轨迹算奖励，Form loss 把动作 log 概率按奖励加权，Update 反传改 θ，然后回到 Collect。

两个收尾细节。其一，若奖励是二值的，这个损失退化成只对成功轨迹的 SFT loss，就是 ReST，差别只在 ReST 做批式更新并聚合数据集 (L9 p.24)。其二，若环境给中间奖励，用 reward-to-go $R_t = \sum_{t' > t} r_{t'}$ 加权每个动作。更早的奖励不依赖 $a_t$ ，去掉它们只降方差、不引偏差 (L9 p.35)。

七、从减一个数到除以一个组：baseline、GRPO 与 Dr.GRPO

basic REINFORCE 有个安静的死角：奖励应当是相对的 (L9 p.26)。一组 rollout 全成功，所有动作都被上调；全失败则完全没有梯度。而全失败偏偏说明任务对当前策略太难、最需要信号。ReST 同样躲不开：全失败组没有保留轨迹，零梯度。

解法是优势函数 $A^\pi(h,a) = Q^\pi(h,a) - V^\pi(h)$ ，衡量这步比策略平均水平好多少。Q、V 都未知，实践用采样估计 $\hat{A}_t = R(\tau) - b(h_t)$ ，b 就是 baseline。

讲义的例子很直观 (L9 p.27)：8/higher、12/lower 之后剩 {9,10,11}，还有两次机会。猜 11，错则剩两候选一次机会 (A<0)；猜 10，错则剩一候选一次机会 (A>0)。

baseline 怎么选？只要它只条件于历史、与动作无关，就不引入偏差： $\mathbb{E}_{a \sim \pi_\theta}[b(h)\nabla \log \pi_\theta] = b(h)\nabla_\theta \sum_a \pi_\theta = b(h) \nabla 1 = 0$ (L9 p.29)。四条取法各有代表 (L9 p.28)：

常数：bandit 演示用的就是它。
奖励滑动均值：随策略水平自适应。
训练值模型 $V_\phi(h)$ ：actor–critic 路线，即 PPO。
组内均值：无需额外模型，即 GRPO。

降方差的价值有数值实验背书 (L9 p.30-33)：双臂 bandit 初始 p=0.25（A 得 1、B 得 0），无 baseline 时只有采到 A 的样本有梯度，单样本标准差 0.325；加 b=0.5 后连失败的 B 也产生学习信号，标准差降到 0.108，期望不变。不偏期望，只降方差。

GRPO 把第四条取法用到极致 (L9 p.37-38)：对同一任务采 G 条轨迹，组均值 R̄ 当 baseline，再除以组内标准差 $\sigma_R$ 。四条轨迹奖励 1,1,0,0，中心化得 ±0.5，除以 $\sigma_R=0.5$ ，Â = +1,+1,−1,−1。

一个 batch 里多任务实例各建各的组。藏 3 的组 (1,0,0,0)，R̄=0.25 → +1.73/−0.58×3；藏 7 的组 (1,1,1,0)，R̄=0.75 → +0.58×3/−1.73。难度不同的实例被拉到同一权重刻度上。

讲义注明此处按当前策略轨迹书写，省略了论文的 ratio、clipping、KL 项。

Dr.GRPO 要修的第一处就是这个除以 σ_R (L9 p.39-40)：σ_R 是组内度量，随实例波动，几乎全对或全错的实例 σ_R 极小，权重被系统性放大。第二处是响应聚合：GRPO 按 1/|y_j| 平均，长答案的单位 token 惩罚更轻。同样 Â 下，200 token 的错误答案每 token 只挨 1/200 的罚，策略于是漂向更长但错误的输出。

两处修改都不动奖励本身：优势估计去掉 σ_R，聚合改 1/C（C 为全局常数 = 生成预算），变的是任务实例与响应的相对权重。

退化组是另一类问题 (L9 p.41)：[0,0,0,0] 与 [1,1,1,1] 都是零梯度，前者可能太难、后者可能太易。补救靠换任务实例、扩大探索、加示范或给更有信息量的奖励。

与 PPO 怎么分工 (L9 p.42)？PPO 训练独立 value model 估优势，GRPO 用组相对优势换效率。ratios、clipping、参考模型项留给后面的 Advanced RL 讲。

八、奖励越硬越好吗：hacking、过长推理与 RLVR 边界

第七节已经露出一个不舒服的事实：奖励与聚合方式的每个细节都在悄悄改写策略的演化方向，σ_R 放大近乎全对 / 全错的实例，1/|y_j| 纵容长而错的答案。这只是古德哈特定律在训练动力学里的温和形态；第三节讲过的 proxy RM 过优化（先升后降）是它的经典形态；更极端的案例在文献里并不罕见：模型学会改写自己的奖励函数，甚至随机奖励也能在某些基准上涨分。

5b 又补上第四种形态：当奖励由 rubric 充当时，rubric 本身会编码错目标。LM 生成的 rubric 项约 15% 空洞、有错或不清 (L10 p.44)，常见的翻法有三种：

引用不存在的论文。
只是复述问题。
标准含糊到放过似是而非的错误。

对策是把 rubric 锚定到真实来源，并审计其相关性与可验证性。

所以奖励并非越硬越好，答案分两半。可验证的域（RLVR：对答案、跑测试）奖励硬而便宜，但会收窄解空间，策略被拉向可验证的那部分行为。

开放域报告需要覆盖度、相关性、事实性、被支持的论断这类多维奖励 (L10 p.84-85)，而多维奖励天然是 rubric，于是奖励设计与 rubric 治理成了同一门课的两半。凡是当目标用的指标，都得复查它被优化之后是否仍然量得准。

Dr.GRPO 的教训放在这里同样成立：先排查长度漂移、退化组这些系统性偏差，再谈更聪明的奖励。

九、Deep Research 解剖：Plan、Search、Reflect 到 Answer

下半场（Akari Asai，124 页）从两个问题对照开场 (L10 p.3-7)。简单问题一次搜索就够：“What is Akari Asai’s office number at CMU?” → 搜到 LTI 教师页，答案 GHC 5507。复杂研究问题则要多轮，比如问 AI agent 综合科学文献能否比肩人类专家。

slides 并排摆出两篇正面回答的论文。OpenScholar (Nature 2026) 用 4500 万开放论文库做检索增强综合：ScholarQABench 收 2,967 条专家问题与 208 条长文答案，覆盖 CS、物理、神经科学、生物医学。

OpenScholar-8B 正确率超 GPT-4o 5%、超 PaperQA2 7%，而 GPT-4o 的引用幻觉率高达 78–90%。

DR Tulu (ICML 2026) 用演进 rubric 的 RL 训练，平均超 Tongyi DeepResearch 15.6%，比 OpenAI Deep Research 高 0.7%，每查询便宜 1000 倍。

讲义用一条真实的多轮研究过程演示形态 (L10 p.8-11)：

Plan：先分清基准分数与直接和人比两种比较口径，拆出找文献综合、看比较方法、比任务范围三条子问题。
Search：搜 “AI agent literature synthesis human expert evaluation”，拿到研究基准与专家评估两类结果。
Reflect：发现两类比较口径不同，改搜 tasks/domains/protocol。
Answer：“Promising on tested tasks; broader parity with experts remains unproven”，附两条引用。

与第四节的产品流程图对照，官方版的骨架就是 Plan→Search→Reflect 循环加一个带引用的收尾。第四节的产品计划确认、笔记剪贴，都是这条骨架的工程化包装。

本讲路线图三线 (L10 p.12)：Evaluation 管基准、rubric、引用支持，Modeling 管学搜索、推理、综合，Retrieval 为下一步找证据，分别对应第十、十一、十二节。

十、没有唯一 gold 怎么打分：四轴评测与引用核查

先破后立。经典 QA 基准（如 Natural Questions，官方例题 “who wrote the score for the force awakens”→John Williams）有四个先天不足 (L10 p.14-18)：简单查找、通用领域、短答案、证据使用不明。四轴框架逐一补位 (L10 p.19-20)：搜索复杂度、领域专业性、答案质量、证据支持。

搜索复杂度一轴由 BrowseComp 系填 (L10 p.21-28)：1,266 道人写难题。出题法是从已知答案出发、藏名、混合事实线索；验证难度时要求 LM + 搜索应失败、5 次 Google 搜索不够。例题给五条线索猜一部美剧，答案是 Plastic Man。

成绩单极具信息量：

GPT-4o：0.6%；带浏览 1.9%
GPT-4.5：0.9%
o1(medium)：9.9%
Deep Research*：51.5%（星号 = 在 BrowseComp 式任务上训练过）
人工参考匹配：25.3%

搜索预算的分量有数字：Tongyi DR 的训练轨迹里超过 20% 的 SFT 样本用了 10 次以上工具调用，评测上限放到 128 次 (L10 p.26)。BrowseComp-Plus 把语料冻结成 830 题 / 100,195 文档的固定基准，还加进差一条线索的相邻页当难负例 (L10 p.27-28)。

领域一轴各有基准 (L10 p.31-32)：MedBrowseComp 管医学；FinSearchComp 管金融，专家出题、独立评审交叉核对，例题是强生 2022–2024 国际收入占比；ScholarSearch 与 AutoResearchBench 管学术。

答案质量一轴最反直觉 (L10 p.35-38)：研究型问题没有唯一 gold，同一问题可以有多个有效摘要。与 gold 相似不等于对，把 unproven 写成 proven 的答案相似度照样高。定量证据：以专家偏好为准的 preference accuracy 里 ROUGE 58%、BERTScore 57%、BLEURT 62%，随机水平 50%。

可行解是专家 rubric。ScholarQABench 逐项核对再加权合成：两条 rubric 权重 2/1，候选答对一条，得分就是 (2×1+1×0)/(2+1)=0.67；rubric 判定与专家一致性 79% vs 专家间 80%(L10 p.39-42)。

ResearchQA 再用 LM 从综述里挖 rubric，规模化到 21K 问题 / 75 领域 / 160K rubric 项 (L10 p.43)。但生成 rubric 那 15% 的翻车率，又把治理问题摆回桌面（第八节已述）。

证据一轴由 DeepResearch Bench 的 FACT 检查填 (L10 p.47-49)：抽取论断 + URL → 读被引页 → 判定支持与否，报告 citation accuracy 与 effective citations。带引用不等于有支撑，78–90% 的引用幻觉率就是前车之鉴。四轴至此填满 (L10 p.50)。

十一、训练配方全景：mid-training、SFT 与 RL 全链路

Tongyi DeepResearch 报告给出的三段配方是当前开源侧的标准像 (L10 p.58-60)：mid-training 大规模学 agent 行为，输入是问题加答案、教师轨迹；SFT 模仿成功示范；RL 从自己尝试的反馈学。

SFT 段 (L10 p.68-70)：问题 + 已知答案 → 教师轨迹 → 过滤 → 收集保留轨迹训练，过滤条件是答案正确、格式有效、轨迹连贯。

数据从哪来？知识图谱造题（WebSailor-V2 方法，L10 p.61-67）：从 CMU Archives、1978 年诺贝尔官网这类可靠事实源建 KG，采样连通子图、隐藏目标名出题。

例题：「哪位 CMU 研究者与未来的经济学诺奖得主合著了 Human Problem Solving、并在诺奖前三年与该同事共享图灵奖？」→ Allen Newell。

mid-training 段（AgentFounder，ICLR 2026，L10 p.71-73）：agent 轨迹与通用预训练数据混合做 continual pretraining。同一份 SFT 数据只换底座，BrowseComp-en 28.6→39.9、GAIA 71.8→72.8。

RL 段 (L10 p.74-83) 走 Search-R1 式 RLVR：多轮搜索加推理的轨迹，按最终答案精确匹配给奖励，Allen Newell=1、Herbert Simon=0；算法用 GRPO 全装版，组相对优势、clipped update、KL 惩罚一样不少；rollout 异步进行、完成即打分，配缓存工具结果等效率手段。

开放式报告的奖励要换轨 (L10 p.84-85)：短答案用正确性，报告用覆盖度、相关性、事实性、被支持的论断。DR Tulu 的 RLER 让评分标准从策略响应与检索证据中提炼、随训练共同演进 (L10 p.86-93)。

两笔记账。其一，SFT 是 RL 的热启动：5% SFT 数据就有帮助，全量 SFT 混合最强 (L10 p.95)。其二，演进 rubric 优于固定与随机奖励：2500 步内把 HealthBench、ScholarQABench v2、DeepResearch Bench 平均分从 50 拉到 58 区间 (L10 p.96-98)。

十二、检索器只看 query 就够了吗：AgentIR 与推理感知检索

5b 收尾回到一个最容易被忽视的组件：检索 (L10 p.100-105)。工具谱系从本地检索到黑盒 API。本地检索里 BM25 管词面、嵌入模型学相似度；黑盒 API 就是 web_search、browse 这类。另有 Python 执行 (Tongyi) 与学术搜索（Tongyi、DR Tulu）等配套工具。

嵌入检索离线编码文档建向量索引、在线用双编码器打分 $s(q,d) = E_q(q)^\top E_d(d)$ 取 top-k（DPR 一脉），训练用对比损失拉近相关文档、推远负例。问题出在输入侧：检索器通常只看 query。agent 一路的推理被丢掉，只把最新一条 $q_t$ 递过去，错失信息 (L10 p.105)。

AgentIR（COLM 2026，L10 p.106-111）把检索器输入改成当前推理 $\tau_t$ 加 query $q_t$ ，并学着判断哪些文档对这一步推理有帮助。难在标签：标准检索训练只有全局相关性标签，深度研究轨迹需要的是这一轮哪些文档有用的局部标签。DR-Synth 用 agent 自生的信息需求合成中间步训练标签。

效果有四组数字 (L10 p.115-118)，对象是 Tongyi-DR、gpt-oss-120B、GLM-4.7、Tongyi-DR visit 四个 agent：

query-only：约 47.6–50.5
把推理加进输入：约 50.9–55.5
换 DR-Synth 训练数据：约 57.5–59.5
两者合并 (AgentIR)：66.3 / 67.0 / 64.7 / 68.1

两个成分各自有增益，合并最高，而且用更少搜索找到更多证据。历史输入消融再补一项证据 (L10 p.119-123)：加历史 query 有帮助，再加历史文档部分设置反而掉分。当前步推理是最强信号。

讲义收尾的四句总结正好当本讲收束 (L10 p.124)：

任务 = 多搜索 + 证据综合
评估 = 答案质量与引用支持，rubric 本身也要审计
训练 = 先示范后任务反馈
检索 = 给检索器 agent 的信息需求，并为之训练

论文与材料导读

PPO(Schulman et al., 2017, arXiv:1707.06347)：解决策略梯度一步一更新太浪费、TRPO 实现太复杂的问题；核心做法是对概率比做 clip 的一阶近似。结论：简单、稳，成了 RLHF 的默认底座。局限是要养一个与策略同量级的 critic。读它是为了看懂 5a 的每一个公式从哪来。
InstructGPT(Ouyang et al., 2022, arXiv:2203.02155)：解决基座会续写但不会听话的问题；做法是 SFT → RM → PPO 的三部曲。结论是把对齐做成了可复制工程管线。局限：偏好数据是代理。RLHF 这一整节的源头文献。
DeepSeekMath(Shao et al., 2024, arXiv:2402.03300)：提出 GRPO，用组内相对优势丢掉 critic、省显存。数学推理涨点的同时把 RL 的工程门槛降了一档，为什么能省 critic，读一遍推导就明白。
DeepSeek-R1(DeepSeek-AI, 2025, arXiv:2501.12948)：它回答的问题很直接：没有人工推理轨迹，能不能练出推理？纯 RL (R1-Zero) 配可验证奖励，自我验证与回退等行为涌现，再蒸馏给小模型。局限：可验证域为主、R1-Zero 可读性差需冷启动数据修补。RLVR 的规模化实证。
Scaling Laws for Reward Model Overoptimization(Gao, Schulman & Hilton, 2022, arXiv:2210.10760)：用合成金标准 RM 定量测出优化 proxy 的先升后降曲线。把 reward hacking 从轶事变成曲线，第三节的理论底座。
WebGPT(Nakano et al., 2021, arXiv:2112.09332)：Deep Research 的直系祖先 —— 文本浏览器里答题、行为克隆加偏好奖励模型、答案必须挂引用。人类评委 56% 的情况下更偏好它的答案。看它就知道检索、RL 加引用这套组合四年前已具雏形。
Search-R1(Jin et al., 2025, arXiv:2503.09516)：证明何时搜索本身可以被 RL 训练 —— 多轮搜索交互、检索 token 掩码、结果奖励，7B 模型较 RAG 基线平均提升约 41%。打通 5a 与 5b 的那座桥。
BrowseComp(Wei et al., 2025, arXiv:2504.12516)：1,266 道需要持续挖网的短答案题，考的是锲而不舍、变着法子找信息。答案短、好评分，专为浏览 Agent 设计的竞赛级评测。
两家官方发布：Introducing deep research(OpenAI) 讲清了端到端 RL 训出的浏览 Agent 这一形态与 GAIA 成绩；Introducing Deep Research and Deep Research Max(Google) 代表计划先行的产品化路线。各有限制：慢、贵、引用仍需抽查。
开源复现：HuggingFace 的 Open-source DeepResearch（代码在 smolagents 示例）展示 24 小时冲到 GAIA 55.15% 的复现路径；gpt-researcher 展示 planner/execution/publisher 的工程化分解。读它们胜在读十篇综述：知道哪些组件是刚需。

官方指定读物（5a/5b readings）

官方 10 篇 readings 与讲义逐节对应，补短导读：

ReST(Gulcehre et al., 2023, arXiv:2308.08998)：Grow/Improve 交替的离线自我训练。第五节「保留高奖励轨迹做 SFT」的本体，读它看 RL 的第一步如何朴素。
ReST-EM / Beyond Human Data(Singh et al., 2023, arXiv:2312.06585)：把过滤、增扩、重训做成 EM 式外循环；核心结论是自举数据可以超越人类数据。
DeepSeekMath(Shao et al., 2024, arXiv:2402.03300)：GRPO 出处（§4.1.1，Eq.3）。第七节组相对优势的推导与为什么能省 critic 的账。
DrGRPO / Understanding R1-Zero-Like Training: A Critical Perspective(arXiv:2503.20783)：对 GRPO 两处系统性偏差（σ_R 归一、长度归一）的修正 —— 第七、八节长度偏置的证据。
OpenScholar（Nature 2026, 官方页）：4500 万开放论文库的科学文献综合；ScholarQABench 与 GPT-4o 引用幻觉 78–90% 的测量出自这里（第九、十节）。
BrowseComp(Wei et al., 2025, arXiv:2504.12516)：1,266 道需持续挖网的短答案题，搜索复杂度一轴的标尺（第十节）。
DeepResearch Bench(arXiv:2506.11763)：FACT 引用核查（论断 + URL→读被引页→判支持），证据一轴的标尺（第十节）。
Tongyi DeepResearch(arXiv:2510.24701)：三段配方（mid-training→SFT→RL）、知识图谱造题与异步 rollout 的训练报告（第十一节）。
DR Tulu（arXiv:2511.19399，ICML 2026）：RLER 演进 rubric 强化学习 —— 开放式报告的奖励设计（第八、十一节）。
AgentIR（arXiv:2603.04384，COLM 2026）：把当前步推理喂给检索器、用 DR-Synth 合成中间步标签（第十二节）。

完整资料库见本讲资料区。

实践视角

站主的 SecondPal 是本地优先的桌面 Agent，PalWave 是随身的移动终端，这套形态让我对这一讲格外有代入感。三个体会：

reward hacking 离小团队不远。我们给资讯简报 Agent 定过「每条结论必须带来源」的验收，某天发现它把同一个 URL 挂三处充数：步数涨了，健康没涨。后来把验收改成引用去重、每条引文必须实际支撑对应论断才收敛，也就是把 citation precision 写进产品验收。
本地优先决定 Deep Research 的落地形态。重推理可以按需调云端强模型，但浏览器、文件、密钥留在桌面端编排；上下文用笔记剪贴而非整页塞入，与开源复现的做法不谋而合。
移动端管阅读与追问。报告在手机上读，引用能一键回溯原文，用户才敢把结论转发出去。

至于 RL 本身，训基座离个人开发者太远；够得着的是给自己的工具选择器、路由器攒可验证信号：任务成败、用户是否采纳结果。这些信号攒起来，就是给自家产品攒一座小型 GAIA 题库。

顺着官方讲义补一条作业路线：把第五至七节串成代码，用 Number Search 或任何可验证小任务各实现 GRPO 与 Dr.GRPO 一版。两版差异只有两处：优势估计去掉 σ_R，聚合从 1/|y| 改 1/C。对照训练曲线看长度漂移与退化组的行为。

想继续深挖的，按 ReST → ReST-EM → DeepSeekMath → DrGRPO 的顺序读官方 readings。reward hacking 主题再配 Anthropic 两篇一线报告交叉印证。

本讲要点

RL 三件套在 LLM 语境里：策略是模型、动作是 token 或工具调用、奖励是轨迹末尾的稀疏验证信号；GRPO 用组内相对优势省掉 critic，RLVR 把奖励交还给规则。
奖励是代理指标，凡有 KPI 处皆有刷法 ——reward hacking 不是 bug 而是古德哈特定律的必然，靠 KL 惩罚、奖励更新与人在环压住。
Deep Research = 研究计划 + 多步检索 + 综合引用的长报告，评测从单轮问答走向 BrowseComp / GAIA 这类任务级基准，引用质量是独立指标。
SFT 的三错配（任务 / 数据 / 暴露）在 Number Search 上逐一显形；ReST 证明保留高奖励轨迹做 SFT 已是 RL 第一步，REINFORCE 则是带掩码与再加权的 SFT：策略自己生成的 token 才进损失。
baseline 只降方差不偏期望（bandit 实验 SD 0.325→0.108）；GRPO 用组均值免 critic，Dr.GRPO 再修 σ_R 放大与长度偏置两处系统性偏差 —— 奖励与聚合的每个细节都在改写演化方向。
Deep Research 三线：评测没有唯一 gold，靠四轴框架 + 专家 rubric + FACT 引用核查站住脚（相似度指标与专家偏好一致性只比随机略好）；训练走 mid-training→SFT→RL 全链路，5% SFT 数据即可热启动 RL；检索要感知当前步推理（AgentIR 把 48.7 拉到 66.3）。

概念回顾

翻卡自测 · 「再看看」的卡进入复习队列

别只读一遍——先看正面的问题，心里作答，再翻面对照。主动提取比重复阅读记得牢。

官方指定读物

本讲资料库

查看全部 31 条 →

AgentIR: Reasoning-Aware Retrieval for Deep Research Agents ↗论文2603.04384前沿官方指定
官方指定（5b reading）。2026（COLM 2026）。检索器输入=当前推理+query，并用 DR-Synth 从 agent 自生的信息需求合成中间步训练标签；四模型消融 48.7→66.3，「当前步推理是最强信号」。中间步监督标签全自动合成，无需人工标注每一步该查什么，端到端可复现。客座讲师 Asai 组工作，L10 第四节主线。
为什么读
检索的时机与内容由推理驱动：reasoning-aware 检索的定盘星。
Jimmy Lin 组（University of Waterloo）等，Akari Asai 合著 · COLM 2026 · 2026
支撑：让检索器看见推理：从 query-only 到 AgentIR
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents ↗论文2504.12516前沿官方指定
官方指定（5b reading）。2025。1,266 道需「锲而不舍、变着法子挖网」的短答案题：答案短好评分，专考浏览 Agent 的持久性与策略切换，配反作弊措施。题目刻意选搜索引擎首屏查不到的，逼出多跳跳转与深翻页；普通 QA 模型得分个位数，差距一目了然。DR 评测从 QA 走向任务级的标志。
为什么读
把「会找」从「会答」里拆出来单独考试：浏览持久性第一次有了标尺。
Jason Wei、Zhiqing Sun、Spencer Papay 等（OpenAI） · arXiv · 2025
支撑：当答案不唯一：给深度研究打分的四条轴
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models ↗论文2402.03300进阶官方指定
官方指定（5a reading）。7B 模型 MATH 51.7%，同时提出 GRPO：组内均值当基线、丢掉 critic。同一 prompt 采一组回答，组内归一化的相对优势直接当优势函数，PPO 的 value 网络整个消失，训练开销大幅下降。L9 p.36-42 逐公式推导的对象（§4.1.1），开源 RL 默认算法的出生证明。
为什么读
把 advantage 从「学出来」变成「算出来」：RL 训练成本骤降的分水岭。
Zhihong Shao、Peiyi Wang、Qihao Zhu 等（DeepSeek） · arXiv · 2024
支撑：GRPO 家谱：奖励为什么必须「相对」把 Dr.GRPO 写成掩码再加权：一份作业路线的展开
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents ↗论文2506.11763前沿官方指定
官方指定（5b reading）。2025（ICLR 2026）。100 道博士级研究任务覆盖 22 领域，配套报告质量维度评测+信息检索（子节点目标+证据支持/FACT）两套评审法。报告侧按覆盖率、结构、洞察等维度 LLM 评审，检索侧用子目标分解与事实支持度量化「查得全不全、用得对不对」。把「报告写得好不好」操作化的第一份系统基准。
为什么读
DR 评测从找答案到交报告的分水岭：质量第一次被拆成可测维度。
Mingxuan Du、Benfeng Xu、Chiwei Zhu 等（微软） · ICLR 2026 · 2025
支撑：当答案不唯一：给深度研究打分的四条轴
Understanding R1-Zero-Like Training: A Critical Perspective（Dr. GRPO） ↗论文2503.20783前沿官方指定
官方指定（5a reading）。2025（v2 2025-10）。指出 GRPO 除以 std 丢失题目难度信息、除以 token 数奖励「又长又错」的回答，提出 Dr.GRPO（只减均值、除以常数上限），7B 达 AIME 2024 43.3%。两处归一化各删一项，难度偏置与长度偏置同时消失，简单题上的过度思考显著收敛。改动只有几行代码，却是「读公式要读分母」的最佳教案。伴读实践篇的作业路线会实现。
为什么读
一处分母一个世界观：归一化里藏着的偏置足以左右模型行为。
Zichen Liu、Changyu Chen、Wenjun Li 等 · arXiv · 2025
支撑：GRPO 家谱：奖励为什么必须「相对」把 Dr.GRPO 写成掩码再加权：一份作业路线的展开
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research ↗论文2511.19399前沿官方指定
官方指定（5b reading）。2025（ICML 2026 Oral）。RLER（演进式评分标准）训练：超 Tongyi DR 平均 15.6%、比 OpenAI DR 高 0.7%、每查询便宜 1000 倍；5% SFT 数据即可热启动 RL、演进 rubric 带来 50→58 的持续增益。评分标准逐轮加严并按区分度过滤，「标准本身在学习」，给开放式任务的 RLVR 递了可复用的钥匙。
为什么读
奖励工程的下一个台阶：连评分标准都可以进训练循环。
Allen AI 等 · ICML 2026 · 2025
支撑：先模仿还是先试错：Deep Research 训练配方逐段对账一次搜索与深度研究之间隔着什么：从 GHC 5507 到带引用的综合

自测13 题 · 约 23 分钟即时反馈 · 答错的题自动进入复习队列先自己作答，再看解析——检索练习的效果来自「先提取、后对照」。数字键 1-4 可快选，Enter 提交。

01单选基础L9 p.3-7 ↗

用「带实习生」比喻，RL 对应哪种培养方式？

1照着优秀范例抄一遍2只看交付打绩效3先上两周培训课再上岗4直接发毕业证

02单选进阶L9 p.42 ↗

GRPO 相对 PPO 的核心节省是？

1不需要奖励信号2省掉与策略同量级的 critic（组内互为基线）3不需要参考模型4不需要采样多个回答

03多选进阶arXiv:2210.10760 ↗

缓解 reward hacking 的工程手段有哪些？（多选）

1KL 惩罚压住策略偏移2奖励模型 / 验收标准定期用新数据更新3把同一条 KPI 指标数字加倍加严4生成器与验证器分离，加人在环抽检

多选题：选出所有你认为正确的选项后提交。

04判断基础arXiv:2210.10760 ↗

「把手机绑在狗身上刷步数」是古德哈特定律的一个实例。

正确错误

05排序基础arXiv:2203.02155 ↗

把 RLHF 三部曲按执行顺序排列：

按你认为的正确顺序依次点击各项；点错可「重排」。选项展示顺序已打乱。

06简答进阶L10 p.47-49 ↗

简报 Agent 为凑「每条结论必须带来源」的验收，把同一个 URL 挂了三处——验收应该怎么改？

07单选挑战L9 p.37-38 ↗

同一任务（藏数 3）采样 4 条轨迹，奖励为 [1, 0, 0, 0]（L9 p.38）。按 GRPO 的定义——先减组均值、再除以组内标准差 σ_R——这 4 条轨迹的优势 Â 依次是多少？

1+0.75、−0.25、−0.25、−0.252+1.73、−0.58、−0.58、−0.583+1、0、0、04+0.25、−0.25、−0.25、−0.25

08多选挑战L9 p.21-22 ↗

一条猜中（R(τ)=1）的 Number Search 轨迹被串成 chat 模板（L9 p.22）：<|im_start|>user 任务指令 → assistant guess(8) → tool higher → assistant guess(12) → tool lower → assistant guess(10) → tool higher → assistant guess(11) → end-of-turn。在 L_PG = −R(τ)·Σ_{t:m_t=1} log πθ(u_t|u_<t) 中，下列哪些 token 属于 m_t=1（参与求和、获得梯度）？

1user 段的任务指令 token2assistant 段的 guess(8)/guess(12)/guess(10)/guess(11) token3tool 段的环境反馈 higher/lower token4assistant 段末尾的 end-of-turn token

多选题：选出所有你认为正确的选项后提交。

09单选挑战L9 p.39-40 ↗

GRPO 按响应长度倒数 1/|y_j| 聚合（L9 p.39）。同一任务组里两条失败轨迹：甲答案 200 token、乙答案 50 token，都 R=0。为什么训练久了策略会慢慢偏向「更长的错误答案」？

1逐 token 惩罚为 |Â_j|/|y_j|，长答案被 200 除后每 token 惩罚更轻，策略漂向更长答案2长答案让组内标准差 σ_R 变小，梯度被系统性放大3训练器隐式给长答案更高的 R(τ)41/|y_j| 使长答案的总惩罚更大，策略会主动变短

10排序进阶L9 p.23 ↗

把 L9 p.23 的 RL 训练循环四步按真实执行顺序排列（每轮从采栽数据开始、以参数更新结束）：

按你认为的正确顺序依次点击各项；点错可「重排」。选项展示顺序已打乱。

11配对挑战L9 p.28-33 ↗

L9 p.28 给出 baseline 的四种取法，各有代表场景/方法。请把取法与对应项配对：

常数 baseline

奖励滑动均值

训练值模型 V_φ(h)

组内均值

12单选挑战L10 p.35-38 ↗

某团队宣称其 Deep Research 系统 SOTA：在 100 道开放研究问题上用 ROUGE-L 计算与 gold 报告的相似度再排序，击败所有对手。参考 L10 的证据（以专家偏好为准的 preference accuracy：ROUGE 58%、BERTScore 57%、BLEURT 62%，随机 50%；且存在把 unproven 写成 proven 的相似但错答案）。评审时最该挑战的一点是什么？

1ROUGE 类相似度与专家偏好的一致性只比随机（50%）略好，且「相似但错」的答案会拿高分——相似度排序本身不成立2100 道题的样本量太小，应该扩到 1000 道3应该改用 BLEURT 就没问题了4应该拿 BrowseComp 上 Deep Research* 的 51.5% 来对比

13简答挑战L10 p.44 ↗p.47-49 ↗p.84-85 ↗

你要训练一个医学文献 Deep Research agent（产出开放报告+强制引用），预算只够一个奖励信号+一套评测。参考 L10 的证据：GPT-4o 引用幻觉率 78–90%（p.6-7）、生成的 rubric 项约 15% 空洞/有错/不清（p.44）、Expert–LM 判定一致性 79% vs Expert–expert 80%（p.42）。写出你的奖励设计与评测方案要点，并说明各自挡住哪类翻车。

题目与解析 AI 辅助生成 · 人工审校。答错的题会按间隔重复节奏在复习页重现。

AI 生成 · 仅供参考，与课程方无关。选中正文里的任何一段也能直接问我（解释一下 / 举个例子）。

英文文字稿（YouTube 自动字幕机转，未校对）

Okay, so let’s go ahead and get started. So today is going to be about an intro to reinforcement learning which will build on the SFT for agents uh that we had last week. This is going to be the first of three lectures that we’ll have on reinforcement learning. And so in

this first one, we’re going to be talking about basic poly gradient methods, which are ways where agents interact with the environment and get feedback and then update their weights to make things that were successful that got high reward more likely and things

that got low reward less likely. We’ll then have a second lecture which will actually be next week on uh more advanced RL algorithms talking about off policy methods about um some of the things that you need to do to stabilize RL and get it to work. And then our

third lecture will be on RL systems and going into some of the practical frameworks and you know things that you need to think about and pay attention to in hyperparameters when you’re training these agents. And all of these should set you up uh pretty well for assignment

three, which is going to be focused on training models with reinforcement learning in two settings. So the first setting is going to be a simple kind of sandboxed setting, which we’ll actually use as a running example in this lecture. It’s going to be called number

search. And the second one is going to be like a simple kind of uh crafting environment that’s a little bit like Minecraft. So for these, they’re definitely going to require more compute than your previous assignments have. Um so you’ll be using the compute credits

more heavily, but uh you should have the foundations to get started on those assignments after these first two lectures. And um yeah, and we’re looking forward to uh to seeing what you all do for those. We’ll also have a guest lecture on Thursday from um Akari Asai

who’s a faculty here at CMU uh who’s an expert on deep research and she’ll be talking about that as another domain for agents uh that complements the the ones that we’ve talked about so far. Cool. Any uh any quick logistic questions before we get started?

Okay. Yeah. So uh today we’ll have a bridge from uh supervised fine-tuning to policy gradient methods which are kind of the most uh the most basic but also one of the most elegant forms of reinforcement learning. So the running example that we’ll use is

this task that we’ll call number search which is basically like training an agent to guess a number based on feedback about whether you’re too high or too low. So this is a pretty simple task but you know easy to understand and easy to come up with a reasonable policy

for it. So the setup is there’s this number which is part of the environment for this task but isn’t known to the agent and it’s an integer between 1 and 16. The agent is going to get four attempts to guess the number. And after it makes a guess, the if it got it

correct, the environment will say correct. But otherwise, the environment will say, uh, you should go higher or you should go lower. And then the agent can make another guess up to four guesses in total. If at any point the agent gets it correct, the task ends and

it gets a reward of one. But if you reach the end of your four guesses and you haven’t gotten it correct, then it’s a reward of zero. Okay. So, a lot of the trajectories that we’ll walk through are from an example of this where the hidden number is 11.

So, the agent is going to try to guess that. And here’s four possible attempts at this where in the first one the agent says 8, gets feedback, you should go higher. It says 12, gets feedback, you should go lower. uh 10 higher and then

11 so it gets it correct on the last guess. In another attempt we might start with a lower number. So this is actually what’s the optimal policy here? What’s the best starting number? The middle, right? So it’s kind of like

20 questions. You want to like split the space, right? So the best first guess is eight. But uh and actually how many of you folks have played Wordle? you know the guessing game from the New York New York Times. It’s a fun game. It’s kind of in a similar space and

there’s like an optimal first guess that you can make which reduces the the the space in an information theoretic way. Similar concept here but simpler. So you could start off with a you know suboptimal first guess of four but you could still succeed in the task right by

taking this feedback into account going higher then going higher then going you know lower and eventually getting it correct but you could also go wrong right so we could start with a very bad guess of 16 and um unless we got that right the

feedback’s going to be lower and in this case we might time out so we don’t get it within four guesses and similar Similarly, we could have another pretty bad policy of starting with one and then going two and three and four and um and timing out and failing at the task.

Everybody clear on the the setup? Okay, great. So, we’ll also have rewards for trajectories. And a reward is just a scalar value which tells you how good or bad that trajectory is. So in agentive

settings, this might correspond to like did the code that you write pass all of the unit tests in like Swedbench for example and you get a reward of one if it passes all the tests a reward of zero if it doesn’t pass all the tests or you could have you know like a value that’s

not zero or one that’s not a binary reward too like what’s the fraction of tests that you passed but in a lot of the domains that people are training on these days um in what’s called RLVR like reinforcement learning with verified rewards. The rewards will be either zero

or one. Um but the methods that we’ll be talking about today are more general. Um they also apply to scaler valued rewards. So just a quick reminder of how SFT works. So in supervised fine-tuning, you have a demonstration of a task and

you’re training the agent to copy that demonstration by maximizing the probability of the demonstrated actions. Right? So if we break down one trajectory into these steps, we have a prompt that’s given to the agent. Um we have a guess um feedback from the

environment higher. We have another guess more feedback from the uh environment and so on. And our task is going to be given these demonstrated actions which we’ll write a star. We want to maximize the probability that the agent assigns to those actions

conditioned on all the past history up to that point. Right? And so this will be just you know the supervised fine-tuning loss is maximizing the log of the probability of that sequence which is the sum of the log of the probabilities of the individual actions

right and there’s a lot of ways we could get these target actions that we’re trying to imitate. So they could be from a person who gives a demonstration from the task. They could be from like a stronger model that we’re trying to copy to distill from today. They could also

be from uh trajectories from the agent itself that we’re trying to reinforce in the model. But this should hopefully be a review from you know your previous classes on LLM training and and from the SFT lecture.

Okay. But SFT has some problems which reinforcement learning is going to address. So there’s roughly like three of these that we’ll focus on. The first one of these is what we might call task mismatch. So supervised fine-tuning maximizes the probability of

demonstrated actions at each point in time. But what we’re really interested in is maximizing the probability that the agent carries out the task successfully. And these are similar to each other, but they’re not quite the same. And there’s

a couple reasons for this. So the first reason is that there are multiple ways to carry out the task successfully, right? So, uh, starting with eight is kind of the best thing to do, um, as a first guess, but, um, it would also be reasonable to start with, you know, four

as long as you’re able to, uh, take the feedback into account. And for this, you know, here after this first step of feedback, we have various reasonable actions that we could take as well. and a demonstration, whether that’s, you know, from a person or another model,

only shows one way of solving the task, but we want the agent to be able to um potentially use other ways as well. And a second problem is that some actions that you can take are worse than others, but to SFT, they all look the same. So it would be very bad for us to

in response to the feedback you should go higher it would be very bad for us to like output a number below eight right but SFT doesn’t make a distinction between the actions that weren’t demonstrated and so it would get the same loss if um

you know the agent had highest probability on um a lower alternative action rather than a higher alternative action that wasn’t the one that was demonstrated. So that’s one problem of the mismatch between this SFT loss and the objective

that we’re actually interested in of maximizing you know expected reward or the probability of success on the task. The second problem is data mismatch. So we’d like to be able to learn from um suboptimal data too. Even if we assume that our demonstrations are the best way

to achieve a task, there’s, you know, if we had access to other demonstrations that were less optimal or maybe even weren’t successful, those could provide useful signal for our models too. So like uh say that we know that this is a bad trajectory like it times out. We

wouldn’t want to just do SFT on this, right? We wouldn’t want to maximize the probability of those actions, but we might want to uh have some learning method that could um have the model not take trajectories like this. And reinforcement learning is going to be

one way of doing that. The third problem is what’s known as exposure bias. So in SFT, you’re training a model to make predictions of the next action given a fixed sequence of past actions, right? And that fixed sequence of actions comes from

somewhere. But then at evaluation time, the previous actions are going to be generated by the model itself, right? Because you have to produce actions one at a time. You have the actions you produced in the past. It’s this auto

reggressive process of interacting with the environment. So if the agent um produces an action that is very out of distribution from the histories that it was trained on, it might just not have the the response

that it produces might just not be calibrated because it never saw data like that. So say that we only trained on demonstrations where uh the next integer was, you know, did match the feedback that we got. So it was always higher if the feedback was higher. Now

if at generation time the model makes a mistake and samples a lower number, it might just totally throw it off because it didn’t see data like that in training time if we only trained on these optimal policies. Does that make sense?

So one way of viewing reinforcement learning is it’s a way of letting the model generate its own training data and also learn how to recover from the mistakes that it makes so that it’s prepared to handle its own mistakes when you generate from it. And um this

property of having the model generate its own training data is called being on policy. It’s generally a good thing to have um because it prepares the model to you know deal with its own mistakes and the supervision that you get is a good fit for what the model’s able to

currently predict. Um but it can have some efficiency trade-offs and in Graham’s lecture we’ll talk a little bit we’ll talk a lot actually about um uh methods that relax these assumptions that go more off policy. Cool. So these are three problems with

um supervised fine-tuning for agents and for sequential prediction generally. Any questions about them? All right. So, reinforcement learning the general idea is instead of having like a uh trajectory a fixed trajectory which you learn to imitate by you know

maximizing the probability of the actions. We’re going to have the agent or you know the model um generate its own actions and then we’ll evaluate the reward of those actions in the trajectory and um then we’ll adjust the parameters of the agent so that um it

makes high reward trajectories more likely and lowreward trajectories less likely. So formally and we’ll see you know the formulation of this in a little bit we’re going to be training the agent to maximize its expected reward you know expectation in probability under the

agent and under the environment. So for um just schematically if we have you know a task like this number search game with the underlying goal in the environment B to guess number 11 we would have the agent generate multiple trajectories just by um producing likely

actions at each step from the agent. You know just in the way that we would typically evaluate the agent in the environment for each of those trajectories. uh we’ll get a reward um by uh by executing these actions in the

environment and then we’ll use that reward as feedback for the reinforcement learning method which is going to change the parameters of the agent uh to make these highreward trajectories more likely. So this is going to address the three

problems that we uh laid out with SFT. So task mismatch is addressed because we’re now training the agent directly to maximize expected reward. I haven’t shown you how that’s done yet. A lot of the first part of the lecture is going to be about how we do that. We’ll

formulate expected reward. We’ll take derivatives of it. We’ll optimize it directly. It will also fix data mismatch because as we’ll see especially towards the middle and end of the lecture, the agent will now be able to learn from all

trajectories. So it will get some signal from these bad trajectories and will avoid taking actions that lead to trajectories like that in the future. And it will also help deal with exposure bias because the agent is now going to be generating these trajectories that

are used in training. It’s going to be on policy. So if it makes mistakes and sees that those mistakes lead to a bad reward, it will get penalized for that and then learn to avoid those in the future. So it’s sort of being prepared to deal with its own errors.

Yeah. And um yeah, like a lot of the issues that people had with like language models up to maybe like three or four years ago where they were um being like uh they had kind of repetition biases. They would get stuck in loops. A lot of those issues got

fixed once people started doing a final post-training step with reinforcement learning where the models were now able to recover from their own mistakes. Okay. So we’ll talk about how we you know formulate interactions with the environment as trajectories which will

then let us set up the objectives that we optimize. Okay. So um we won’t spend too much time on this because it’s should just be a recap from the first part of the course. But we have an agent which is going to be uh represented um as a policy which

we’ll term um pi theta. Theta are the parameters of the language model that you know that drives the agent and the agent’s going to be producing actions um by sampling from the language model and you know then applying those um uh those samples in the harness that

it’s operating in that will produce actions um which you then take in the environment and the environment gives you back observations and also rewards. A lot of the settings that we’ll look at only give you a reward at the end of the task, like when you actually execute the

test cases. Um, but pretty much everything that we talk about will also apply to settings that have um intermediate rewards where for individual actions you take, you can also get some reward about um the value of that action. But just some examples

of how you can apply this general framework for text generation. Um, the task instance might be the prompt that you’re trying to produce a completion to. The observation might be the tokens you have so far. The action might be the next token. And the reward might be the

score that you get from a judge model. For example, if you’re doing um reinforcement learning from human feedback for a web agent, like the setting we talked about a week or two ago, the task instance might be the task and the

website. And here there’s a difference between the state of the environment, sort of everything that determines the environment and what the agent gets to observe. There might be some hidden state like what’s the content of the backing databases, but the agent only

gets to observe the page that it’s currently on. And it can take actions like clicking, typing or scrolling. And it might get reward like intermediate reward like what are the number of subtasks from the the task description that it has completed so far. And for

number search we’ve uh seen the task instance is the number that the agent is trying to guess which is going to be hidden from the agent. So in number search every task looks the same to the agent at the start. Um the observation would be high or lower or correct.

Action would be the next guess. And the reward is given if the agent gets it correct that’s a one or um if it times out then it gets a zero. So given these um these observations and actions and tasks we can start to formulate some notation which will let

us derive the um reward terms that we’re expected reward terms that we’re trying to maximize. So a trajectory is a sequence of interled actions and observations and we’ll represent um the sequence of observations and actions up to a given time step as the history h.

So the history ht is going to have all of the actions and observations up to that point in time. And our agent policy pi theta remember theta are the parameters of the language model inside the agent. that’s going to give us a distribution over the next actions that

I would take um g at a particular history. Right? So you take the history, you plug it into your agent framework, you get a distribution over next tokens which correspond to actions in the environment. Any questions about that? Okay. So we also have rewards and in the

most general setting you could be getting rewards along the way after each um action or observation that you take. Those formulations whether it’s after actions or observations can be equivalent to each other. Um in the number search setting, we only get a

reward after you um either time out or get the feedback that it was correct. So in the number search setting, your rewards will be binary and they’ll just come at the end of the trajectory. And we’ll write the reward of the trajectory as the sum over the rewards

at all intermediate time steps. So toao is going to represent trajectory which is you know the sequence of these observations actions and rewards. Okay. So that’s the notation and given this we can also you know have probabilities over trajectories and the

probability of a trajectory is going to depend on two things. So the first thing is the probabilities assigned by the agent. So we can decompose this into a product of probabilities over time steps where um h sorry I dropped a p here but uh there should be a p before this term

which is the probability from the environment of getting this observation and this reward conditioned on this history the action taken by the agent at that time step and the task and that probability from the environment we’re going to multiply by the probability

ility from the agent policy of producing this next particular action. So this is just the chain rule um decomposing this probability of producing this entire trajectory into the product of its component parts. And a key thing in RL is we don’t

actually know we we don’t have to know anything about the environment. We don’t have to know, you know, what’s the probability of getting correct as our next observation or what’s the probability of getting a next observation as I’m interacting with

Amazon.com or something. The key thing about RL is we’re going to interact with the environment which will by taking actions which will be a way of sampling from this distribution over observations and rewards. And that’s actually going to be enough information for us to be

able to learn how to differentiate um an objective that depends on this with respect to our parameters and update the parameters. And that’s kind of magic in some sense, but it also makes reinforcement learning algorithms a bit slow because you kind of just have to

like try stuff and see what happens. Um, and that’s why reinforcement learning takes a lot of compute to run because you have to do these experiments um, experiments with the environment, see what happens, and use that as a somewhat indirect source of feedback about how

your reward depends on your parameters. Any questions about this? >> Yeah. Yeah. >> Yeah. So I’ve uh rewritten this a bit from so the question was in classical RL there’s a marov assumption which um says

that the transition to the next state depends just on the previous state that you’re in and the actions. So we’re actually working with um a palm dp here a partially observed marov decision process um where there’s a difference between the underlying state which you

can’t see and the observations that you can. So in a palm dp um your probability of the next observation has to depend on all of the previous observations and actions because you don’t have access to the state. Um but for a pom dp these formulations should be equivalent and

this one’s just a little bit easier to work with for our purposes. Good question. Okay. Yeah, sorry about that P missing. Okay, so our objective is going to be to maximize the expected reward. Um, and so what does this mean? So what we want to

do is we want to uh take samples from our agent like we want to have the agent generate trajectories through the environment. Um and like uh we’re typically going to be doing that by sampling um actions from the language model and then feeding

those actions through the agent framework, right? Um so we can write that um as sampling a trajectory from this distribution which is parameterized by the language model itself. And for each of those trajectories that we get by sampling from that distribution, it’s

a very complex distribution that depends on the agent, on the language model, and on the environment, we’re going to get a trajectory and we’re going to evaluate the reward of that trajectory that gives us a number. And we want to maximize our chance of getting a high reward here. So

that’s going to be maximizing this expectation. And we’ll just write that objective as J of theta because it’s a function of the parameters of our language model. So in our very simple setting um let’s uh you know we have this number search

we have say there’s only four trajectories which are actually producible by the language model. So it’s learned kind of a very sharp policy. Um say that it puts probability of 0.3 on this first trajectory 0.2 on the second and 0.25 25 on the third and

fourth. Uh generally blue is going to mean success, a reward of one and red is going to mean failure, a reward of zero. So the expected reward if these are the only four trajectories which the agent can produce is going to be 0.5. Right? So just you know taking a weighted sum

of these trajectory rewards weighted by the probability under the agent policy and the objective of training is going to be to optimize this. So we want to adjust the parameters theta to maximize this expected reward. So you can think of this as like this is kind of a bad

current agent because it has a pretty high probability 0.5 of producing a trajectory with zero reward. We want to shift some of that probability. We want to shift ideally all of that probability to predicting trajectories that have reward of one. So shifting the

probability mass from red to blue. This is obviously simplified because you know really the agent could produce an inf not infinite but like combinatorally exponentially large number of possible trajectories but we’re just kind of

simplifying the visualization a bit here. So one way that you can do this is a pretty simple modification to SFT that’s really well motivated. It’s called expert iteration. And um a particular

uh type of expert iteration is called reinforced self-training or rest for short. So the idea in rest is going to be that we will just generate data from our current um policy from our current parameters of the language model and then we will choose the trajectories

that are successful and use those as fine-tuning data and update the model on those. And so that’s where the reinforced term comes from. It’s going to sort of reinforce the correct behaviors that the agent is already producing. Um so it has two steps which

are going to be iterated. Um rest em is actually named like rest I think expectation maximization because it looks really similar to the em algorithm uh which some of you might have seen in an ML class. But the idea is we will alternate between these two steps. The

first step is you sample trajectories from the current policy. You get the rewards of them. The second step is you will um have the model make those uh trajectories very likely by just doing a bunch of SFT on those collected trajectories. Um and then you repeat you

will get more trajectories from the current policy and you add those trajectories in to the existing set that you have of good trajectories and then you’ll um train your model on all of those. So you have this growing data set which you are having the model um

iteratively get very good at producing. Um yeah so that’s kind of a high level description and uh the way this works mechanically is you will generate a bunch of trajectories. You will um evaluate the reward of each one. You’ll select um

using like a threshold and this is easy if your reward is binary. Um, if it’s binary, you just keep the trajectories that have a reward of one and throw out the ones that have a reward of zero. And then you combine all of these kept trajectories across your instances. And

those are the um those are the next iteration of your training set. And so then the objective is just the SFT objective. We want to maximize the probability of these actions in these cap trajectories. the one where the reward is for binary rewards one.

Does that make sense? Any questions about that? So, schematically, this is actually going to look quite a bit like the reinforcement learning methods that we will talk about. Um, this isn’t like technically referred to as reinforcement

learning. Um, but it really has a lot of similarities to it because you’re generating data from the model. um you’re updating on that data in a way that is determined by the reward of the trajectories. And um we’ll see how reinforcement learning does this in a

little bit more of a fine grained way later. But this is a very simple and pretty effective uh baseline that you can start with which reuses all the same machinery of um supervised fine-tuning and is a good fit if you have binarized rewards.

>> Yeah. Why is that self? >> So the difference from self distillation is just that you’re using the reward to filter. Self-distillation could also refer to um training on all of the trajectories produced by the model. >> Yeah. So self it’s called selfraining

because it looks a lot like um you know like that but it’s reinforced because you’re using the reward and so you’re reinforcing the good trajectories. I mean this is a more basic question but how can you guarantee trajectories that do have a positive reward?

>> What if like for example in the worst case you just have a bunch of bad trajectories? >> Yeah. So that’s a great question like how do you guarantee that some of your trajectories have good reward? Um if your model can’t produce trajectories

that have good reward then reinforcement learning and this method um are probably not a great fit. So there’s a few things you can do. one is you can um have some demonstrations of good trajectories which you train the model on. So you know like um the R1 paper um and a lot

of reasoning models um will have an initial what they call cold start stage where they will train the model on reasoning traces um before they start doing reinforcement learning. So that just makes it easier for RL or something like this to be able to produce

trajectories that have high reward. You can also like look at curriculum. So you can um select problems that do have enough trajectories which produce reward. Train on those first and then as the model gets better you’re able to tackle harder problems. You can also

have like intermediate rewards like if you have some knowledge of the problem and like know it’s useful to explore the code repo first. You could give the model reward for doing that and then hopefully like after it learns to do that it’ll kind of unlock solving the

whole task. But this is a really big kind of issue that RL algorithms have to grapple with and those are a few solutions to it. Yeah. Cool. Okay. So now um let’s uh actually get into the policy gradient methods.

So we’re interested to maximize this quantity uh J of theta the expectation of reward. And we can’t just use back propagation to do this uh for two reasons. The first is that we are sampling actions from our policy. And the second reason is that we are then

executing these actions in this like blackbox environment which is potentially which is almost always like non-ifferiable. Um and so this breaks our ability to do back propagation to you know directly

you know like use the chain rule like you saw in a machine learning class to get the derivative of um your reward with respect to the parameters. So um actions are discrete and they’re sampled. So the derivative with respect to um the parameters of a sample is zero

almost everywhere because you know the parameters change the probabilities but don’t um until you get to kind of a step point don’t change the action sampled and the environment is usually not differentiable. So the general solution to this is actually a pretty classic

method called reinforce um which was from Williams in 1992 and um it’s going to use this uh definition of the expected reward and then basically just some algebra tricks in order to um compute an estimate of this that uses um samples from the model and um the same

back propagation procedures that we’ve looked at before um to get par you know to get partial derivatives with respect to these parameters theta and we’ll show you how to do that on the next slide. So this is the objective that we want to optimize. It’s this expectation over

trajectories from our environment and our policy and the expectation of the reward of those trajectories. So this is a if you recall what an expectation is it’s a sum um over these things these trajectories uh of the term in here the reward but

weighted by the probability of that trajectory. So this is a sum over a really large space like all possible trajectories from the language model. We’re almost never going to be able to compute that exactly, but this is what it would come out to be

if we wanted to compute it. Taking the gradient of this. So we just, you know, take the partial derivative with respect to the parameters theta. Um, we can move that uh inside the sum and this uh reward term depends only on the trajectory. So once we know the

trajectory, we know what the reward is. it doesn’t depend on the parameters anymore. The parameters produce the trajectory. Um but then the reward is just a function of the trajectory. So um this reward term is going to be multiplied by this partial derivative of

this probability. But this is still difficult because this is a sum over like a potentially very large number of trajectories. So we’re going to do an algebra trick here. And this is called the score ratio trick. um we’re going to just uh multiply and

divide by this term p of theta. So we have that over here and we also have that in the denominator. Those cancel. So this is the same as that. And then we can use this trick which is um the partial derivative of log p with respect to theta. We can just use the chain rule

and it’s get it’s the partial derivative of um p / p. So we can use that identity here and we get this term. And this is going to be kind of magical because we now have this which is going to be an expectation over trajectories um times a reward term times this term

And um this term is going to be this is the same computation that we do in SFT. Right? In SFT, we’re also taking a partial derivative of the log of a bunch of unit tokens um with respect to the parameters of the language model. And um we’re going to be able to uh estimate

this quantity by sampling trajectories and then computing this term here. And this will give us an estimate of the um of the gradient of this uh expected reward. Any questions about that? >> So this is really general doesn’t make any assumptions about um the language

model or about the environment. >> Are you going the series of states and actions but the actions are generated by large are also sequences of >> yeah

normal way to reconcile that >> yeah we’ll we’ll break that down in a couple slides but yeah it’s a good question yeah so um we’ll just copy that term here and then we’ll um break this into the component pieces so um these

trajectories are going to be sampled by having the language model and the agent on top of it interact with the environment um and that’s you know the same procedure that you would use when you’re just running this agent. That gives you

a trajectory. Um and we uh uh this should be um a sum over this should be an average over trajectories because we can estimate an expectation by drawing some number of samples from this distribution and then averaging the quantity inside the expectation. So if

we average over a bunch of trajectories drawn from this distribution this quantity here um then we will get an estimate of this gradient. So um what are these uh terms inside here? Well it’s um since this is a trajectory we break this up into steps.

So it’ll be a sum over time steps of um the gradient that’s due to that particular action. And we’ll see how to compute that in a slide. So yeah, so the three components here are the trajectories. We get those by sampling. Um the task level evaluation

which we get from the environment. And then this gradient here which is going to be produced using the same way that we get gradients in in supervised fine-tuning. But the key thing is these actions here are going to be the ones sampled from the agent rather than ones

that are demonstrated by uh you know some ground truth. Okay. So yeah, how do we get log probabilities of trajectories? Uh I’ll go a little bit quickly through this. The the key thing to keep in mind is that the probability of a trajectory

depends both on the probability from the language model plus agent but it also depends on the probability of getting the particular observations and rewards that you did from the environment. But the key thing is is that this probability doesn’t depend on the

parameters of the language model in any way. So if we differentiate this log probability of you know the uh trajectory with respect to the um parameters this entire term just goes away because it’s a constant with respect to the parameters. So we can

just ignore it and this um gradient uh for the trajectory is the sum of the gradients for the actions within the trajectory. So this is mostly just kind of like uh bookkeeping for rigor. Um but one thing that is important to think about is

these actions that are being produced by the agent um might correspond to multiple tokens from your language model, right? So like maybe you need to say uh you need to say you’re you’re making a complex tool call and that splits up into multiple tokens. Um, of

course, the log probabilities of those tokens are just going to be um summed up to produce the log probability of the entire action. And when you have this uh gradient of a trajectory, the gradient for a trajectory, you can just multiply by the

reward, which is a scalar value from the environment to get the contribution of this trajectory that you sampled to the expectation. And how does this uh like actually look if you are running this on a real example. So you would um have your

trajectory which is sampled from the agent. And um these blue tokens are the ones that are actually generated by the model. All the gray tokens are ones that are either like part of the prompt or part of the environment or added by the scaffold. But these are the ones that

are actually produced by generating from the language model. and those are going to be the ones which um contribute to the loss for policy gradient. So we’ll have all of these generated tokens. We’ll get um log probabilities for those um of that generated action

selected from the distribution of the language model over particular tokens at that point. And then we will just scale all of those um token log props by the reward that corresponds to this Does that make sense? And then if you take the gradient of this, the gradient

passes inside and gets applied to this log probability here. So that your gradient is going to be the reward times the gradient. So when I first looked at the code that implemented reinforce on like a pretty complex task, it was like using RL to optimize dialogue models

from a paper called deal or no deal. It was kind of like really surprising that the implementation is so simple like you just do basically an SFT gradient computation and then multiply that gradient by a reward or multiply the loss by a reward and take the gradient

of that. But it it works and the math works out. Yeah. So we’ll have a mask here which is going to be you know you mask only to select the tokens that you generate and then you scale all of those by the reward for the trajectory.

And that’s how you implement the reinforce policy gradient algorithm. And so the training loop is just you sample task instances you evaluate reward on them. um get the loss using that slide that we just showed you and then um uh do an update on all of the

instances in your batch of examples Later on we’ll uh when we cover gpo we’ll talk about how you could generate multiple possible trajectories for a given task example I and then use all of those in the update. But if you’re doing just like standard reinforce, you would

just be using one trajectory for each example and then um computing the update from that. So your um batch size would be the number of task instances that you’re training on. Um but when we get to gpo um you will also have multiple samples per task instance. Any questions

about this? Okay. Yeah. So um just very quickly with binary rewards this looks a lot like the rest um procedure that we talked about. Both of these are just um scaling log probabilities of actions by a reward.

The difference is is that this reinforce method is like fully online. You’re like sampling these trajectories and then updating on them directly. Whereas with REST, you build up a very large data set and then train on all of those and maybe do like, you know, a full epoch or

multiple epochs on all of those instances. But um the intuition behind them is basically the same. Okay, but this has a lot of issues too. And so it’s very rare for people to just run standard reinforce policy gradient out of the box.

So the biggest I think the the the way of kind of the strongest motivation for the next methods that we’ll talk about is thinking about how rewards should really be the update that you do from rewards should really be relative. So let’s look at these two different

trajectories. Um this one is successful and so all the actions are going to be upweighted by a reward of one if we use the update rules we’ve talked about. This one is unsuccessful. So we had a bad first guess, we eventually time out. And if we’re just

using basic reinforce or arrest, we are not going to do any update based on this example at all. Right? Because it’s scaled by the reward. The reward is zero. So but we actually want to be able to learn not to do this, right? Um so the intuition for the next part is going

to be about shifting our rewards so that um we still get a correct estimate of the gradient but um we’re able to learn from um uh rewards at all different um scales or shifts. Um and another way of thinking about this is that the update that you do should depend on how well

the policy uh would be expected to do on this example. So say that this is just like an easy example and um it and the model like 95% of the time gets a reward of one on it. If 5% of the time you make a really dumb mistake and get a reward of zero, you should update very heavily

against those um costly but rare mistakes. And um baselines and advantages that we’ll talk about next will be a way of doing that which will compare rewards to the reward that you would expect to get from the policy running on this example.

And the math for that looks like this. We’ll define an advantage which is going to be a function of the history up to a particular time step and an action. And that’s going to be um say that you take this action from you take this particular action and then you sample

from the language model for the rest of the trajectory that will tell you and you contrast that against if you just sample from the language model including that action and then continue on. This will tell you how good is this particular action um

relative to uh how good is the what’s the value of this particular action taking into account that the language model is going to take all the rest of the actions in the future. So maybe a concrete example will will help with this. Let’s say that we are

partway through this uh guessing game and we’ve guessed eight and gotten feedback higher, guessed 12, gotten feedback lower. So um we know that the numbers have to be between 8 and 12. So they can either be 9, 10 or 11. So we don’t know exactly what this uh value

function is. It depends on what our policy is. But we can think about um it’s probably something like 2/3 or one, right? So like if the policy is is uh does the best thing possible uh always you should be able to just always win from this situation. But if

it’s a more random policy within the set of things that are valid it might be more like 2/3. But we can think about the advantage functions of particular guesses relative to this baseline. So like a guess of 11 would be bad because

um it’s at one end of this range and if 11 is not correct then we have two candidates and one guess left so this advantage should be low whereas 10 would be good because whether or you know even if this is wrong then we have one guess left and we could go either higher or

lower. So a lot of reinforcement learning algorithms are derived from different ways of estimating these advantage and value th this advantage as a function of uh this individual Q value for actions and values and we’ll talk about a few of

those for the rest of this lecture. But a key idea is that if we don’t know um qpi or vpi, we can estimate them uh by we can estimate the expected reward of this particular action just by acting and we can estimate um this value of how well we would expect to do using

what we’ll call a baseline function. We’ll see some examples of baseline functions uh for the next few slides. So a baseline is just a way of um rescaling your rewards. So this is the reinforce estimator of the expected um return here. Um which

you know scales uh the reward by um the log probabilities of which uses the reward to rescale this gradient of the log probabilities. And um these advantage-based methods will generally have this form here of um subtracting off a baseline from this reward.

And these are different these are not numerically equivalent to each other but they are both different valid ways to estimate the policy gradient. We’ll uh have a slide next that shows that this is also a valid unbiased um estimator of this same quantity.

But for different like baselines that we can define um we’ll get different reinforcement learning algorithms. So we’ll have an example coming up which kind of motivates why subtracting a baseline uh can be a good thing to do. Um so and in that example the baseline’s

just going to be a constant. The baseline could also be a running average of all the rewards that you’ve gotten so far from all the examples that you’ve seen. It could also be a trained like neural model and this is the case in actor critic methods or a method called

PO which we may cover in the next lecture. Um but you need a separate model from your policy for that that can be expensive. And in the final part of this lecture we’ll actually not have a separate model but we will just estimate this expected reward under the policy by

sampling from the policy itself. Um, and that’ll be the gpo algorithm. We’ll see a little bit more motivation for baselines in the next few slides, too. [snorts] So, just a quick uh note about um about

uh baselines and bias. both of these terms are estimates of the underlying expected reward. And um an estimate uh could have a bias. Estimates have bias and variance, right? A bias is like um how much is the mean of this estimate potentially shifted

from the true value. And the variance is like how much spread is there in the values that I’m uh drawing for that estimate. So uh you might be worried about bias um in your estimator. And there’s this uh this lema that shows that subtracting a baseline which

depends only on the history up to this point which all of these baselines that we’ve defined here do um does not bias the gradient. So the way to see this is this is our advantage our baseline based estimate of the expected um return and we can uh decompose this the expectation

of this into these terms here. So um using just the associative property putting this here um that’s the definition of uh the uh partial derivative that’s the that’s the um the gradient for J minus this expectation here of B * this and this is going to be

um equal to this if this um expectation of this term here is zero And we can see that by just um taking this term here and uh writing out the expectation like this. Um it’s the expectation over um actions uh sampled from this policy. Um and bringing out

this baseline term which does not depend on anything other than the history up to this point. We just get the sum over actions of the probability of those actions. And um that’s going to uh we can use this same uh log derivative trick and

rewrite that as this. And then this will just become one um and it’ll be um a bias term times the partial derivative of one which is going to be zero. So this tells you that you can uh subtract um a baseline which depends only on the history of um actions up to

that point, actions and observations up to that point without biasing your estimate of the policy gradient. But let’s look at an example of how these like baselines can actually reduce the noise in your estimate of the gradient. And this is going to actually

be really important because uh a huge amount of the work in reinforcement learning is uh reducing the variance in your estimates of the gradient um so that learning is stable um and robust to all the variation that you have from sampling from the policy and from the

environment. But to motivate this we’ll look at just a very simple example um where you just have two actions. Uh there’s no multiple time steps. There’s just a single decision that you make and two actions. You’re either guessing a or b. And we’ll have just a very simple

policy which only has one parameter theta. And we’ll get a probabilistic policy from this by applying the sigmoid function to theta. So sigmoid is this squashing function, right? Which um maps a real value from negative infinity to

infinity um into a value between 0 and one. and we’ll take that value as the probability of choosing a. So say that for our current policy we have this theta is negative 1.1 which will give us these uh very convenient probabilities of 0.25 for guessing a

which has reward of one and probability 0.75 of guessing B which has probability zero. Is this a good policy or a bad policy? Think about how the probabilities correspond to the rewards. This is bad, right? Because it puts

higher probability on the option that has no reward. So we want our policy gradient to update this parameter theta. Which way do we want theta to get updated? Yeah, we want it to become more positive, right? Because um if theta is

larger, we’ll be farther up on this curve, we’ll get a higher probability for guessing A, which is the option that has higher reward. So let’s compute our gradients uh without using a baseline and see if policy gradient tells us the right

thing. So um we can compute the um expectation we can compute um the uh values that we we would get um as estimates of the policy gradient from each of the actions that we could take under the policy that we could sample. So if we sample guess A, what’s our

estimate of the gradient? Well, we get this by multiplying the reward by the partial derivative, which um using properties of sigmoid, the partial derivative here is 75. So the gradient is just the product of those, it’s 75. We can do the same thing. Uh what’s our

estimate of the gradient if instead of sampling A, we sample B? That because the reward is zero actually gives us a zero gradient back. So this is kind of looks like the trajectories that we saw before where we didn’t get any gradient from ones with reward zero. Here we’re

not getting any gradient if we sample this action which has reward zero. But um if you take kind of the exact definition of the policy gradient which is the expectation over all the actions in this setting where there’s only two actions we can compute that exactly. we

can just uh multiply each of these gradients by the probability of sampling that action and that’ll be like we’ll see the exact number in a couple slides but I think it’s like8 so the actual policy gradient the expectation over all the actions is8

and um but we will only we’ll have kind of a very noisy estimate of that where 0.25% 25% of the time we will get a sample of 75 which is much higher than 0.18 and then 75% of the time we’ll get zero as an estimate for it. And what we really want is to have both of those

estimates that we could sample be much closer to the true value.18. So a baseline actually does this if um so the modification from the past thing is we’re just subtracting a baseline from our reward here and otherwise the computation is the same and it works out

to be that if we sample um guess A we now have an estimate for the gradient of 375 and if we sample B we have an estimate for the gradient of.125. So the expectation uh multiplied by the probability of these actions is still

going to be like.18. So it’s going to be the same as what we had before. But these individual samples are much closer to that true value because the baseline actually reduced the variance in the samples that we get. We’ll have a visualization of this in

another slide. But the key thing here is that that expectation does not change. It just kind of like makes the um the variance in the estimator more narrow, which is a good thing because you’re going to be drawing these estimates on real data, doing updates, and you don’t

want your gradient to bounce you around a bunch more than you need to. Okay. Yeah. So, here’s just a visualization of this. Um if we have a baseline of zero, here’s our gradient estimates. They have this expected value, which is8.

And as we drag the baseline, we can see that um we’re more we get these estimates for the gradient which um converge on this uh on this true value for the expectation of the gradient. And in all cases we are going to be

doing an update which makes the model more correct. We’re going to be doing an update which shifts theta to be more positive which is going to increase the probability of a put more reward or put more probability on the thing with reward one. But um we’re just going to

be doing that more quickly if we have lower variance here in this estimate. And here’s a graph that shows that. So um the purple line here is going to be doing draws of um the estimate for the gradient without using the baseline. So no B term just advantages sorry just

scales of one or zero. And then the green line is going to be estimates that subtract a baseline just a very simple baseline of 0.5 which was you know which looked like this on this plot. So it’s moved them a little bit closer to this true expectation.

So uh we’ll just like draw some samples and this simulates like doing steps of policy gradient on the exact same example over and over again. So here are sampling in this case we sampled guess A. So we got positive gradient um and we got a much larger

gradient if we weren’t using a baseline. So we’ll be taking a very large step and we sample A again. And this line here is going to be our running average over all of the samples of the gradient so far. So when we sample B, now we have um a sample of the gradient that is

below the true value. So we’re going to be closer, our running average is going to be closer to that true value. But you can see here we’re um we have less variance when we have the baseline. So our running average is already closer to that true value.

And this continues like this. We happen to sample a lot of guess B’s in a row. We sample guess A, it goes back up again. And it does converge eventually, but we’re generally closer to it when we’ve reduced the variance with this baseline.

Okay. Hopefully this wasn’t too in the weeds, but like um baselines have always been a little bit mysterious for me and I wanted to make these visualizations also to convince myself how they work. Um hopefully they’re helpful for you too. The intuitions for this very simple

uh two armed bandit setting do um transfer generally to these more complex um you know sequential decision-m settings too. Yeah. So um baselines are helpful because they can mark a trajectory as better or worse than expected. And uh if

you’ve chosen a good baseline, a lot of reinforcement learning is about choosing good ways of representing this advantage function. It can help to reduce gradient variance and and stabilize your learning. Um but you have some downsides too and a lot of choices about what you

use for the baseline. So with um and I’ll skip over this um other than just to say if you are working in a setting that gives you rewards at intermediate time steps rather than just at the end of the trajectory um you’ll you’ll replace your

reward scaling with um the reward to go like the sum over rewards um over future time steps and this will also help to reduce the variance. Um, we’ll have this slide here and I we can also talk about this offline if it’s something that’s relevant to you. Um, but uh it’s um how

you would handle not having all of your rewards come at the end of the task. Okay. So, with the last um bit of time that we have, we’ll cover a simplified version of um GRPO um which is uh group relative uh policy optimization I believe. Um, and it’s a way of um e of

getting this advantage function of estimating a baseline that uses rollouts from the model itself. Um, and this was um this was actually introduced pretty recently. So it was in one of the papers from deepseek that was training reasoning models um with a big

focus on efficiency. So a lot of past work on um applying RL to language models required a separate model to predict this baseline. Predicting baselines to you know reduce the variance is very important but was challenging and people were using

separate models for it. But DeepSeek was very interested in efficiency um and training models without using much compute and having a separate model takes a lot of compute. So they proposed this approach which gets rid of the separate model and just uses multiple

samples from the model itself to estimate the expected value the expected reward under the policy which is really kind of elegant. Um so the idea is that we can sample multiple times from the model for a given instance and then get the reward on each of those. Um and then

we can um center that reward by taking the mean of the reward across all of these instances in the group. We get a mean. We um basically use that mean as our baseline. And this makes sense, right? Because you know the baseline is your expected a good baseline is your

expected reward under the policy. And a really good estimate of that is just running the policy multiple times and getting the reward taking the mean of those. So we get our centered reward here. And in the original GRPO paper, they’re also standardizing this. Um so

they’re not just shifting it, they’re rescaling it. they compute the standard deviation over the rewards in this group of trajectories and then they divide by that standard deviation um to get the advantage functions. Now there’s um like actually it seems like dividing by the

reward is not necessary and a lot of implementations of uh that that call themselves GRPO actually get rid of this um division by the standard deviation. So it more closely matches the advantage term. Um there’s a paper we’ll talk about in a few slides called DRGP uh Dr.

gpo or done right gpo um which talks more about the motivations for that um but actually in a lot of implementations of gpo they won’t divide by the standard deviation they will just subtract the mean um but yeah the gpo update is very simple

looks a lot like policy gradient that we defined before but so you have like a task instance which is you know like one RL problem to solve you will generate multiple rollouts from it. So you will solve the same problem multiple times, resetting the environment, um getting a

full trajectory by sampling from your agent and um this will typically uh Aorva in his lecture will give some guidance on like choosing values for this group size I believe, but this is typically something like eight. Um and uh you’ll get the mean of rewards over

those rollouts and you’ll get your advantage estimates from that. Um, we’ll show two different ways of getting those advantage estimates in another slide, but that gives you a group of rollouts for one trajectory along with advantages for them. If you have room in your GPU

uh to fit in more examples, you can also get groups of the same size for additional instances. And you’ll be computing group means um only within each example. So this is important right because the value function should just be an

estimate of how um it should be an estimate of the reward on this particular example. Um so our advantages should only look at the reward within this example. But you will get an advantage for each trajectory by um scaling the reward for

that trajectory using the um the formula that shifts by the mean and maybe divides by the standard deviation of the rewards within that instance. So for policy gradient loss we were just taking a single sample for each um for each instance and then rescaling. But in

gpo we are going to be um having multiple instances in our batch and then multiple trajectories within the group for each instance and we’ll get an advantage for each of those. We’ll use that to rescale our log probabilities before differentiating.

Any questions about this? Yeah. >> Um is this unbiased? Yes, because um you your advantage function here um depends on nothing except the prompt and the model. So your baseline function here um which you’re subtracting from the reward is just um rollouts from the

language model. So this is unbiased. Okay, cool. Yeah. So, um, Dr. GRPO makes two changes to to GRPO. And I should note too, GRPO also, so what what I wrote here is not the loss for GRPO as defined in the paper. Um the loss in the paper has some

additional terms which I think Graham will talk more about in his lecture which also like clip the gradient so that you don’t take steps too far in one direction or another and also regularize the policy so that you don’t uh deviate too much from the policy that you

started with. Both of those are additional things that help stabilize training. We’re not going to talk about them today. We’re just talking about the advantage functions within GRPO. Um but uh yeah, this will help you uh get started on um assignment three when that

comes out. Okay, great. So, we’re good on time. Um so, uh DRGRPO um makes two changes to GRPO. Um and you will also do these in your assignment. Um and they seem generally helpful to do um because they help avoid some biases

within GRPO. And the two changes are just the first one we’ve talked about a little bit already. It doesn’t divide by a standard deviation. It just subtracts the mean of rewards within the group. And um yeah, the motivation for that is that uh uh if you’re dividing by the um

standard deviation, your advantages are no longer aware of the difficulty of the problem. um because you’re sort of like rescaling all ranges of difficulties to be standardized. Um so that can actually be bad. GRPO also um divided by the number of tokens generated which can

bias the model towards um uh towards uh long wrong answers and so DRGRPO just divides by a constant which is the max number of tokens that could be generated. Okay. So, same figure that we saw before. How does this actually look when

you’re implementing it? You have your generated tokens in blue. Uh, you compute an advantage function which is going to um take your reward for this trajectory and just subtract off the mean uh reward from all of your other trajectories.

And then you get a per token reward which just normalizes that by a constant C, which is typically the number of tokens that could have been generated. And um you do this for all of the rewards in your all the trajectories in your group which could have varied

rewards positive or negative. And the loss looks like this. Yeah. And um yeah, so just to note again the original paper also has like a a ratio which helps regularize the policy and clipping, but we won’t talk about those for now.

Okay, so we had a question before about like what if all of your rewards are zero? Um, that’s bad because if all of your trajectories fail, that means it’s just probably too hard for the current policy and you’re not going to get any signal from that even if you have a

baseline. Um, you also have a problem if all of your trajectories succeed because they’re too easy for the current policy. So like you know if this is binary rewards and um one is the best that we could possibly do um it’s bad um and generally it’s bad if we don’t have any

spread um within our examples because then we won’t have any signal from the advantage function as estimated by gRPO. So what could we do if every you know sample trajectory fails? We could choose different task instances. We could um inject some some knowledge into the

prompt that encourages the model to produce more guided solutions that might be correct. We could add demonstrations. Um there’s ways of doing off policy RL that put like uh correct demonstrations into the group. Um or more simply, you could just fine-tune the model first on

those correct demonstrations before running RL. Or you could have a more informative reward which um uh encourages the model to make correct intermediate decisions before producing an entire trajectory. That’s called reward shaping. So the like giving a

model reward for finding the correct file could be an example of that. Um we’ll skip this because we’ll uh potentially talk more about PO later on. But PO basically just um instead of using the trajectories to compute the advantage function has a separate model

which usually shares a backbone with the policy um to predict the the baseline. Okay. So very quickly just comparing the methods that we’ve seen so far. every method has the agent take actions in the environment and then we’ll just um

compute a loss uh using those actions produced by the agent. All of these look like selftraining in some way but they just differ in the way that they rewe those generated actions. So restm um we assume that we have binary rewards of zero or one and we just filter to the

trajectories which are correct and then do sftt on those. Um reinforce uses the reward of the trajectory to rewe the loss which corresponds to reweing the gradient which we spent a lot of time showing is correct. And then GRPO uh uses these advantage functions where the

reward is um is uh shifted and potentially also rescaled by the reward of other trajectories sampled by the model for the same example. And all of these methods help to address the three problems that we had at the very beginning of the lecture. So um

task mismatch uh should now be solved because we’re now maximizing reward directly. um expected reward directly data mismatch. We can now learn from trajectories that have reward zero as long as we’re doing baselines in the appropriate way. And also um the on

policy nature of these methods helps with exposure bias because the model’s like essentially driving at training time. So it should be learning to recover from its own errors. um it just comes at the expense of you know you have to actually just try stuff in the

environment which can be costly can introduce some noise into your gradients and we’ve covered some ways of trying to make that process more robust but in the next two lectures starting next week we’ll we’ll talk about some others too okay that’s all we

持续耕种，欢迎订阅 🌱

RSS全站 ·原创 ·策展 ·雷达；公众号「炼金术」同步更新。

本讲伴读笔记以 CMU 11-768 AI Agents (Fall 2026) 课程大纲为参考框架独立撰写。原课程由 Daniel Fried 与 Graham Neubig 讲授，原始课件与视频见cmu-agents.com；音频镜像与文字稿仅作大陆可达性便利，版权归课程方所有。本文观点与文字为作者原创。

本讲地图

双讲地图：从模仿到试错，从单搜到深研

SFT 到底错在哪：任务、数据、暴露三错配

REINFORCE 力学：带掩码与再加权的 SFT

从减一个数到除以一个组：baseline→GRPO→Dr.GRPO

奖励越硬越好吗：hacking、过长推理与 RLVR 边界

Deep Research 解剖：Plan→Search→Reflect→Answer

没有唯一 gold 怎么打分：四轴评测与引用核查

训练配方全景：mid-training→SFT→RL 全链路

检索器只看 query 就够了吗：AgentIR 与推理感知检索

实践视角：把 Dr.GRPO 写成掩码再加权

伴读笔记

导语

一、把 RL 老三样翻译成 LLM 语

二、从 RLHF 到 GRPO：奖励从哪来

三、Reward hacking：凡有 KPI 处皆有刷法

四、Deep Research Agent：规划、检索、综合的集大成

五、SFT 到底错在哪：任务、数据、暴露三错配

六、REINFORCE 力学：带掩码与再加权的 SFT

七、从减一个数到除以一个组：baseline、GRPO 与 Dr.GRPO

八、奖励越硬越好吗：hacking、过长推理与 RLVR 边界

九、Deep Research 解剖：Plan、Search、Reflect 到 Answer

十、没有唯一 gold 怎么打分：四轴评测与引用核查

十一、训练配方全景：mid-training、SFT 与 RL 全链路

十二、检索器只看 query 就够了吗：AgentIR 与推理感知检索

论文与材料导读

官方指定读物（5a/5b readings）

实践视角

本讲要点

延伸阅读

概念回顾

官方指定读物

本讲资料库