什么是 Agent：从一次工具调用开始

Okay. Yeah. Thank you so much for kind introduction. Um uh hi everyone. I’m Akali and I’m a new assistant professor at OTI. Um this is technically like my first lecture at CMU. Um today I’m going to talk about um DP search agent. So let’s start with this question. What a

deep search agent? Um first I want to explain why increasingly complex information needs code for like more agentic system beyond just a simple like for example like viable and da uh augmented generation pipeline and then I will give a brief overview of what DPS

agent do. So now let’s think about this simple question what is my office number at CMU. If you want to answer this question, we can simply like search uh Google and then we can identify uh faculty web page at the department

website and then you can find information uh indicating my O number. Um or we can set up a quite simple v augment generation pipeline where like uh language model called external v system once and then like uh read the viable document and here like language

model can just extract information and then answer this question. So in other words this entire process uh can be solved with one search uh one go to web page and then just like answer based on this specific web page but now many research questions don’t

look like this um suppose this example can AI agent synthesize uh scientific literature as well as human expert now to answer this question we have let me try [snorts] Okay. Um is it better? Okay. Um let me know

like if you cannot hear well. Um so to answer this question now we have to check many different website or um papers like you can identify open scatter papers or like some other papers to collect the information about this particularation.

But the challenge is that now there’s no single code document that perfectly answer this question to answer those complex information seeking questions. Now DP search agent um approach by repeatedly planning um using external search system and

defecting. Given this prompt the agent might first say like oh to answer this question I needed to collect this kind of information. For example, like I need to like find studies of vitro synthesis or like I need to check different studies um of like language model

against human expert for this task. Based on the initial plan, it issues a search code such as web search web search with this particular query and then we use external search system which returns a set of document. Then the agent defect on what it found so far. Do

those result actually answer the question or do we need a different set of evaluation? So it can uh based on those information it can revise a question and then search again and after this kind of iteration now like agent collect enough information and then

synthesize final answer with external citation. So those deep research agent pose a lot of new interesting research questions uh from like evaluation model training and also like how we can build better retail system. So today I’ll first talk about

evaluation of such DP search agent system uh so that like you can get a better sense of what those DPS task look like and then I will discuss how we can build DP such agent so that we can perform well on those complex task and I will briefly talk about BTO system or

like BTO foundation of DP agent because to build strong DPS agent we also needed to have better search systems. Um and also like feel free to interrupt me anytime if you have any questions. Um so first let’s start with benchmark and evation.

If you are familiar with um area called question answering NLP it is a big kind of longstanding uh research area. So question answering is essential like uh given a question we find like external evidence and then like answer those questions. But a lot of existing

benchmarks are quite simple compared to what people use those agentic system. For instance, this is a question from natural questions um benchmark from Google. So this benchmark uh collect existing um Google search queries from real users and then they annotate

answers based on Wikipedia at that u at that point like 2018 or 2020 Wikipedia year. So when you look at those questions, those questions are kind of simple um and like you can easily find Wikipedia pages um to answer those questions.

But when we think of more like real world DP use cases, this doesn’t like defect uh today’s user use cases anymore. For instance, this question can be answered based on the single web page. But as I said um a lot of those deep

research questions are not the questions uh cannot be answered based on a single web page. I’m not sure maybe it’s chat um or a lot of those existing benchmarks are based on English search barrier. So the questions are typical like pretty much uh domain uh

like general domain questions about like sport uh entertainment well like now people use VP such agent for much more complex expert domain or like longtail uh information need and also those answers are typically like showed um like typically just

entities uh while like a lot of questions require like long form open-ended generations. Lastly, we also don’t know like where those answers come from. When we pump language model, they can provide answer. But especially for information seeking

cases, we also want to know like why those model generate this answer or this answer is indeed correct. Um so we want to understand what kind of evidence support this final answer too. So now I would like to talk about how more recent DP search benchmarks address

those limitations one by one. Um the first um question I would like to ask is how can we build complex DPS agent benchmark that require more than single uh lookup. One of most widely used DP benchmarks nowadays is bloic benchmark from openai.

So this benchmark contains more than a thousand human written questions. Um interesting uh point of this benchmark is that the authors work kind of backward from a known answer or like the gold answer such as the person event or like artifact and then they collect a

set of clues and then they combine those multiple clues to formulate a quite like complex questions. They also conduct multiple um validation stages where like they first use language model uh plus additional search to make sure like those questions cannot be answered by

just uh prompting language model or like use five Google searches or they also ask for subset of questions they also ask humans to make sure like those questions cannot be answered under like 10 minutes for for instance. So this is an example question from this

benchmark. Um as you can see those questions don’t really like look like natural question anymore. It has like m multiple like conditions and the final answers the final answer needs to satisfy all of those uh five six conditions here.

Then the final answer is do a short here like plastic man entity answer. So we can easily verify if predicted model answers is correct or not based on um whether this answer is included in the final response. So this is a result from the original um

paper. As you can see back then the state of state of jet models in 2025 like GPT4 2024 uh GPT4 or like 01 model still struggle a lot in this benchmark even when they have access to external web search and even human expert only perform like

25% at on this benchmark and interesting finding from the paper is that if we specifically trained powerful language model for DP such task then now we can push the performance up to 50% % in the second section I will talk about how we can train such DP such agent um by kind

of mimicking how we construct those complex questions the original blog paper doesn’t provide much um information about like how many tokens those motors consumed or like how many search to codes they did but there are like a lot of photo work that show

that to perform well on this complex benchmark we have to increase the number of context token and also number of the maximum to call. For instance, this Tony DP such paper which I will introduce more in the second section shows some

ablation in terms of context length and also number of search codes. And as you can see, if we only set the maximum context length to 8,000 tokens, then the performance is almost 0%. But like if we increase it to like 128 then now um we can see that the performance uh gets

like like another like 40%. And also the paper show that uh we should like set maximum to codes up to like 128 128 codes to get the highest performance. So this kind of suggest that um those complex benchmark requires like hundreds of different search ticles and pretty

long collision. Um but this also like raises a practical problem. Now like we have to for each trajectory uh those agents needed to call like 100 student codes and each such to uh each code typically like use external web search API which can cost

and also which is could be like slow if we want to use it for like hundreds of times and also if we use those production search system then we cannot reproduce a exact result because Google update web search result like almost every day.

So this makes um the repeated evaluation harder to reproduce. So there’s another popular benchmark called blazeron plus uh which address this issue by fixing the document collection the agent can retrieve from. So instead of using the external real web API they collect um a

set of like useful and unhelpful documents and build a kind of simulated controlled corpus. What they did is that they start with original reasonable question and answer and they identify potential like helpful document using language model and then verify using

human annotators. After that they mix a set of negative document that can be that look helpful but indeed it’s not helping to answer the question and then like they combine those helpful and unhelpful documents to build a corpus with like 100k documents.

um with the reason like 830 questions. Um so this benchmark enable us to reproduce result and also we can build behavioral system because now we have fixed corpus which I can discuss in the last section. So those two benchmarks enable us to

test agent ability to use many different tools. But still like those benchmarks are based on general domain knows and like focus on like um kind of like entertainment person those um entities but now we use those GP such agent for more expert domain high-tech field. So

we want to make sure that those agent can perform well beyond like just answering sports or entertainment related questions. So now let me talk about some um additional benchmarks focusing on more expert domain.

One of those benchmarks um so here like I introduced fin search comp which is basically like bl but for uh finance domain. So what they did is that they collected a set of uh finance GP such questions using uh domain expert like people with like MBA or like people who

work in finance for many years and so they start with some like verified sources like financial databases and then they ask expert to form questions and then like annotate answer using standard tools uh cardate um numbers and so on and then like another set of

expert verified the final question and answers. So again like those question look quite complex and also now this question doesn’t look like ba comp. It requires a lot of domain knowledge in finance and also you have to like use external

calculator tools to compute exact final answer. There are like some other uh domains uh more like focused expert domain benchmark like like this search comp u met browser comp is a deep search benchmark and medical domain and also

research bench is a deep search task where you identify the exact paper that satisfy conditions several conditions. Um so those benchmarks the important finding is that to con to build high quality expert domain DP such benchmark we need expert to annotate and also

verify final answers. So so far I just discussed two um eight areas of DP such benchmark development but both benchmark kind of like focus on uh complex but still like short-term question answering setup which is not realistic in today’s like language model

use cases. So now let’s move to the form of the answer. Instead of just um asking like short answer questions, we want to make sure like those agent can answer more like open-ended questions where we have to generate longhome research uh

synthesis and evaluating longhome answers itself is quite challenging in in NLP in broader NLP community like historically like we annotate defense answers like one or like a couple of defense answers and then we evaluate the similarity ity

between model prediction and those like defense answers. But this has many limitations especially when we try to use those evidation for more like open-ended generation task. For instance, there could be many different ways to answer those

open-ended task and we cannot um animate all possible acceptable answers. Well, even if we can annotate all the possible answers, still there’s a challenge in terms of evaluating similarity. Um, for instance, in this case, the

second response has a lot of lexical overlap with the gold reference, but the main point is completely incorrect. And some of those similarity metrics may not capture those consequential error. On this point, there are like many interesting empirical studies in NLP.

For instance, this SEO paper focused on longhome question answering setup. So like pretty much deep search setup and they evaluate the reliability of those defense based metrics such as like dubix and but score um to see like if they can actually predict expert preference. So

here like if you have two responses uh they compute the score of those responses using different metrics and then they also like ask expert to pick their preferred best. So ideally the preferred best should have higher score in terms of like B score, bridge score

or like whatever score they use. But they found that all of those metrics have like accuracy from like 57 to like 62%ish which is like still like beyond uh random performance like 50% but it’s not really like u significantly better.

So the conclusion is that for more open-ended DP use cases, we cannot do just use those defense based metrics that have been widely used in NLP. So what we can do instead um what we did in our open scatter paper is that we introduced a new way to

evaluate such open-ended um responses and specifically we use Rubicaration schema. So instead of annotating all the possible defense answers, we ask expert to give us a set of evaluation criteria or like in other words Rubik’s and each

criterion include a set of like uh concrete uh description of like what good answer should include with important score. And when we have model prediction for each criterion, we use a language model to judge if this criteria criterion is

satisfied by the model response or not. And then we eventually uh calculate the weighted sum of those lubic score. We found that if we have high quality lubics then like expert have pretty high agreement uh with language model judgment in terms of if this lubric item

is satisfied by um the model response or not. So when we uh check the agreement between expert and expert for lubic judgment it was around 80% and like language model and expert is around like 79%. Meaning that if we have high quality lubics then like we can use uh

those like we can use language model to scale up this lubic based evalation. But the question is that how can we get high quality um lubics for open scatter what we found is that each dubic annotation can take almost one hour for like PhD expert. So if you want to use the schema

for like many many more questions for evaluation or training then it’s really hard to scale up. So some work take a closer look at how we can automate or like how we can generate dubics using language models so that we don’t need to annotate

everything using expert. This paper research QA try to oh like it’s a little bit designed but this paper research QA tried to generate um a set of lubics and question question lubics based on survey papers. So what they did is that they take the survey paper and then if they

pump language model to generate a question and also a set of rubik. Uh by doing that they can scale up annotation now they can have like 21,000 question and also 120k rubik. So if that works, it’s amazing. But in the paper, they found that there are a lot of caveats.

In research QA’s uh expert audit, they found that those language model generated parametric lubics have errors in like 15% of the cases. And those errors include house edited papers like language models say like oh your vest should site this paper a well like this

paper a doesn’t exist. um or like they can just like repeat question um or like they can give like really high level but big uh criteria which is not helpful to identify good response. So to build high quality rubik um that can determine a good responses then we need to um at

least like ground those model generated lubics or like we need to have like even more context or like privilege information and I will briefly discuss this again in the second section. Um so now I covered those like lubic

basation that can evaluate the quality of those long form open-ended responses. But another question is that even if the model can generate fluent plausible response sometimes the model can comput misrepresent evidence. So we also want to evaluate the quality of evidence or

like identify what kind of citations the model find useful. So there are some benchmarks including scholar events and DP search bench that also evaluate the quality of citation. So in this deep research bench um the paper use a framework called fact and

the basic idea is that they extract all of the claims paired with um external cited URLs and then fetch the source and judge whether these claims are supported by associated website. So in this case there are like two claims uh using the same citation but

only one of them is fully supported by the um this cited website uh site the website. So in this case the citation accuracy is 50% because only one of them is correct and also this uh deep research bench count how many of those uh good citations the model has. So this

is useful framework to evaluate citation quality but it still doesn’t check the quality of evidence of citation itself which could be another open question. so so far I discussed like different uh type of benchmarks in DP search uh which kind of like also provide nice overview

of like what those DP agents do in practice. And now I would like to move on to the model training. Uh but again like if you have any question feel free to ask me anytime. >> Okay. So now we have >> I actually have one question. So like we

have a lot of people in the class who are going to be doing a final project and like if they if you have convinced people to be interested in deep research based on this lecture which of the benchmarks do you think are good ones for like a scoped project

>> you know of the type? >> Yeah that’s good question. So the question is like if I’m interested in working on DP search agent project what kind of benchmark we should use? Um so my recommendation is that if you are interested in like short form uh

verifiable question asking task. Uh browser comp is a pretty good benchmark. Um it’s it has some issues but still like it has a lot of high quality questions and you can evaluate the like different like um like moing technique for example like context management and

you can see like uh interesting phenomena in terms of infant time scaling. um if you don’t want to like uh but browser comp is a big benchmark and also it requires a bunch of uh search uh codes. So if you want to reduce the cost of inference then like blossom compon

plus is another good benchmark and also for blossom plus again like you can like for example train your own embedding model and like evaluate vable system which is interesting area of research. Um if you’re interested in open-ended DP search benchmark um auto benchmarks have

some issue but I think DP search bench 2 is relatively high quality I would say um but DP bench has been widely used but to be honest like DP bench it’s kind of like the number is already super high um also for expert domain benchmark I think f search comp is decent um but there are

also new expert domain benchmark I haven’t really discussed and I’m happy to answer question if you have any specific questions. Okay. Um okay. So, oh yeah. Um yeah, thank you so much for the question. [laughter]

Um okay. So now let me talk about what does DP agents actually do uh with like more like low lower level look. So given the question for instance like can AI agent um synthesize scientific literature as well as human expert first those uh language model agent produce

ving token and then like discuss like plan uh like for example like thinking token and then say like oh we need to find those papers and after that uh those agent like again generate two codes which you’ve been studying in this class

and then we use external search system and those search system return a set of documents and those documents will be a part of will be part of agent context. So now we have initial plan uh previous reasoning tokens and also a set of um to output.

So now language model resume generation and defect on its evidence and then decide whether to continue searching or not. uh depending on the system we use like many different uh tools uh for example like Google web search browsing um or like even paper search

and after enough those iteration the agent provide uh produce its final answer ideally with external citations. So how do we train such model to perform well on those complex DP search task? Um I’ll use Tony DP search from Alibaba Ariaba toy team as an example.

It’s best has three different training stages although like no ABDP search agent uh system use all of those three. So for 20 research they start uh with mid training which expose the base model to agent behavior at scale and then they conduct supervised fine

tuning uh which teaches it to me mimic or imitate teachers projected behavior and finally they conduct logical uh reinforcement learning uh which lets the policy language model learn from feedback on its own attempt. But the important question is that all

of those stages require high quality data. All of those three stages need like input and answer for verifiable questions and also mid training and supervised training. They needed to have high quality trajectory data. So where those where are those data coming from?

To talk about those initial task creation, I would like to briefly recap Navage graph just in case like some of you may not be familiar with Navage graph. Um so nanis graph and navis graph represent entities as node and relationship as

leveled uh directed graph and for example an nuo and kangi manon can be two nodes and then those can be connected by a relationship indicating that he was a faculty member here we can then connect a lot of more entities like people institutions book

and award via those additional relationship and large resources such as wik wikid data contain many of such fact. The useful idea here is that we can follow those connections to collect a lot of factual information

and some DPS search training pipelines uh construct or use such large scale knowledge graph and sample a set of connected facts with random walk. they may view those graphs from web page web evidences or use um existing large benchmark large web uh knowledge graph.

So here we start with this target entity on a new and then gather related fact. So here I highlighted some of the edges and this basically described that this um Arin New was a CMU faculty and also he received this uh 1975 ting award and also he co-authored uh this like human

problem solving with Simon who received the economics no applies. So now we combine those a set of like factor claims to formulate a complex question. So now we have a pretty detailed complex question and then we can use a visual starting entity as a

gold answer. So maybe you can you notice the similarity to blossom comp answer uh construction question and answer construction process. So in some sense this training data construction mimics the structure of the target benchmark and kind of like

reverse engineering the original benchmark and it isn’t qu it is open question if this specific way of construct training data and train model on it actually like transfer to like many different kind of DPS such benchmark but this gives us a scalable

way to construct high quality complex DP agent benchm uh training data um but once we have such question and answer pairs. We still needed to have high quality teacher trajectory data so that we can teach agent model to mimic such behavior.

What we do typically is that we ask a stronger teacher model like proprietary model or like open a much bigger open model to generate a set of trajectories uh for each question and some trajectories can be successful identify correct answer while other trajectories

can include errors in terms of two codes or just like couldn’t answer correctly. We filter we filter out those um unsuccessful trajectory based on answer matching or like a format reward to identify successful trajectories and then use successful trajectories to

conduct supervised fine-tuning uh so that student model can imitate those successful behaviors. Some systems go one stage a year and do agentic mid training. So the idea is kind of simple. We mix large amount of agent trajectories into continue

continual pre-training together with like normal pre-training data and then we do the same downstream SFD. So the intuition is that by doing this training we will have like better agent ready base model than the normal uh standard base model.

So this paper studies the effect of such like mid- trainining or a continual pre-training um for agent for DP such a agent and this actually provides some foundation of the 20 DP search paper. So here we have two baseline one started from quens city based model and the

other one from is from like their mid training model and they use exactly same SFT data. So when we look at the model performance after SFT, we can see that especially on more complex task like blazer comp uh benchmark the mid-trained model shows much better performance than

the original uh base model uh checkpoint. So this kind of tells us that having those agentic mid training can help us to achieve much better final performance after SFD. But SFD and M training kind of teaches the model uh from demonstration and

mimic teachers behavior. But eventually we want we want those student model to learn from its own attempt. So for the task with verifiable body for example for short-term answers we can use uh successful a recipe like reinforcement learning with verifiable body. So those

approachs have shown to be quite effective in a lot of like language model development like deepseek auto and in uh DP search agent there’s a paper called search auto one which kind of introduced this audio approach using JPO and showing result without DP

benchmark for the first time. So for instance given our CMA question the policy model generate multiple trajectories and some end with the correct answer and others don’t we check the final answer correctness based on the difference non gold answer uh using

exact match or sometimes use language model based judge and then we uh compute the final revol update the policy model. The key difference from SFT is that now the training responses come from the policy’s own attempt.

Um let me briefly recap GPO. I heard that you learn GLPO briefly in the previous lecture. uh so I would just like talk about um the basic idea but here uh for instance if we have uh this question and like three model trajectories uh two of them answer

correctly while like the other one didn’t uh find the correct answer. So now we have um one response which has like below group average and like two responses that are like above group average. And in training time um the reward term pushes down those

incorrect ones uh while like the correct attempt uh correct attempts above uh aid will be pushed uh via the training. And so I think you learned about this like advantage score but we also in practice we also have like additional terms for example we have uh the

objective also clips update to limited the size and also the version shows here uses a K kill penalty so that we will discourage uh the model moving too far from the defense policy and those constraints actually quite um important when we train those uh language model

agent especially like like large scale data Um I think you will learn more about Gio uh in other lectures or like in other NLP classes. Another important discussion is that how we can make this training manageable, efficient and scalable. Um, agentic auto

could be quite expensive and also unstable because now we have long agent trajectories with like hundreds of different search to call and each search can cost can cost and also takes longer um sometimes like we have to wait a second to get the final to output. So if

we wait uh every time we need to like use to cause that can be quite slow. So a lot of paper including 20DP research what they did is that um they execute many voters kind of like asynchronously. Uh while like one trajectory is still waiting for a tool response other others

can still like keep generating and then once trajectory finish we score them and assemble the buses needed for training. Um Tony also describes some different techniques to make training more efficient and scalable. for example like they c uh they cache uh to result or

like they use like more like robust API coding or even use like more simulated environment instead of like coding the external wheel API they use like local index and so on. Um so there are like many tweaks to make it more efficient and I think you will learn more in the

next lecture about uh auto system but here like let me just show final result from this like toy auto. So as you can see as training proceed um the vote increases all like um policy entropy remains relative to be stable. Um you can see like all of those like

nice blood from like toy and also like defend papers like search one. Um so this works nicely when the answer is something like an or like showto entities and you can just use emore to compute the final deote. But most real deep research systems are not just about

like exact like those short-term answers. Like for example, a good literature system may need like coverage, relevance, factuality or like appropriate citation. And we cannot just like have like binary correctness score here.

So in our doctor to or like DP to uh we build a new training recipe uh which we called uh auto with evolving lubics so that we can uh train DPS agent for such open-ended task. So here we use those like lubic based validation I discussed in the section in the previous section.

but the important question here is that how we can collect high quality lubics for such large scale auto as I said like collecting lubics can be quite expensive and as you um observe in this like research QA paper if you just for language model to generate a set of

rubik those rubik contain a lot of errors or like they’re just not that useful. So in Dr. to what we did is that we tried to um make those rubic generation model co-evolve with the policy language model.

So specifically we first create a set of initial rubik grounded in external search and then the policy generate multiple orders at training time and we generate new set of rubik during training time. So what we did is that we use language model to generate or like

to contrast multiple different trajectory and then generate a set of new positive and negative trajectories. So uh positive and negative rubik. So positive rubik um reward desirable behavior while like negative rubik penalize specific errors or problems

and then we maintain a buffer of uh useful lubics uh scores the responses and then update the policy as a policy exporters those rubics can capture or like incorporate new like distinctions. So the key idea is that the feedback um develops alongside the model the policy

model uh rather than demanding fixed at the beginning of the training. Oh okay. >> When you say by compressing rollouts you just exception by section compare a bunch of rollouts and like majority correctness.

>> Uh so when sorry so the question is that uh how do we actually like contrast multiple trajectories? So what we did is like we simply takes the final model response and then like we ask language model to uh like multiple responses like for Dr. Tudu the final training we use

like eight votes per question. So like eight uh trajectories and then like we ask language model to generate a set of like positive and like negative rubik so that like we can um distinguish good ones from like incorrect ones. But how to generate a good set of rubik itself

is a big research question. Like for Dr. We did like simple like language model based lubic generation but we didn’t like study like the effect of different uh ablation in terms of number of the trajectory at the same time or like if we should consider like intermediate

trajectory for example like two and so on >> so in Dr. We didn’t do much. Uh I forgot like exactly like how we tried like a different uh setup for example like we assign like uniform score or like we try like

different uh slightly different score but eventually like for doctor to the setup uh we didn’t like spend that much time to like estimate the importance of Rubik’s because even for expert um sometime even expert disagree on like which Rubik’s items are most important.

Uh so that’s another area we can study in ter for lubik based RO. Okay. Um so we found that after large scale RO using this uh lubix um lubric can beat pretty strong performance at the fraction of the cost of those propriety DPS system. So here the y-axis

is the ability score um accus benchmark and then x-axis is the cost of uh inference pquity and you can see like now it’s in like uh pretty high up in terms of performance but uh it used like thousand times less uh inference scores compared to like openp search and so on.

So this kind of uh highlight that with careful training even a small ATP language model can match or even exceed uh much bigger models in specialized agentic task. But I think there’s like I think the more um informative resultation here uh

first I would like to talk about uh the importance of SFT for those agentic auto. So here I show the model without uh four different baseline uh starting from different SFD or like base model checkpoint after um long auto training step. So at the point at the time of 600

steps when we start from base uh SFD model here like no SFD uh um when we start from like base model mean like here quen two model then the this model kind of like struggle and only achieve like 40% on this benchmark but like if we use just 5% of SFT training data

which is this undertrained SFD now we can achieve like um 10% u better performance after just like 600 uh auto step. Um so this kind of illustrate why we needed to have even like small scale SFT if we want to like unlock the uh aentic auto effectiveness. Um for Dr. to

do because of the limit of compute we have we only kept one baseline uh running uh for thousand steps and as you can see after those like 4,000 step we can see huge performance gain on this benchmark another interesting oblation we had is

that how effective this lubic based reward is so here I show multiple baseline using different deodor design and the this dust line is just like SFX performance without any additional training and then if we conduct a online RO with random reward then we cannot

observe any performance improvement if we do training with initial uh search grounded lubics then we can see much better performance showing the effectiveness of Rubik based auto um I didn’t include the result here but we also conduct uh ablation of lubics

generated with and without external information and we found that if we generate Rubik’s just by pumping language model then those Rubik are not as helpful as uh search augmented >> Rubik >> um you mean like the

>> lubrication >> oh yeah yeah >> can you use a model >> yeah that’s pretty a good question so the question is that uh can we use weak motor for Rubik generation. So we have ablation on that in the paper but for

the final model training we use GPT 4.1 uh to generate dubix but we have ablation using um quensity uh base model and we found that although like even quensity based model can give us huge performance improvement if you want to maximize the performance then like we

should use uh GPT 4.1 at that point and there was around like 10% performance gap in terms of final number um so back then we had to use like propiety um model to achieve the best number which wasn’t that fun because it was quite expensive if you needed to generate

dubix at each time step for different loadouts. Um so there are some papers some recent papers that train hoody open lubic generation model and so those model can be better alternative like still those models mostly evaluated um like benchmarks like short-term

evidation benchmark like reward bench. So it is still an open question if we can replace um strong lubic generation model with such open models. Um also we observed that those evolving evolving lubics improve performance much more especially like if we train model

longer. So um so this kind of show that both the quality of initial lubics and also its ability to adapt um is um important. Um so far I discussed about def agent but maybe like it’s like maybe like it’s a lot of information so feel free to ask

question anytime if you have um so before wrapping up I would like to um al uh I would like to talk about how we can train like with foundation for DP such model so far I just discussed how we can evaluate and like train agent

um but I didn’t like le say about like what this like search or like vivo means um but when we think of those deep certification those search foundation is quite important because if we don’t have good search then like even if we have strong agent model those model cannot

identify a good set of document so in different papers they try they use different search system for instance if we know exact target corpus what a set of documents we should retrieve from then we can build local search model. For instance, we can have uh the

discovery table model like BM25 or like we can have neural embedding model uh that compute the similarity between like document and query using trained um models. So for instance in the search auto paper they use um those embedding model local

system and like use which vari 2018 wisher as corpus and then like we documents. But when we build a DP such system for more like open-ended use cases for in uh or like much more complex cases like if you want to answer blog question or if you want to answer

uh scatter events questions then we needed to have u something beyond this like Wikipedia or like much small focus corpus uh so a lot of new papers use more like blackbox API like such as like web search API BS server or like web browsing API um

or like some papers like uh Tonyi or like Dr. Tudu use like paper search API like semantic scatter API or Google um Google scatter API. So depending on the task you can decide like what kind of external search tools you want to incorporate. For instance if you want to

work on finance comp maybe like you should like collect uh financial documents and then build like local data store or like use more like domain specific APIs. But now let me briefly talk about embedding foundation for vivbo. Um

because those embedding system can be helpful not only for just like uh DP search agent but also some other use cases in agent. So this is just like embedding 101 and if you want to learn more please go look at different um classes or lectures. But

in those embedding model we have trained document encoder model and then we encode all of documents we have in this uh data store offline to generate um a set of embedding or like in other words vector database. So now we have like city documents about uh new on Simon and

then we use embedding models to generate uh those embeddings and then at inference time we encode the question and and then generate those uh question embedding and then search for documents uh document vectors with high similarity score to it using this uh using nearest

neighbor search. So the query uh query and document encoders can be same can be different depending on uh system. Um but the important thing is that those u two embeddings um we needed to have embedding model that can locate uh

related positive documents closer to the query embeddings. So to train such strong embedding model quum approach is to use contrastive learning for each query. We assume that we have question and positive document and also a set of negative documents and

then the objective increases increases the relevant passage score uh relative to the negatives. So here the denominator normalizes over the candidate passages all of the passages including negative and positive and then minimizing the negative u minimizing the

negative look probability encourages the model to rank the positive documents higher. The negative documents can come from many different sources. For instance, this DPL is one of the most uh widely used uh approach for embedding base. And

in this paper, they use um idea called inbots negatives. And the idea is quite simple like they use positive documents for different questions in the same mini batch as negative documents or like we can mine more complex hard negatives using a weaker vetiva model such as BM25

and then mix it with other embeddings uh with mix it with other negative documents. But the question is that those embedding models are typically optimized for vable task while we use the final query uh as input. But DP agent has much more like

rich history. For instance, the agent knows the original task and also a set of like previous reasoning um previous queries and also the previous outcome and then like it formulate this current reasoning and then query. But we typically only look at the final query

which doesn’t uh always capture this rich history. So more recently we have been studying like how we can build better embedding model or how we can build better search system for such DP such agent. In this paper agent I io uh studied this problem

by passing both the current uh reasoning and also the query to viva model. So this final reasoning can explain the intend um can explain the intended target uh summarize useful finding from previous uh previous steps and also clarify what we need at this current

stage. So we found that adding this uh to the current combining the reason query and reasoning token already can give us performance improvement but we also found that if we train embedding model for this specific reasoning aware task then like we can get uh more

performance improvement. If you’re interested in the detail of how we train those models, uh please go look at the paper. So with this approach, agent io improves answer accuracy while reducing search codes in the report uh in the original

uh bloic plus experiment. So here each uh square means the model performance def performance when they use BM25 as backbone vivo system. If we replace this BM25 with co city uh 8b embedding model then we can see much

per big performance improvement on this browser plus while like reducing the number of the total search codes meaning that the model is now doing search a little bit more efficiently. We found that agent IO can significantly improve performance um on this basic

compon plus and also some other benchmark while reducing the number of the search much more from this uh queny HB model. And one thing I would like to mention is that agent IO is a 4B uh embedding model. So which is like much smaller than 8B qubased model. And we

were curious like why agent IO is so effective uh despite that the best is kind of simple like combining uh query and busing token and also just doing like uh small scale SFD. So we did some ablation to understand how we can build better foundation for

DPS agent. So here I first would like to show uh two ablation uh first would like to show the ablation of training and inference. So again if we just add um the latest reasoning tokens to the input query then we can see some performance improvement

but it’s not like huge improvement especially like on top of uh GPTOSS or uh GRM 4.7. So we found that um additional training can significantly boost the performance of this approach and eventually if we combine this reasoning aware inference

and the um specific uh specific training for this reasoning aware vivo can give us huge improvement uh acoust agent and one thing I’d like to mention is that for we use this small scale training data uh originally designed for like bliccom plus but we evaluated model

different benchmark mark and also like the training data is based on 20 dp size trajectory. So this results show that uh despite that the training is based on specific data set using one model still this specific can generalize across different agent models and also

different benchmarks. Another question you might have is that I say that we only use the final reasoning token but what if we use different um set of uh input context because DPS sets agent can have a reasonable intent uh a lot of like

different search code uh queries or like previous vising tokens. So maybe question is that can we just use the entire trajectory instead of just using this final vising token. So what we found is that if we use food context it actually doesn’t help that much or even

like it uh affect performance negatively. So here we have baseline without any additional information and also a baseline which used all of the previous queries. Um those previous queries can indicate what kind of information the model was trying to

search in the past but it doesn’t provide information about what they actually found. And so this is still helpful but it doesn’t give significant performance improvement. um adding their associated reasoning tokens can be helpful on top of this

query only setup but the performance gain is kind of small on some benchmark on some baselines or like if we add the entire trajectory including the ving tokens uh output to the output then like it can hurts the performance because now the context get

too long for this embedding model. So eventually we found that just using um PBS reasoning token like one step before the current uh query can be most helpful because those previous reasoning tokens can summarize the uh task and also previous search result effectively.

So today I covered a lot of different topics of DP agent and I’m happy to answer questions uh because we have a lot of time towards the end. Uh but let me briefly summarize what we discussed today. So first I discussed what DPS agent uh look like. Um then like I

talked about how we evaluate those agent. The challenge is that we have although like we have a bunch of questioning data set those data sets are no longer um effective in terms of validating those much more complex user needs or like testing those much more

capable DP agent. So a lot of recent papers try to build more complex benchmark like bic comp or like they try to introduce a better way to evaluate open-ended businesses but still like those evaluation framework have some limitations like for

example like how we can build high quality lubic based separation. So I think there are like interesting research questions on like how to build effective DP search or like information seeking benchmark in general. Um after that I discussed different uh training

detail of DP such a agent and so a lot of work focused on short form verifiable answers and successfully apply autovl uh those successful frontier language model training recipe but when we try to train agent for more open-ended task where we don’t have clear binary training signal

then we have to think of how we can design the for those open-ended task and one solution I discussed is rubik based auto And that shows improvement on data set we have but again like rubic based IO still have a lot of open questions as some people asked during the lecture

and finally I also talked briefly about va foundation of DP such agent and information retrievable itself is a huge research area so I didn’t really talk about uh all of the details but the important question is that those

uh vable systems are primary developed for much more simple va augmented generation setup or like for human users and we still don’t know how to build better viva foundation for those more agentic system and agent io is a good example showing that we have to have new

way to build viva models so because now we have like much more rich information on top of those search query but I think there are like many different ways to improve viva foundations and it’s another like interesting research area I’m kind of excited about too. Um, so

now I’m happy to take questions too. [applause] Yeah, that’s pretty good question. So the question is like uh if you work on it as a research project like which what size or like which model we want to build uh kind of

>> yeah like this is the best 7B model that I’ve experienced in deep research like for coding I know the answer. >> Oh yeah >> but for deep research I don’t know the answer. So I was wondering if you have any ideas about that.

>> So there are a lot of new deep research model that kind of show strong performance on 8B. Um but I think especially these days a lot of um like u model like qu models are pretty benchmarks for like blaz plus or like blazer comp. So even like this they’re

like updated at quen 3.8 a those models can be a good baselines uh if we want to build and it [clears throat] is actually hard to like input performance from those hoodie like benchmarks models um so that’s one for HP I think those are good benchmark we um baseline we should

consider um but ideally like we want to show effectiveness or like we want to show that our trained model can beat even like stronger uh much bigger models for example like uh open AI GPT6 is getting too difficult to beat but like I DP56 um or like DP search models trained

for DP search task for example open AAI DP search or like popex models um for those it’s if you want to build like strong general like model that can be GP6 which going to be like quite difficult but if we understand what kind of task we want to solve um then like

and also like think of careful training recipe for those setup then like there’s a good chance that like you can get uh competive ive or even better performance than those proprietary models on this like benchmark. So that’s another interesting thing you can work on for

your final project. >> Yeah, that’s pretty good question. So the question is like if we have uh if we want to work on deep research now like we have viva model and also we have agent model and like what model we should use if we work on like for

example if we work on agent side which model we should use and like if you want to work on viva model which agent model we should use. Um so I would say uh as a good starting point uh using just using like web search API um it’s going to be a like safe option for like DP search

agent training um for web search people use serer API which is kind of a stable and fast enough API for like model training and evidation and if you but the thing is that if you train model uh across like many different like uh training step for auto and using like

multiple voters it can be quite expensive for example like recently we have been training like um DPS model for bl and like at some point like we use like 3,000 or $4,000 US just for like one auto run so it could be quite expensive if you want to do large scale

auto so for that uh in that case I would say uh one you can focus on uh blood plus which doesn’t use actual real world API or like during training time you can use local index just like use BM25 or like quensity uh 4B or like ATP embedding model using your local

documents and for embedding I didn’t talk about it but there’s a uh some like readable like MT EB uh where like you can see the state of your embedding model so you can pick I don’t know best embedding model from the VA but I think quensity uh 4B or ADB embedding model

are actually quite good vable embedding models to start with um if you want to work on vetiva or like embedding training um for I would say like you don’t need to think of the final like state-of-the-art number but instead like people will be curious like if this

models work well across different uh agent model so I would like pick up some strong uh official um model checkpoint like GPOSS or like GM as in like agent IO and then show effectiveness like defend DP agent instead of like hyper optimize for like single DPS and models.

So let me uh repeat the question. I might not understand correctly but the question is uh like whether we should use DP search as one of the tools uh for agent pipeline or [laughter] yeah that’s sorry it is be hard to hear.

>> So like for the general like strong like powerful motors those models are not specifically trained for deep research. Um now like we are trying to like build uh HB model that can beat those models >> then um

>> what is the capability of >> okay yeah that’s a good question um so to go back um so the question is like um now like we talk about like training like specialized 8B models and also um and then those 8B model can beat a much strong like frontier model that may not

be optimized for DP search agent and like what are the fundamental like capability gap between like those small HP model and like those pretty big Fontia models. So there are like a lot of differences to be honest like I would be like yeah I I want to be honest like

it’s really hard to beat Flia models which might have like uh tens of uh much more like parameters using like ATB models um and for some task for example like blaz uh those bench on those benchmark the strong like models actually show much better performance

than like 8B model and one reason is that those plenty models have much better ability in terms of managing long context and also like plan or like vision and for this capability we found that even though like we can do a lot of uh supervised fine tuning still like the

best 8B model or even like toy 32B model cannot beat the best um like model or even like the best open model like GLM and so for that I think the underlying limitation is like the small model just like don’t have good ability to like manage like many many uh different

search calls or like context or like just like vision carefree um but for open-ended generation task for example like healthbench or like scatter QA bench or dp bench I showed like Dr. performs pretty well and we found that for those open-ended benchmark. The core

challenge is not about just like managing uh a lot of context or like home long flight task but it’s more like uh do like decent job uh collecting information and then like synthesize and formulate the high quality long home responses and this for this aspect we

didn’t see big gap between like model or like model or maybe like we don’t have good we don’t we are not doing great job to evaluate non-com responses but so I I would say like depends on the benchmark um some benchmark require like um skills that models just like don’t have

compared to frontier models. Um so that’s why I say like if you want to work on DP such a agent that can uh beat frontier models then like we need to carefully think of like the target benchmark and task and but I think I personally think that it’s also

interesting to study like how we can build much smaller model that can like um reduce the gap between frontier um and like those open smaller model or like just in other word uh maximize the performance given the compute limit we have. Okay.

Um yeah the question is like what the biggest bottleneck is in DP search agent. So I would say um evaluation uh for now and the reason is that if we know benchmark or like if we know the target we kind of know like how to optimize those model um but the

challenge of um open research when we started working on Dr. to the project. There are like a bunch of paper that work on like short form question asking task uh like basic comp and for those benchmark we have seen pretty strong numbers uh mostly because we know

exactly like how to uh like reverse engineering the benchmark and also like we can take existing successful recipe from like deepse paper or like other coding mathematical reasoning papers and just apply it to deep such task but what people didn’t study a lot is that how we

can build strong open-ended DP such models um like those deep such bench and that’s simply because like we even didn’t have a good benchmark to evaluate those models for um so I think evaluation is basic is now like foundation for model training as well

because if we know how to evaluate those models then like we can like use it to construct those large benchmark large SFX training data and then we can do rejection something or like we can do auto uh with like those evaluation function but if we don’t know how to

evaluate or how to uh measure progress then we cannot really like uh optimize model flow so that’s why I think evac is one bottleneck right now and when we look at the existing benchmark like dp search ben u dp search bbench those like longhorn benchmarks uh kind of like

realistic compared to browser comp but still there are many issues in uh those like lubic basaration and there are like a bunch of papers showing that lubic based separations have their own like biases is for example like biases towards longer bounces um or like those

uh blazer comp is quite interesting benchmark but still the questions are quite synthetic or like kind of artificial uh questions. So it’s an open question if those questions defect actually like users information need. Um another interesting research question

we’ve been kind of working on these days is that existing those DP search benchmark like DP search bench focus on evaluating the quality of final model responses. But when we think of like how people use DP search agent, we don’t like just look at the final response

like we use those responses to improve our like next action or like to make like informed decisions. Um or like we use those research agent to understand the potential bug in our code. So instead of just evaluating those final research outcome, we also like want to

test how those research outcome can help people to uh perform the final task. And there’s no benchmark evaluating this kind of more extrinic aspect and I think this is another interesting idea for research too. >> Yeah. Yeah. So the question is like this

open uh this doctor to approach the evolving uh audio with evolving lubics can be applied to like other open-ended benchmarks. Um so we didn’t work on it. Um but there are like a lot of photo papers recently about like how we can build how we can do like better like

lubic based auto and actually like some papers kind of apply evolving lubik for more uh for openness generation not like deep research specifically for example they use this evolving lubics recipe to improve performance on like uh more like general chat like arpaka hard those

benchmark but I think those uh rubik based audio itself can be quite general and applied to like a lot of like open-ended language model task and I heard that even in like frontier language model development people are still trying to figure out how we can

optimize language model when we cannot define clear binary reode um so this idea is quite exciting for me um in the context of DP search but also like beyond DP search yeah that’s pretty good question um so the question is that when we evaluate DP

search agent uh how can we ensure like those model just don’t like memorize answers from like like doing pre-training or like post training whatever. Um so that’s pretty good question when they build those benchmark they found that even state of jet models

uh cannot answer those complex question from their primic. So they had like some verification process but it’s an open question like if GPT6 memorize those component or not and to be honest I feel like the pre-training data post training data are quite in frontier labs is quite

contaminated so maybe like they have seen the data during training time so it’s it’s a little bit hard to ensure like those mods and not just like answering from their own knowledge but there are some more recent papers that a body those like new more like recent

models on blood comp without external search and they found that still those models struggle and cannot achieve like higher than 20% I would say um but related to the question one caveat we have in deep research evalation uh is that I mean like it’s it’s not about

just deep search but maybe agent task in general is that sometimes the model just directly retrieves the original data set and So in Dr. Tu we I I don’t know if we eventually includes a result in the paper but we found that uh if we don’t have a set of like block list or like

the website we shouldn’t look at then those motors actually like retrieving uh from like hugging face or like the original like open AI data set. So we had like some discussion like uh for DP search or like search agent in general like we needed to like make sure like we

are not directly like retrieving the original data set or like at least like we should like block hugging face for operation that’s kind of a related discussion I would like to mention. So,