高级 RL 与系统：算法四失效，工程一本账

So uh hi everyone. This time I’m going to talk about advanced RL algorithms for agents. Um one thing that I did in preparing this lecture was think of all the things that have gone wrong to me uh while training agents in the past year or two. Uh, and so what I’ll try to do

during this lecture is try to point out a problem that you will encounter when you’re training agents and a solution for it. And so hopefully if you’re encountering any issues during your assignments or during your your projects, the information here will be

useful to help you overcome them. I intentionally didn’t cover anything that doesn’t fix a problem that I’ve never had. So hopefully it will all be all be practical advice here. So we have three lectures on RL because it’s such a big part of AI agents

nowadays and we already covered the RL basics and today I will be covering some advanced algorithmic topics and then in the next lecture we’ll be talking about some advanced systems topics. And so what I won’t be talking about this time is you know how do you sandbox agents to

make sure that they you know don’t destroy your file system how do you uh run them in parallel uh th those sorts of things. Today we’ll mostly be with respect to the algorithms in the math. So I’m going to be using a running example during this entire lecture just

a very simple example of fix getting a coding agent to fix a bug. And so this is also similar to what I talked about in the coding agents lecture, but the prompt is uh fix the retry setting. Zero should disable retries, but the code returns three. Preserve the other cases.

And the initial state is buggy code. And we have a verifier uh that runs tests. And all of these tests need to pass in order for you to get a reward of one. Of course, you can slot in any example that you want here. It could be a math uh problem solving problem like we

talked about before. It could be a web navigation problem. It could be a deep research problem. You know, pick your favorite one. So, let’s look at what a successful trajectory for this task may look like. The first thing that the agent does in

this case is read the function. And it changes the default handling. Runs the four tests. One test still fails. It goes back and changes the code again and then it reruns the four tests and all four tests pass. So this is a successful trial of course, but in the middle it

made a mistake. So there there were some places where it didn’t do as well as it could and it probably could have done a better job had it, you know, not made that mistake. It would have finished more quickly. So of course the final reward here is

one. And so the RL training loop as we discussed last time basically what we do is we sample uh score and update and the this time I’ll be using a bit of math and so the thing that we want to

remember is basically we’re going to be trying to maximize the average reward on the task. We talked about a bunch of different ways to do this like reinforce and gpo the previous time but we’ll all be in service of this uh you know of this job and just to make sure everybody

remembers the math the uh symbols that we’re going to be using we have theta which is the parameters we have p of theta or pi of theta which is our policy tow which is a trajectory and r here which is a reward. So

if we look at what reinforce did and part of the reason why we decided not to uh use reinforce uh overall reinforce basically if you got a reward of one you would increase the likelihood of the output uh by essentially one and if we failed we would do nothing. we

would you know not update uh not update the policy there and this was good but as we mentioned in the previous time this is high variance and high variance is uh leads to optimization difficulties. So we want the uh failed attempt to

still provide a learning signal and we want to also come up with more fine grain rewards than just uh one and zero. So the first thing I’d like to talk about is credit assignment and more dense feedback. And so if we look at what we did with

the gpo previously, we would do something like sample four attempts to the same bug fix uh test task and then compute their rewards and use the group average as a baseline. So this looks like uh this where we get you know plus 0.75 for the positive

example and minus 0.25 uh for the examples uh other examples. one issue with this is that if you looked at a successful trajectory, the successful trajectory might have a mistake in the middle and then you recover from it. And so if you’re giving

a positive weight to all of the actions in the positive trajectory, then you’re also giving a positive weight to the mistake. And sure, that might be fine. you know, eventually it recovered and it did a good job, but in general, you don’t want it to make mistakes. So, if

you can avoid this, then you would like to avoid that. And then for failures, you might have the agent doing the perfectly good job up until, you know, around here, uh, but then suddenly at the end, it falls off the rails and and stops working. And so, you don’t

necessarily want to penalize all of the useful work it did just because it failed at the very end. So this is the idea of more having more fine grain rewards. So turnle rewards uh are a way to do this and there’s a bunch of different

ways to calculate them. So going back to the previous uh class we talked about reward to go. So reward to go is starting uh you know just before A3. how much reward remains on this trajectory. So we add the remaining ones and this

gives us uh the road to go uh big R3 which are the rewards received after H3. All good I think maybe you remember from the previous class. Cool. Okay. So if we look at what the reward to go looks like in the positive and

negative case, of course we’ll get one one one and 00 0. So if we reward the mistakes in the successful trajectory and the good parts of the uh failed trajectory with this obviously uh you know we’ll be not rewarding [clears throat] each

individual action in a fine grade way. So then the question is how can we estimate which turns helped when we only observe the final reward? Any ideas about this? What would you do to predict whether a uh an action was a good one or a bad one

when you only get from your environment whether the tests passed or not? >> Yeah. to to get finer rewards. >> You can write rubrics to get final fine grain rewards. Yes, that’s a great idea. We’re going to talk about it more later,

but that’s not what we’re going to do right right away. Yeah. >> R= actions that are shared. >> Look at all the rollouts that uh do get R equals one and see if there are common actions that are shared. I

think that’s a really good idea, but how would you like actually operationalize that? >> Okay. Um, I I think it’s a it’s a good idea, but it’s not the easiest thing to operationalize, I think. Yeah. We try to get a bunch of trajectories and see what

distinguishes successful sequences of actions pres >> Yeah. So I think it’s a pretty similar idea I guess you know like look at the shared actions and the trajectory. Um I

I think that’s good. you need some way to judge which action sequences are shared uh either with like exact overlap or with a language model or something like that. I’m actually not going to talk about that this time, but it’s definitely something you could do. Yeah.

spread across. >> Yes. So, um the the answer was maybe we could have multiple definitions of reward, not just the final success, but also whether it’s like a shorter trajectory or something. Uh yes, that’s great. I’m going to talk about that

later, too. Okay. Yeah, I I think these are good points. They’re actually uh none of them [laughter] are the ones that I was going to uh going to talk about next but um I I am going to talk about some of them farther down the line. So um

so the way the typical way that we do this is by actually training a model to predict whether it looks like the agent is going to be successful given the current state and use this as uh something called a value function. And so if we look at for example state

three and then we roll out 10 times after this then it succeeds three times and it fails seven times just you know in this particular example. And so if this is the case the expected reward here is uh approximately uh 3 divided by 10. So 0.3

and you know obviously sampling 10 is one way of doing it. If we sample infinitely we can calculate the expectation of the uh reward to go from here and that will become our value from our uh policy. So the problem is doing this at every

step would be prohibitively expensive of course because you need to roll out at every step and so we can’t do this uh realistically. But what we can do is we can learn to a something called a uh like a critic or a value function that predicts this expected reward to go. And

the way we parameterize this is with parameters fi. And given some prediction of the reward to go, we then train a model to predict this. um and then uh we train this with essentially a squared error loss. So the best answer uh if the average is 0.3

then is that we would uh then predict 0.3. And so we can you know parameterize this and train a model. Specifically how do we parameterize this? There’s two common ways of doing this. Uh the first one is more common

than the other one. But basically what we do is we feed in a history uh HT and we train a value network uh which is maybe an entirely different language model and that value network can make a prediction uh over you know what the reward is

going to be next. The other way that we can do this is um you can so this is good but the problem with doing this is that you need to train two models right and training one model is already hard enough. So another option that you can do is you can take

the body of the model that you’re training to do reinforcement learning or to do to be the policy and then add a linear value head on top of it and predict uh from the embeddings that you get from the model. And if you do this, this value head is tiny and so it’s

basically no extra parameters, but it still allows you to predict the value. And then uh one other thing that I should mention is there’s two I I think probably it it’s more common over here. But if you you’re on the left

side, there’s two ways that you can do this. Um one way that you can do this is you can actually have a regression head that just directly um predicts the value. And another way that you can do this is you can put like is this trajectory good?

Yes or no? And then predict the next token. And if you do this and you have a reward between one and zero, you can fit

the normalized probability of yes divided by the normal by the probability of yes and no and use that as your prediction. And so that that kind of only works if you have a probability, but this has been this is pretty effective because you can reuse um you

know kind of the the prior that the model has for making the predictions. basically if you do this um this allows you to get a model that can predict how likely you are to succeed uh going forward. And you can train this at the same time you’re you’re training your

model using RL. Is this clear? Any any questions? >> So my question is If the model is correct or not, why not just like correct? >> Great question. So um the the question

was if the model itself can tell whether the action is correct or not um why not just produce a correct action uh for the model. So one important thing about the value function is the value function actually doesn’t condition on the action in this case. It only conditions on the

history. And so it’s not actually predicting whether the action is good. It’s just predicting whether the history is good is one thing. So um the the second reason is sometimes it’s still a lot easier to tell you’re doing a bad job than to know what you would

need to do next in order to do a good job. So, you know, I’m sure we’ve all been in the situation where we think we’re doing a really bad job and wish we knew how to do better, but we don’t. So, um, in this case, discriminating between good and bad trajectories is easier than

like actually generating a good next. Cool. Um, so now we can predict how well we’re doing. Um, how do we actually use this in training? And there’s a number of ways that we do this. Um one uh thing that we can do is uh if we look at a series of actions like we have

action three, action four, action five, we have our before value, our after value, um before value, after value, before value, after value, and the reward. And we can add the reward, add the before

value or sorry add the after value and subtract the before value. And this essentially gives us an estimate of how good each step is like how much did increase the val uh the value function, how much did it decrease the value function. And so if your action was

really bad like let’s say you you accidentally deleted your codebase and suddenly your value function goes to zero then uh you’ll be able to see that and be able to see that it dropped the value a whole lot here and this is uh called the temporal

difference error and so uh this is signified as delta t where uh we take the reward after value minus before value. So, how can we turn this into like an actual reward for each of our steps?

The most common way to do this is generalized advantage estimation uh which is uh used in PO. And basically the way it works is we have um sorry used in a reinforcement learning algorithm called PO which uh is is pretty common. Um so essentially we

start with each of the temporal difference rewards and based on the temporal difference rewards we have a parameter uh lambda where this parameter lambda uh is basically exponentiated at each step and then we add the temporal difference

reward in all of the lambdas from the beginning. uh starting with uh lambda to the zero and then lambda to the one and lambda to the two and uh you can see that we essentially will get a reward for this step. So the step itself was positive. So we get a reasonable reward

here, but it was followed by two bad steps. And so what this means is essentially step three might have seemed like a good step itself, but then in the future it ended up uh having us make bad decisions. So it maybe is not so great. So we drop its temporal difference

reward by a discounted uh kind of change in the future. So now that we have this, as long as we can train a value model at each step, we can kind of estimate how good each step was. >> Got it.

>> So we kind of collect steps and then >> I mean the if I go back, sorry, I I think I had this a really long time ago. If I go back, I I was using um

mu. Yeah, sorry I can’t find it on the slides, but I I was using mu here. And mu is basically our rollout policy. So it’s whatever policy we use for inference. and that if we’re doing complete on

policy learning that will be the same uh like policy that we’re updating but I’m going to be talking about async learning where we do rollouts and updates with different policies but um like you can you can think of it as the same model we’re training

>> yeah It’s basically a hyper it’s a hyperparameter. Um a typical hyperparameter for this is one um when you’re training PO and what that means is essentially we’re not considering the temporal difference. We’re only considering the final reward

because if we don’t do any decay the it will be the difference in the value function every time. And you can kind of tell like if you if you add these up and you don’t have any discount then essentially each time these will cancel out and

you’ll only get the the last one. So sometimes often with PO we won’t do any uh we won’t do any temporal discount but you can set a temporal discount if you have a good idea that like number one you have a good value function and number two you have like or you have

like intermediate rewards uh through rubrics like what was mentioned before for instance. Cool. Okay. So this is uh this is the typical way of learning intermediate some sort of uh you know value estimate. Another way that you can do this uh

which has been used uh to some extent although not quite as uh not quite as widely is to train something called a process reward model. And what a process reward model is is it’s essentially a model that was trained a priority or separately from the uh learning process

to predict how good each step is. So in contrast to the value function which predicts you know directly as part of the the learning process the process reward model it is trained on some sort of annotations and so this uh paper uh let’s verify step by step this was a

kind of famous paper by open AAI kind of early in the uh reasoning model phase they basically just had a whole bunch of annotators annotate which of the steps was uh was useful. I would say this is not used very widely because you know you can learn it as a byproduct of

training but it could be a very good way to kind of bootstrap a value function. So if you are able to somehow get lots of these annotations you could learn an initial better value function which would make your training faster beforehand.

Okay. So the um the partial rewards are a good way to increase the stability of learning by giving you by you know giving the model a better idea of which actions are useful and which actions are not useful. So the next problem I would like to uh

consider is that reinforcement learning for agents it can be very slow. Uh and you will certainly experience this and be frustrated by it in your uh in your assignment. So this is an example of uh synchronous reinforcement learning. and synchronous

reinforcement learning. Basically what you do is you have uh you know three GPUs uh for instance where you have uh a model running on each GPU and you use each GPU to generate uh using an agent. Um it doesn’t necessarily need to be

split by GPUs either. You could just have like multiple agents sharing the same inference server on a single GPU. Either way, you want to have multiple agents running at once because agents running is is slow essentially. And here this figure is a little bit charitable

to the models or to like what actually happens. But in reality, if you’re training a coding agent, it will be very common to have one straggler that takes like twice as much time or four times as much time as all the others. So all the others are just sitting there and doing

nothing while you’re waiting for that one straggler to finish. Um then after that you do an update synchronously and then you generate again. So there is one advantage to this. Um does anyone maybe someone familiar with

reinforcement learning know what the advantage here is? Any ideas? >> Yeah. It’s on policy. So it’s completely on policy. And what I mean by that is the um the policy that is doing the

rollouts, the policy that’s doing the generation is exactly the same as the policy you’re updating because you know you’re doing the the generation and then you’re doing the updates. So we had uh the mu pi mu or p mu and then we had pi uh theta and in this case mu and theta

would be the same. So we’d have exactly the same thing uh generating and updating. So to fix the wait time, a method that you can use is async RL. Um and async RL basically what this does is it will do an initial it

will start initial rollouts. it will accumulate a batch of rollouts and then once you have accumulated a certain number of rollouts you’ll start doing an update but while you’re doing the update or while you’re calculating the update you

will continue generating and this is good obviously because you never waste time waiting uh to generate. The downside though is you’ve started generating here and here before the update is finished. So the generation uh either up to this point or up to this

point is using the uh old policy before the update. And so now when you do the second update um you need to consider the fact that essentially uh the parameters were different uh for part of the generation. So this is absolutely essential in

Aentic RL. Nobody who’s doing Agentic RL seriously would do synchronous updates now, especially now that the software has gotten pretty good to support um async updates. But the problem is uh this can lead to a lot of instability in in training your models.

So how can we what is the problem through math and how can we fix it? So this is an example of what it looks like when you have rollouts from an older policy. So if you have your rollout policy uh mu and your learner policy uh pi theta,

it might be the case that there is a very different probability for each of the outputs uh for your rollout policy and your learner policy. And what that means in math is essentially

um we are ignore this for a second. Um we want to calculate the reward according to the update policy but what we actually would get is the reward according to the um uh to the rollout policy. So fortunately there’s a way that we can

fix this uh important sampling and I I think maybe many or most people have heard of important sampling. How how many oh actually surprisingly few. It’s um so the way important sampling works is um if you have one distribution that you

want to sample from but you cannot sample from that distribution and instead you can only sample from another distribution. You can correct the expected value that you get by sampling from the sampling distribution by having this uh ratio here where you divide by

the probability of the original of the sampling distribution and multiply it by the probability of the distribution that you actually want to sample from. Um I ju just trust me [laughter] this is uh This is uh true if you haven’t encountered this before.

But this is the way that we can fix this problem because you know this is the distribution we actually sampled from and this is the one we want. So uh we can apply this importance weight uh which is uh an adjustment factor like this. And for instance if we have the

correct edit where the current one is 0.7 and the behavior is 0.4 we would get an importance weight of 1.75. And um uh we might get a importance weight of 0.5 uh over here. And so essentially this will um upweight ones that are more likely under the um under

the true distribution and downweight ones that are less likely under the true distribution. there’s um so this is good but the problem is if you just use the importance weight as is you can get a lot of instability

and the reason why is if you get e even if each action is only a little bit different within an agentic trajectory there could be many many actions you know 100 is a charitable interpretation you know an agent agentic trajectory will have like 50,000 tokens or or

60,000 tokens or something like this. So even if the differences are small, the number can get very very large. So that’s a problem. Uh but um yeah, so so that’s a problem. The

uh exactly how much of a problem it is depends on how far into the trajectory you go because you can actually calculate the variance of the log the expected variance of the log importance weight by using the fact that as you add numbers together their variance will go

up by the square root of the number of items that you add together as long as they’re IID. You can derive this from the law law of large numbers and how everything can conver converges to like a Gaussian distribution as you add more iid numbers together. And so this is uh

from a paper called CTPO where basically they um they actually measure this empirically on reasoning trajectories. And you can see that indeed it is going up by approximately the square root. So another thing that you need to worry about is that earlier actions uh do edit

uh do change later histories. And so the next action ratio is um you know one in general on average it will be one but uh you know you the you can have compounding effects and you can get very very large numbers here. there’s a number of ways that people

deal with this. Um the most common way that people deal with uh with this importance weight getting very very large is through P uh the method that is used in PO and GRPO. And the way they do this is essentially they throw away

they they don’t consider the fact that the importance weight is actually the product of these weights and they only consider the importance weight for the current action. And so this is not actually correct. It’s not actually the correct importance

weight for the next action, but it’s good enough and it gives you some idea of uh like gives you some idea of how unexpected the action is. Another way that you can do this is you can do all of the previous actions. The problem with doing all of the previous

actions is that it will get very big as I mentioned before. Um, and so in order to fix this, uh, both PO and G, uh, GRPO and CTPO do some variety of clipping to prevent the, uh, importance weight from getting too large.

You could also do the full ratio. Uh, this is bad for the same reason. You know, it would get very large. So there’s another method that was proposed called GSPO. And what GSPO does is it essentially divides by the sequence length. So it considers all of the

actions but then divides by the length of the sequence which brings it down to being in the same scale as uh what we use for PO and GRPO. this is a bit of detail but like long story short you want to use importance weights but you don’t want to let them

get out of uh you know out of control. And actually I I realize now that maybe I should have put this slide um should have put this slide earlier but this is actually um an actual experiment that we ran in my lab. Uh so thanks to Aditia one of the TAs for sharing this but this

is essentially the um importance weight the maximum importance weight that you get in any individual batch of training. And the orange the orange thing here is uh the importance way when you’re running asynchronous RL with a maximum off policiness of one. And so maximum

off policiness basically means how many updates can you do uh how many updates can intervene between the things you’re currently updating and the rollout policy. And so this means that you can only use rollouts from the previous policy. you can’t use rollouts from any

earlier policies. Um the green one is a maximum off policiness of three. So this means that you could have up to three updates since the previous uh since the previous policy. And so what you can see is and this is the maximum importance weight.

So you can see that around uh step 17 of his RL there was actually one importance weight that was 1,00 you know on the order of 1,400. Um, and so you can imagine what would happen if you have a multiple of 1,400 in your update. You’re going to

make a huge gradient step and mess up your model and you might completely destroy training uh destroy training overall. um any do people understand the basic idea? I’m going to talk about how we fix this

next, but is the basic idea clear? Okay, cool. Um, so the way we fix this, uh, is somewhat huristic. Um, but basically in the PO paper, they introduce a way of clipping the probability ratio and and actually I don’t think they introduce it, but they

they use it. A previous paper introduced it. it you take the uh you take the probability ratio or the importance weight and you allow it to expand linearly but then you clip it off uh after it gets above a certain size. So uh for instance uh in

PO the standard way of doing it is like uh 1.2 and 0.8. We don’t let it get any larger than that. And there’s a few ways that you can use this clipped importance weight. Uh the way that PO does it is they essentially multiply the clipped importance weight

by the advantage. Uh so you know like which will be positive if you have a good example, negative if you have a bad example and then you take the min here. Um, basically it’s a little bit hard to intuitively understand, but basically what it’s

doing is it’s um stopping you from updating uh any it’s stopping you from updating beyond kind of the the region where the importance weight is very high. Um and then siso is just another optimization algorithm and it does something similar uh except it clips the

importance weight directly. So this is um a little bit uh like more detail on what PO uh what the PO clipping looks like. So here uh what we do is for a helpful action for one where the advantage is possible we stop rewarding a ratio uh which is above 1

plus epsilon where epsilon is something like 0.2 too. And for a negative advantage, for an action that is not helpful, we uh we stop like negatively rewarding it after uh the ratio is below one minus epsilon. Cool. Um, so basically by doing this,

what we can prevent is we can prevent these really crazy spikes in importance weight from causing our model to go off the rails. Cool. Um, so actually maybe I should briefly uh

mention what will happen uh in this case. So typically what happens uh when you when you have a really bad spike and you don’t appropriately account for it is the model will suddenly start doing something degenerate and these can be things like repeating the same token

over and over again. Uh other things include generating really really long outputs, really really short outputs. uh these sorts of things and it’s kind of hard to say it’s going to be any individual thing. It’s just like suddenly things look really weird and

you don’t really know why. Uh it might be because of this. It might be because of other things but it’s typically because of some you know large gradient uh step that you took that uh that caused the model to go off the rails. And the problem is it’s hard to recover

from this because uh once you do that you start not being able to get advantages, not being able to get you know good rewards. So yeah, it it will fail. >> Can you just clip the gradient instead of clipping the

>> Yeah, you can you can clip the gradient also. I I think the importance weights tend to be the biggest cause of the gradients going out of the way, but you could clip the gradients as well. I I actually don’t know. That’s a good question. I don’t know whether that’s

like the most common practice or or not. So I’d have to go back and look. Cool. Okay. So, other ways that you can help uh prevent the model from go uh completely going off the rails is to have a reference model. And the reference model is essentially, you

know, your initialized model. And you can uh prevent the reference model uh you can prevent the KL divergence between the updated model and the reference model from uh becoming too diff different. And so essentially this will allow you to um to prevent big

updates as well. It’s kind of another way of doing it. But um maybe the more interesting thing specifically for reinforcement learning is another thing that can go off the rails is uh the entropy of your model either collapsing or exploding.

And entropy just uh to to make sure that people are awake. Uh what what does entropy mean either in mathematical or kind of uh you know intuitive terms? Yeah. >> Yeah. How broad the probability distribution is. So, um, that’s a good

good way of saying it. Another way of saying it is how likely are you to select a variety of different outputs instead of a single, you know, a single output. And so, if your entropy is very low, it means that your model is likely to sample the same thing over and over

again. If your entropy is very high, it means it will sample diverse things. And so the problems we often see in reinforcement learning are the entropy will go way down and the model will lose diversity and so it won’t have a bunch of different things to compare. So your

gpo loss will um will disappear as I’ll talk about in a moment. On the other hand, if the entropy goes too high, that means it’s sampling a bunch of random stuff and it’s not like actually likely to get good uh good results. So both of them can be a problem. And so you can

essentially um set entropy regularization to uh prevent that from happening. So I introduced a bunch of concepts one by one uh because I I feel like this may be a easier way to intuitively understand each of the algorithms. But

I’d like to finish up this part with a table of uh many of the algorithms that people use. And so the first one is uh the weight that you get from the reward, whether you have a learned critic. Um whether you have an importance ratio and whether you have clipping and basically

um reinforce like we talked about before uh very simple. We just use the reward to go. Um no importance ratio uh although you you could add that if you’re doing uh not fully on policy, no clipping. So very very simple. Um, PO uh uses the GA uh from the critic as I

introduced before. Um, it requires a learned critic and it uses a current token uh importance weight and I talked about the clipping. GRPO is the same as PO but it uses group comparison. Uh, siso is another example. It’s been used by the miniax models and other uh

other things like this. It has a different variety of it’s similar to PO but with GRPO but with a different variety of importance weight clipping. GSPO uh does the importance weight but it uses a whole response length normalized.

Um and uh one other and deo I’ll talk a little bit more about it in a bit but one interesting thing in this algorithm is they actually have asymmetric clipping. So they they use a larger

upper bound than they use a lower bound which is basically to reward positive example uh positive advantages more. Um and so basically it’s saying we want to we want to maximize the uh we want to maximize the like benefit that we get from the positive examples

and then clip the negative examples uh like a same amount. So um if you want to you can go go in and uh read more details in the papers but this is the high level overview of the differences. So next I’d like to talk about task selection and reward reliability. So

this is uh le maybe not exactly on the algorithmic side but it has to do with being able to tell if you’re going to get a good result from the training examples that you have. So if you’re using something like GRPO, a really good value to monitor is the percentage of

zero advantage uh outputs. And so or or nonzero advantage outputs. So in order to learn from gRPO, your outputs must have at least one negative example and at least one positive example because otherwise there’s no variance in the um in the uh

advantage and they all revert to the mean and you don’t get any uh you don’t get any gradient. So it’s a problem both if all attempts failed and if all attempts passed. So, oops. So, if we look at why every trajectory might fail, there’s actually

a number of reasons. Um, the first one is verifier quality. So, um, it might be the case that you actually got a correct answer, but the verifiers did not grade it as correct. And this is actually very very common. So, you

should be aware of this. Another one is uh current capability. The model itself might not be good enough. Um, another thing might be lack of exploration exploration. And so it might be that the model could do it, but it just didn’t uh didn’t get enough um outputs. And so

to check all of these, we can look at the requirements in the failed tests um whether stronger model demonstrations would succeed and whether the trajectories differ. for false negatives in the verifier, this is essentially if you have some

sort of test that is unachievable. in the running example that I’m using, ideally your test would you know directly test the inputs and outputs, but you could think of a bad verifier where it tested directly that value is none was included in the program or

something like this. And you might think this sounds very silly, but agents will write these sorts of tests all the time. Uh they’ll write, you know, tests that don’t actually test the underlying behaviors. So, uh you need to be very careful about this.

So, there’s a number of results of false negatives. Um the first one is reward noise. So, you know, if it actually got it right, but it got it marked wrong, you’ll get more noise in your reward than you actually need to have. Another problem with it is if you’re

looking specifically at benchmarks, benchmarks can plateau at lower values and so you might not actually be able to measure your actual progress. And so, basically, OpenAI has a blog post about SWEBench verified, which is actually, you know, a benchmark they created. So

they’re criticizing criticizing their own benchmark, but basically they said only about 81% of them are actually solvable. Um and if you look at the remaining uh unsolved ones, a lot of them are just verifier noise. Um and then another popular benchmark called

tobench at one point had a ceiling of about 70% on some tasks, 92 on other tasks. So something to be like even very popular good benchmarks can have this problem. So another issue is false positives. Um so false positives

can uh be the result of uh can result in reward hacking. And so basically what this means is if you keep the same testing requirements, but if you only have uh the tests check a subset of the things you actually want to be testing, um you could have a degenerate solution

that nonetheless gets a good answer. Um and so it it would not pass other tests, but this is incomplete. And so people have also pointed this out about a number of benchmarks. uh Sweetbench. Everybody loves to pick on SWEBench because it’s so popular, but there’s um

a paper called Swebench++ which basically uh examines, you know, all of the things that are not rigorous enough. but there there’s a number of problems that you actually very practically will encounter if you’re using strong

language models to try to solve these tests. So these are all ones that I have encountered uh not you know not going out of my way to encounter them like just agents will spontaneously do these things. So if the agent has internet access uh the agent will go

look the results up online. Um I’m sure a lot of people heard about the open AI hugging face thing. Um there the agent very aggressively looked things up online to the point where it hacked into Hugging Face. Um and so the answer to this is uh don’t uh don’t give it

internet access or at the very least don’t give it internet access to the sites uh that had the answers directly on them. Another example which has appeared in multiple coding benchmarks is they left the git history in the coding benchmark.

And because the get history was in the coding benchmark, the agent would discover uh that it still had the git history. It would fast forward the history to code that had the answer in it and then just use that code. And um I’ve I’ve fixed the benchmarks that had

this problem and and sent you know sent in them. This is another interesting one that I just encountered like a few weeks ago. Um, which was I had a model API key within the container that the agent evaluation was running in because I needed a model API key to power the

agent. Uh, and this resulted in that agent finding the proxy server, finding that the proxy server had other uh, agents on it and called a different stronger model to try to ask the stronger model an answer. And so uh specifically DeepSeek uh Deepseek V4

Flash was calling Claude Opus. So it was like um it knew that Claude Opus might have the answer to it. So it decided to uh decided to ask its manager or something. Um the final thing is if you give it access to tests or the test runner, it

will write specifically for the tests and um and go in and uh solve that. So all of these are are problems. um all of these are problems obviously because it’s not learning to solve the actual task. It’s learning to do something else. The even worse part is if you do

this during training the model will learn that this is a very effective way to solve that variety of problems. So um again talking about the open AI hugging face uh incident um one of the problems was they gave it um

the they had a problem of both false negatives and reward hacking where they gave it an impossible task where the there was no way you could solve the task without breaking out of the sandbox and using the internet. and they had a problem in the sandbox

that let the agent break out and and use the internet. And so it basically reinforced this behavior more and more and more because it’s like this is the only way to solve it, but this, you know, is a way to solve it. So it got baked into the parameters of the model.

Um, for me personally, I use DeepSeek uh to try to solve some problems and it called the API I think 60% of the time and 60% of the time is a lot, right? So my intuitive uh my intuitive thought about this is they must have had this problem while they

were also training the model because otherwise it wouldn’t have this behavior so baked into the model itself. So um it’s something to be very very careful about when you’re training and I think the people are becoming much much more uh familiar about this and now you can

like actually view what the model is doing make sure it’s not violating any issue violating any uh policies or stuff like that. So um there’s a number of ways you can do that. Um you can also do uh independent checks of uh progress. So

essentially um you you do want to measure the training reward but you also want to be checking on uh unseen tasks and uh monitoring the behavior like I mentioned. And another thing that you can do is you can check uh you know the costs. Um sometimes when training is

failing it’s not due to the agents behavior but it’s because some infra fell over because as we mentioned um you know RL is a very big systems problem. So, it’s also good to be monitoring, you know, just all of your training runs to make sure that nothing is breaking. And

fortunately, now having an asking an agent to periodically monitor the training run is, you know, a very effective way of of doing this, even if you don’t have special tooling. So another um another reason why things might fail uh I mentioned is your

entropy collapses and it leads to a lack of exploration. So I haven’t seen like a ton of people use this method but it is one that was used in um in large scale training uh which is essentially if you get zero advantage uh rollouts you can roll out again and uh hope that you get

a mixed group again. So you you basically keep on rolling out until you get uh get a positive example. Um so this is one way you can deal with uh the fact that there’s very little uh very few positive examples or very few negative examples.

Okay. So um another thing that uh I would like to talk about is partial and auxiliary rewards. So basically right now we’re becoming more and more ambitious about the types of tasks that we would want to tackle. And

essentially um you know because we’re becoming more and more ambitious we might start out with a model that is not able to tackle these tasks at all. There’s a few ways you can fix this. you can come up with like simpler tasks and you can train the model on simpler tasks

first and then uh then train it on uh others uh other tasks later. But um one thing that you can do is you can give partial rewards. And so partial rewards are basically the simplest example of a partial reward is if you have four tests you give uh a

reward of number of tests passed divided by total number of tests. And so if you do this then you can get a a score a non-zero score even if you would have had a zero success uh reward and you can uh calculate something like GRPO based on that. The problem with this is

it’s much harder to design rewards because you’re no longer directly optimizing the final accuracy that you want. And if you design rewards that aren’t, you know, aligned with doing better on the task eventually, then this can cause you to, you know, essentially

like optimize those rewards at the cost of optimizing the main reward. Um, so I have two actual examples of this. Um, not to pick on OpenAI, but they’re both from OpenAI. Um, so one example is GPD 5.1 had an auxiliary reward for

using the web browser because they wanted to teach it to use the web browser. So basically what it did was it called the web browser uh navigated to a calculator app and used the calculator app to solve math problems. And so they basically ended up training a model that

every time it wanted to solve a simple math problem like oneplus 2, it would call the web browser and use like five extra seconds to get that like extra auxiliary reward. Um, this one’s even more funny. Did did anybody read this blog post on goblin where the goblins

came from from OpenAI? So for a while the GPD models really love to say like goblins. It’s like there’s a bunch of coding goblins in this uh in this uh problem or something like this. And the reason why is because they introduced they briefly introduced character

training. And the character training uh gave a little bit of a bonus if like the uh the character was like nerdy or the character was friendly or the character was concise or something like this. Um, but for some reason the language model that they used as a judge of the nerdy

persona thought that if you said goblin or gremlin, uh, it was very nerdy, I guess. And so they got extra award for saying goblin. And then over time, this just kind of like amplified itself and the percentage of goblins got larger and larger and larger. So this is a

fascinating blog post. If you haven’t seen it before, you can go in and see the like unexpected uh, side effects of auxiliary rewards. Cool. Um, yeah, actually I already kind of mentioned exploration collapse when I talked about uh Dapo. So I uh you know

it means that it will only pick like one type of action. Um so uh I’ll I’ll skip that slide. The final thing I’d like to talk about is learning from a teacher. And so this is particularly useful if you have uh a stronger model that you can use to train

a weaker model. So this could either be you know one of the one of the strong models that is just available uh nowadays or it could be like a larger model of the model family that you want to train um that sort of thing. And so

a very common thing or I I would say every model is doing this nowadays is starting from demonstrations. And so all the large language models will add mid training where essentially they add uh add demonstrations and uh uh already talked about this extensively in the

previous class so I won’t cover this um but the next thing that you can do is uh you can start with some simpler tasks and then gradually um move into harder tasks. And there’s a bunch of different ways to do this. I don’t know if there’s any like

science about exactly how people do this, but you can gradually modify uh modify your mix or gradually remove tasks from task categories where the model is mostly getting them right, etc., etc. Another thing you can do is uh

distillation. And so basically uh what you do here is you have a stronger model roll out on whatever uh reinforcement learning environment you want to be training on and then you take only the examples that passed and then you train on that. So this is also like called

various things like reinforce fine-tuning or something like that but bas basically it’s a it’s a method for distillation. Um so oh yeah sorry actually the next part is important. So the the one thing that’s

important here is um one popular method for distillation is not just training on the examples but it’s actually m matching the probability distribution from the teacher. And so instead of uh maximizing the log likelihood of the next token, you minimize the kale

divergence from the distribution over next tokens from the teacher. Um, and so it’s kind of like if the teacher says the next token is like a b c with a probability of 0.5 0.3 0.2 instead of

just taking the the one token that you sampled you actually match this distribution here. So this is good. Uh but the problem with this is if this is a roll out from the teacher, the teacher still may be like very successful at getting to this point

despite the fact that the model you’re trying to train is not successful has not been successful at getting to this point. So it like might fail to even get this far. And that leads to a problem uh that Daniel mentioned last uh last lecture. Does anyone remember the name

of this problem? Begins with an e Okay, it’s uh exposure bias. Um and so exposure bias is basically like you’re you’re never seeing the examples. um you’re never seeing the examples that you would actually see at test time. And

so a way to solve this is uh on policy distillation. And on policy distillation, essentially what it does is um you do a student roll out. Um you do a roll out with the weaker model and then um you take the next action from the student model. uh you take the next

action from this roll out of the student model but then you train the student model to match the teacher’s probability distribution. So the trajectory history is from the student model but the next target is from the teacher model essentially.

Yeah. Okay. And sorry I uh I wrote this on the board already but basically it’s saying the same thing. you you’re maximizing the um uh the probability between them. And then um there’s also a paper um on policy

self-distillation. And the basic idea here is that the student model is the model you’re trying to train and the teacher model is the model that you’re trying to train but you tell it the answer. Um and so essentially it already knows the answer

in advance uh so it’s able to pick next actions more uh more appropriately. So I think this is an interesting problem uh an interesting method uh but it is a little bit biased because of course you know the actions that a teacher is going to take if it already knows the answer

are going to be different uh than if it didn’t know the answer already. So um there there’s probably ways that this could be further improved as well. Cool. So, that’s all the stuff that I wanted to talk about today. Um, essentially to break it down, um, I

talked about methods to get better and more useful feedback, including more useful feedback on each token or each action. I talked about stable updates, so how to prevent your uh, reward from crashing uh, a little while into the training. And I talked about how to get

more reliable rewards. So make sure that you’re getting signal and not getting all examples where the model cannot pass or all examples where the model can pass. So are there any questions? Yeah. >> Talking about the actual training procedure. Do you follow the same

guidelines as for SF? So only actual tool calls each token this kind of stuff. Do we uh what do you mean by each token? Sorry. >> So you calculate the signal whatever token tools

>> yal how does it translate? >> Yeah great great question. So um how do you how do you calculate the loss? Do you only calculate it on tool calls or do you calculate it on like the

observations as well? Um the answer there is interesting which is typically you only calculated on the tool call. So you only on the actions and so that includes the chains of thought and you know all the stuff that the model produces but there’s actually a paper

called echo um it’s something like echo uh terminal agents learn a world model for free and basically the only thing they do is they also train the model on the observations and they demonstrate that this improves performance and so the thinking behind this is that

actually predicting the state of the world is also an ability that you need the like the language model to have. Like if it can predict the state of the world, it should also be better at picking the next action. And so if you’re not training on the observations,

you’re essentially throwing away this multitask learning training signal that you could use to train the model better. So um long long story short, you can do both. And it might be an empirical question about which is better, but if you have a really rich environment, then

that could be a useful training signal. Yeah. Great. Any any others?