上下文工程：清桌面、进柜子、上台账

Okay. Uh hello everyone. Can you hear me in the back? All good. Okay. So, uh well, welcome back everyone. Am I uh echoing? All good. Okay. So, um, today I’d like to talk about context management for long context agents. And

I at first thought about titling this as like context management for long context LLMs because most of the stuff I’m going to be talking about today is kind of more general LLM stuff, but long context is super super important for when we’re working on agents and it’s not

necessarily covered uh, very well in all of the LLM prerequisite classes. So, I’m going to cover it in a fair amount of detail today. I’m also going to try to go through kind of the long context architectures that are used in many of the kind of state-of-the-art open uh

models so you can kind of see the best practices that everybody is using in order to handle a very long context. So first I’d like to talk a little bit about um how long an agent context can grow. So basically the input is quadratic uh

from the uh or the number of token accesses that you do is quadratic uh based on the linear history growth. So um if each call uh you make for an agent takes about you know five tokens and let’s say you start out with a an initial system prompt of one token uh

you’ll get six tokens in the first call 11 16 uh 21 and 26. you know, this will increase and increase and increase. Um, and so by the time you have gotten to the end, you will, if you do things completely naively, you will have had to process ADK tokens uh for these five

calls. And you know, this can continue for a very long time. I I don’t know the longest anybody has used a single coding agent session or something like that, but I’ve certainly done them for like multiple days uh where I’ve been working with it for almost the entire day. So

you could imagine this could get uh very very large. Um the actual computational complexity of this is uh is actually even larger than this. Um well no sorry. So yeah the the computational complexity is is quadratic here.

So why do agent prompts get so long uh so quickly? I did a little bit of data analysis on uh the composition of prompts that I used in um uh 1,500 sessions from our uh coding agent openhands. And each each session was relatively short.

These are not like the long multi-day sessions that you do. This is more like solving a single task. But uh each one used about 78,000 tokens on average. Um 23% of this was a system prompt in all of the tool descriptions. Uh this is relatively large for a coding agent. I

think our system prompt is somewhere on the order of like 15,000 to 18,000 tokens because we’re very opinionated. Uh the open ants agent is very opinionated about how people uh do work. Um these tend to vary from like 8,000 to uh 8,000 to 20,000 tokens depending on

you know how many tools you uh you pull in and stuff like this. Um separately from this uh 9% were user messages. So this is the user either saying the initial task or following up with uh additional information after that. 9% were reasoning traces um and you know

where the model is thinking through things. 2% were model replies. Sorry, this isn’t showing on the slide, but 2% were um the model replying directly to the user. So, it’s spending about um five times as many tokens freezing as it is spending communicating uh with text

to the user. And 20% were tool calls. Uh so, you know, uh you know, writing code, uh reading code, that sort of thing. And 37% were tool results. So, this is uh you know, getting the result of, you know, reading a file or something like this.

So you know each step can uh will consist of all of these and that tends to add up uh very quickly. So when you’re handling this very long context uh there’s two big challenges here. Uh the first one is capacity and so whether your model is able to handle

this large amount of uh evidence in the first place and this is a challenge uh because as I’ll talk about in a little bit training models to handle very long context is hard um and so in order to manage this you need uh things like architecture

position encoding training data uh and uh you also need to be able to evaluate whether you’re doing a good job of this. And then the second thing is efficiency uh and whether the system can afford to serve it. And there’s a bunch of techniques to do that as well. And I I

think some people might be taking this for granted if you’re just using, you know, cloud code or codeex or something like this, but they’re doing some pretty monumental feats of engineering to take a, you know, multi- trillion parameter model and serve it within seconds uh to

uh over and over and over again for mil I don’t know codeex has 25 million users. So you can imagine uh 25 million daily users hitting it every few seconds for this huge model. Uh if you want to you can calculate in your head how many flops per uh per second that might be.

It’s a lot of flops. Cool. Um so any questions or things? Probably pretty straightforward I guess. Okay. So uh let’s go into inference. And so in order to make this happen uh the you basically need to be able to um calculate uh prefill and decoding. So uh

calculate your inputs and your outputs at each step. And this uh you know I’m sure everybody has done in your previous classes uh for generating from uh you know a language model. some things you may or may not have thought about in your previous classes. If you took how

many people took like uh language model systems or deep learning systems or something like that actually more than I thought. So if you took the systems course you you probably handled this. Um but other things you need to think about are how do you schedule uh multiple

concurrent queries? Um and also how do you save the results in the KV cache uh so you can reuse them and not have to reuse them over and over again. uh not have to calculate over and over again. So going into a bit more detail here, um we have a few phrase uh phases in

inference. The first one is called prefill. Um you could also call it encoding. Um but I think pre prefill is kind of the like preferred term of art nowadays. And so you take in all of your input tokens uh that you don’t need to generate and uh you generate the key and

value pairs uh from each of them. And then uh when you’re decoding you basically uh take the prompt uh and key value pairs um generate an output and append uh the key value pair that you get from the output. So um you do this kind of incrementally whereas this can

be done in a batched sequence. And so uh when we think about this as a serving system, there’s a bunch of ways we quantify how good a serving system is. Uh the first one is time to first token. So this is how long you need to start um you need to wait between when

you start uh entering a query and when you start seeing the first output. And so like I’m sure everybody you know who uses chat GPT or something like that uh you put in a long output you notice it takes longer to start getting a response than if you

put in sorry put in a long input you see it takes longer to start getting a response than if you put in a short input and that’s like your TTFT is higher. Uh then uh your time per output token is the uh time like the amount of time it takes uh between generating each

token. And if you go to a website like open router um t typically uh here this will be your time per output token there I am pretty sure they’re not counting prefill here and it’s just like the time between uh

each of the tokens that might be a little bit small with the black background but you can see that um for GLM 5.2 to uh you have like 63, 48, uh 24, uh that sort of thing. And you know, some providers are drastically better than others. Like this provider,

not to pick on Mistl, but they’re getting, you know, three three tokens per second and then these other ones like Parasel and Coreweave are getting, you know, hundreds of tokens per second. So um so basically uh you know this makes a huge difference in your

experience using um uh using agents. Uh one other interesting thing is um there’s different ways people can use agents. And so in some cases you might really hear about this. So like let’s say you’re working together um with a an agent on a interactive task in your

favorite coding CLI. you may really really care about, you know, TTFT and Tapot. Um, if you’re running an agent in the background to do some background tasks that run overnight, you might not care at all. Um, and there’s trade-offs between them, between like throughput

and cost and how fast uh you can get the the tokens. And like I you know also um serve coding agents uh with my company and when I talk to providers they’ll often often ask which do you care about more so we can optimize the system uh for the one you care about more and so

um there’s uh there’s ways you can do that and I’ll I’ll talk about those too. Cool. Um yeah so sorry just to finish the slide. So throughput is um the completed tokens or requests divided by the wall clock time. So this is kind of seen

usually on a more holistic level over the whole system. And then uh the cost is you know how much it costs. So as a refresher um I I hopefully everybody should know this already. Um but you have uh the uh query key and value calculation in uh

typical attention and this is uh so you have this here and then you take the softmax um divide it by this. You might have some sort of uh you know positional uh bias and then you calculate your output and typical full attention uh looks uh like this. you mask all the

previous tokens and and or you consider all the current tokens and you mask all the um the future tokens. Um so this is of course very expensive because it’s quadratic in the sequence length. Uh and so a lot of the stuff I’m going to be talking about today is about

how to fix that. Um so this is on the efficiency side and then on the effectiveness side just because you have a model or just because you have a serving system that can take in you know a million tokens doesn’t necessarily mean that you can use that a

million tokens effectively and um this is a very like old school example of this I guess this is a needle in the haststack evaluation how many people have seen this figure for maybe some but not not that many. Okay, so to explain what this looks like,

basically what this is saying is um it’s a very weird evaluation. Basically, they um they took a whole bunch of Paul Graham essays uh by the like famous entrepreneur and head of Y Combinator Paul Graham and they stuck in uh something about like the best place in

San Francisco to eat a sandwich is something or other and then they ask a question where’s the best place in San Francisco to eat a sandwich and then the language model has to answer that correctly. um and they made the context longer and longer and longer and then

they evaluate uh models with the longer context and eventually the weaker models start getting worse at these very very long contexts and there’s a bunch of examples of things like this. Um ruler and helmet are kind of the the common benchmarks that people use here. Um but

basically uh the long the summary is that if you have very long context your models might start getting worse at processing them. So has anybody noticed this when you’ve used a coding agent for a long time like are there common

or used any agent for a long time are there common like failure cases that you see? What what kind of things what kind of things have bothered you? Anybody? >> Yeah.

>> Forget the previous context. >> Yeah. Forgets the previous context. Did that cause any like problems for you? with me telling it not to do something in the middle of the conversation and then it goes and does it. Um there there was a really bad example uh where I

think actually Daniel talked about it here where the person had said do not delete this and then it went and deleted it anyway because it had forgotten the context. So yeah, this can be a really big problem in agentic settings because you know the user is giving these

incremental instructions. So so I’m going to be talking about several different things. I’m going to be talking about architecture. I’m going to be talking about training inference and compaction. So going first into architecture. Um this is both about

efficiency and accuracy. And so as I mentioned standard attention is global. So it makes pairwise comparisons across uh all of the tokens here. And um the problem with this is uh like this is quadratic obviously. And does anyone know the like standard

length of the of models nowadays of these uh of like state-of-the-art language models? >> 256 to a million. >> 256 to a million. Yeah. And I I’d say most people are aiming for a million now because they don’t want to be the, you

know, the model that has 256 when everyone else has a million. So very quick math in your head, how many pairwise comparisons is that? >> 10 12 >> 10 10^ the 12 or in English. >> A trillion. Yeah, that’s a trillion.

That’s a trillion comparisons every time. uh like when you’re uh generating a sequence of a million long and each comparison uh requires you to multiply large vectors by each other over and over and over again. So you’re starting to get up into the you know 10 to the

I guess it’d probably be 10 to the 20ish uh which is a lot that that’s a lot of compute um for a single sequence. So um there there’s ways that people solve this and basically almost all of them fall into the category of taking this quadratic computation and putting

some sort of constant limit on the number of comparisons that are made uh at each time step. And that brings your uh n squared down to owl where I I wrote w for window but it doesn’t necessarily need to be a window. It just needs to be the cap on the number of comparisons

that you’re making. Um so there’s the idea of local models and global models. And so local models are are basically models where this cap on computation happens within a local window. And then global models you have a cap on you might or might not have a

cap on computation but that cap on computation is not localized. So you can pass information uh throughout the entire sequence. And the I’d say almost all of the models uh that exist nowadays or are widely popular nowadays are some variety of hybrid model where they have

many local computations and then one global computation then many local computations one global computation and specifically I’m going to talk about six uh models that we have here. So uh the most recent Quen model uh has uh local computation of uh gated delta net.

I’ll explain all of these uh or most of these in a bit. And then um sparse retrieval uh for the global computation and the ratio is 3:1. Uh GLM 5.3 is uh also a variety of delta um and sparse attention uh with 3:1. Um

uh Kim K3 is uh a variety of Delta and um this is a dense uh dense attention 3:1. Uh Neimotron is Mamba uh which is uh kind of like recurrent style model and uh dense attention 4:1. um to uh window uh sliding window

attention uh dense attention 5 to one and deepseek’s a little bit different. It kind of has uh window windowed attention but then some uh sparsity uh sparse stuff with interle branches. So um basically the the standard way of doing things is hybrid model uh with

some sort of local thing and uh either sparse or dense uh global attention. So I’m I’m going to be talking about this. I’m not going to go into like a ton of details for all of them, but uh it should be enough that you could be able to look them up later.

So the first one is uh sliding window attention. And this is maybe the the simplest conceptually. And what you do is you take chunks or uh you take chunks of the output and basically you’re able to attend to your current chunk and um all of the previous chunks and uh

or not all of the previous chunks, a fixed number of previous chunks. So if this is your current chunk, um it might be a chunk of like 512 tokens or something like this. You would have an additional 512 and 512 tokens in the previous chunks and you would attend to

those. Um so pretty simple, no changes to attention, but you have something like this. But the nice thing about this is on the first layer you’re passing information from the previous one or two chunks, but on the next layer the uh previous chunks also have information

from their previous chunks. And so you kind of get this gradually expanding window. Um so uh this uh this is pretty widely used. Um but the more popular way of doing things nowadays is using some variety of

linear attention. And so the way uh linear attention works is basically we take our uh standard attention and standard attention has this um softmax operation included in it. And this is the reason why it’s hard to compute because you’re taking the

exponent and um and dividing. And because you do this there’s you basically have no choice but to expand all of the uh key and query computations uh like key query multiplication computations in and be uh quadratic uh quadratic

complexity. What linear attention does is you basically start from this uh kind of standard attention formulation but you take you remove the softmax and the normalization here or I mean the normalization doesn’t really matter but

you remove the softmax and you get just the key query multiplication and [clears throat] this is definitely less expressive because basically what you’re doing is you’re moving from something that

takes in a bunch of uh like numbers. Anyway, like pretend pretend that this bottom one adds up to one, but you’re going from a thing that takes a bunch of real numbers and normalizes them into probability like numbers that are between zero and one and add to one. Um,

so this nonlinearity allows you to focus on like only some of the previous uh like tokens. And this is kind of the good point of attention. That’s the selling point of attention that allows you to focus on the relevant previous context. And you’re basically taking

away that ability, right? You’re you’re removing the relative comparisons and just getting um the the dotproduct between the queries and keys. But it allows you to do something really uh nice which is that you can use uh the associative

uh you can use associivity and basically rearrange so that um you can convert the sum of all the key and value uh multiplications before into a single uh state matrix. And so this state matrix is essentially a function that map it’s a linear

function that maps from a query to a value. So attention is also a linear value. So if I go back to this here you’re basically calculating you’re taking the query you’re calculating the attention over all the keys you’re multiplying it by the values and you’re

getting a value here. Um and this state matrix is also that. But the nice part about it is this state matrix can be decomposed. So you can update it. You can keep this state matrix around and update it in constant time every time you get a new value. So this essentially

becomes like a recurrent neural network. So it it allows you to update the state matrix um in linear time without having uh without requiring the quadratic computation. So this basically solves your quadratic

problem and it solves it so much that um O of N squ becomes O of N. There’s no like there’s no even W term in this because uh each time you’re uh you’re updating based entirely on the previous state. Um but there’s there’s caveats here. So um is this clear though

uh so far? Okay, cool. Um, so the big caveat is exactly the caveat I mentioned before. Like this is just a less expressive function. Um, and it doesn’t allow you to make like relative comparisons between the

previous tokens, that sort of thing. So there’s a bunch of uh workarounds around this to try to make a more expressive function that nonetheless is useful in um in kind of uh doing attention-like calculations. And so um most of these are

if you dig deep into the math in them, they’re basically trying to become better predictors of the final value that you would get from attention. Um and you you can dig into the papers to see how that works, but I’m going to explain them mostly uh mechanistically

here. And so um the first thing is Delta. And the idea behind deltaNet is um you can predict what you can essentially predict what value you would want to be getting from the uh from the calculation of uh KT um and S. And so you know KT here is uh

the key at time step t and you are comparing it with the matrix at time step t uh t minus one and uh you’re calculating this v hat t and on the other hand you um so basically this this is the prediction that you would get of kt But on the other hand,

you know what the uh output VT should be uh based on um what the query times uh or based on uh what the Hang on one second. [laughter] I don’t want to I don’t want to get this wrong. So you calculate KT with respect to the

matrix. And you can view this matrix as a way of uh calculating the uh the way of calculating the mapping from a query to a value. And you can also get the the actual value and subtract out uh what the value you get when multiplying it from the previous uh together with the

previous matrix is. And this is essentially the error uh that you get here. And then you want to reduce the error. And so in order to reduce this error, what you do is you um essentially add in this uh this term which is KT uh times the error term. And after you do

this, if you multiply in KT to the new matrix, you will no longer have any error. But you don’t want to make an update that is too big. So you add this beta term. This is essentially a learning, it’s similar to a learning rate. And this learning rate will make

the updates uh smaller. And so uh if you if you do this, this allows you to basically nudge the matrix in the direction of uh the in the direction that will cause a query that looks like the current key to output the value that you want essentially. So it’s it’s

reducing the error uh reducing the error in the prediction of the value that you want. So um as I said this is going to be a little bit high level. Um the the papers are are pretty good uh descriptions of Um so after this um this basically what

this is doing is this is pushing you in the direction of the uh of making the the matrix a better predictor of the uh the value. But one problem with this is you can build up lots and lots of context uh from the previous uh spots and linear attention doesn’t have any

mechanism to prevent very far away cont uh context from having a bad influence on your next predictions. And so if you have like a very weird key far back in the future, it might have a very big influence on you in the f uh in the future. And so the way they fix this is

basically they have a decay term to make uh to make the matrix decay a little bit at every time step. And by decaying the matrix a little bit at every time step, they’re basically discounting the the far in the past

context. And so you multiply um the matrix by alpha t and then you add this uh st tilda into uh into your calculation here and update the matrix. And so the stronger you make this decay basically the the more quickly uh it will forget the previous context. And so

you can make this more local or or less local. And so this is uh gated delta is used in the Quen models. So it’s one of the more uh the more popular methods now. Um but both uh GLM and uh KI which are kind of two of the state-of-the-art models use a

slightly more expressive version of this uh where they basically have a per element uh decay term. So each element can decay at a different speed. And uh because each element decays at a different speed, you can have some uh portions of the context that uh you know

carry over very long time periods, some portions of the context that carry over very short time periods. Cool. So hopefully this was more or less uh more or less legible. Um any any questions here? Long long story short, it’s basically a

way to turn uh into um it’s a way to turn the idea of attention to something that you can calculate um in context. Mamba is uh something that’s very similar um but it’s a little bit more mathematically involved um uh and I I

decided to skip over it. uh I covered it more in the uh language model uh description course but similar idea but just a bit more complex uh in the math. So the next thing I’d like to talk about is sparse attention. And so sparse attention is uh basically looking up a

not attending to everything in the previous context but only attending to some things in the previous context. And essentially you limit the number of things that you look up in the previous context. There’s a few ways you can do it. Um originally in there was a sparse

attention paper by OpenAI uh quite a while ago like 5 years ago or something like this. Um and basically uh what they did was they had sparse attention every you know uh 512 tokens or something like this. And what this does is it means that you still have a fixed it’s still

quadratic in the sequence length but it’s 512 times less than if you attended to every token and that’s like good enough uh that you can manage to calculate it over the context that you’re interested in. Another option is to do um content

dependent sparse attention. And basically what this does is you have a very cheap way of calculating attention. Like for example, you can down project the attention vectors to be very very small and then you do attention over these very very small uh attention

vectors. You pick the top k ones and then you do full attention over the top k. So uh that’s uh another way that you can do this and this is what’s used in deepseek. Yeah. >> How do we define the metric for which we

do the top similarity compared in this in the shrine space we just do like coine distance. >> Yeah. I I think it’s typically not cosine but it’s just product but yeah >> I I I think although maybe maybe they’re normalizing beforehand. They actually

don’t know that detail. Any other questions? So basically um these th this is a very very cheap way to achieve global attention because you can just uh write a kernel that will do this for you. Um this over here will uh

require you to have two passes, but you know it’s potentially uh more accurate. So um I I’ve seen both of these used pretty widely. So a final um a final thing is KV compression. How many people saw this in a language modeling course?

Maybe some people. Okay. Um so I will uh I’ll explain this. So basically um this is standard multi head attention. And so in standard multi-head attention, you have a bunch of heads and each one has values and keys and all of these are a certain dimension. So they’re like

dimension, let’s say they’re dimension D and you have uh you have like a certain number of heads. Let’s say that’s H. Um so the two main ways that people solve uh or help uh this problem of having very uh large and expensive KV caches is uh

grouped query attention. And grouped query attention basically uh just uses a smaller number than h um you usually h divided by some uh power of two uh number of uh keys and values. And because it’s the KV cache, it’s not the KVQ cache. Um

if you reduce the number of uh keys and values that are used, you reduce the amount of memory that you need to hold the KV cache. This is really important because if it gets very large then um it limits the number you can fit on one GPU and so this can usually be reduced by

like four or by eight or something like that. Um multiquery attention is just like grouped query attention with a single one. So you can kind of almost skip over that. You already know what it means. Um multi head latent attention the the um

is essentially down projecting uh the size of the keys and values. So it’s not reducing the number of heads, it’s making each head smaller. And so this is similar to what I talked about with the top K attention. Um and of course you can combine both of these together as

well. You could have group query attention with um you know smaller uh keys of values. Um so the these are the method the architectural methods that people use to handle very long contexts and with these um you can handle up to a million uh

tokens. So like this is uh you know this makes the efficiency possible. The next thing is uh training long context models. And this is actually uh pretty hard. And the reason why is because um if we want something to handle 128k tokens or even more, there’s

actually not that much data on the internet that is 128k coherent tokens. Um it’s actually a lot of a lot of data. it’s more than most Wikipedia articles, you know, um you’re starting to get in the in the order of like having a book uh that you need to have and there’s not

that many books on the internet. So, typically what you do is you start out with like short pre-training. Um you do more continued training with longer documents and then you do long adaptation. Um I wrote packed sequences here. Um,

but packed sequences are are useful can be useful for efficiency, but they’re not super useful for learning long contexts because um every time you like let’s say you pack 10 documents into a sequence, you’re not learning to attend across the documents and in fact you

probably should not be. So um you really need coherent uh long sets of data to do adaptation to very long contexts. Um so I’m going to talk about a few things that we do here. Um the first thing I’m going to talk about is uh handling positional encodings. Um so

absolute positional encodings uh I I hope everybody has seen in their language modeling class. Um but basically uh each individual entry in the uh each individual word embedding uh gets a position vector and that position vector

is typically calculated by uh some sort of function like using signs and cosiness and other stuff like this. So hopefully you know most people have seen this. So I’ll I’ll kind of skip over that. Also maybe most people have seen rope. Are people generally familiar with

rope? Okay, I see a fair number of people. I see actually less people uh than I would have expected, but um uh maybe people are just uh not interested in raising your hand. Um [laughter] but uh but basically um rope rope is pretty elegant mathematically in my opinion. Uh

basically the problem they were trying to solve is they wanted to have a positional embedding. um where the positional uh or positional encoding where the positional encoding is entirely determined by the relative distance uh

between two tokens. And the reason why they wanted to do this is they didn’t they didn’t want the absolute position to have an effect on uh they didn’t want the absolute position to have an effect on whether

two tokens would attend to each other. And so this is this has been quite effective and a lot of uh a lot of models use rope. Um the the math I’m not going to go through here because it’s kind of not the main point, but they do some

interesting stuff with rotations and imaginary numbers and stuff like this. Um but there’s also this like super counterintuitive uh nope. which is uh don’t don’t use positional encodings at all. Um and if you took

your uh class on transformers uh you might have heard this scary stuff which is like you need positional encodings for transformers because otherwise something in position one in position five will have exactly the same representation. So if you have like this

this is a cat but this is not a cat it will attend to both of the cats the same amount or something like this. Um theoretically when the transformers were originally created um this might have been true but now actually it’s it’s not true anymore based on the transformers

we use now. Um anybody have an idea why this is the case? I’ve kind of given a hint in the slide but you might also need to know a little bit of the history of transformers uh which is uh before we were using encoder decoder models uh where the encoders

encoded all of the input and then the decoders uh generated the output any idea >> because we’re doing causal masking exactly so um to elaborate on that a little bit if I go back to our transformer equation

Um, we have the keys um, and the queries. And if we look at the queries, even if you have cat at position two and cat at position five, the cat at position two is only attending to the previous token where the cat at position five is attending to the previous four

tokens. Um, and so both of them are like actually different. um starting at least in the first layer of attention. And so you can tell like even if they are exactly the same word, the embeddings starting in the first layer of attention will will not be different. And there’s

kind of uh this counterintuitive paper that demonstrates that you don’t need uh positional encodings at all. And this is especially true um this is in auto reggressive uh transformers. In non-auto reggressive transformers where you’re attending all the tokens are attending

to each other. You still do need positional encodings. Um this is especially true when we start talking about like gated delta because gated delta has like the decay term and uh other things like this which causes each uh token to be different. So once

we start having those sort of local biases, we actually don’t need positional uh encodings anymore. and if you uh if you look um we uh the first two here have rope uh or at least partial rope. The second two

have nope uh no positional attend encodings. Um the the fourth one uh relies entirely on implicit order and mamba and some of them in uh and inkling has relative positional encodings. So you can see that it’s kind of like half split between having relative positional

encodings and not at all. But like let’s say we did have positional encodings. Uh one thing you need to worry about for long context is uh position interpolation or position extension. And what this means is if you have a pre-training range, um, you need to

essentially extend your model so that it’s able to handle longer sequences. And so the the promise of rope was that rope relied only on the relative position uh between the different tokens in the output. But that that’s fine as long as the relative

position is only um well okay there there’s two reasons why this is not fine. So number one if you’re training on sequences of length L and then you extend to sequences uh of length SL then it’s never seen two tokens attend to

each other over a distance of more than L. So the tokens that are more than L away from each other are going to have problems attending to each other and you’re not going to be able to uh manage that very well. Um the other problem is that actually relative positional

encodings are only one part of the story and like actually the auto reggressive mask has an effect in all these other things. So you can’t just take rope and expect it to work uh really well on longer sequences. And so the typical way you fix this is

there’s a parameter in rope that basically uh let me see if I have it in my slide. So I I don’t I don’t really have it in my slide, but there’s basically a um a parameter in rope theta that uh tells

you how much you rotate at each time step. and you divide this theta by the uh multiplier by which you extended the sequence. And so basically you’re stretching the rope position so that the rope uh position at position zero would um or the rope position at position like

five would be 5s uh would be equivalent to the position 5s in your uh extended context. And there’s also um a more sophisticated way of doing this called yarn uh which basically is uh similar in spirit uh to like the delta uh the ki

delta attention um which basically said that we want some of them to be very long distance and some of them to not be very long distance. In yarn essentially you have some of them be uh extended more and some of them be

extended less and so you don’t um you don’t necessarily interpolate all of them at the same frequency. So um yeah in the interest of time I’m not going to go into like a lot of detail about this but basically these are um uh these are the various methods

that you can use here. So, um, I’d like to talk a little bit now, um, taking a step back from the detailed modeling stuff and going into the data that you can use to train this. So, um, as I mentioned before, packed examples,

you can just pack a whole bunch of stuff in there, um, a whole bunch of documents in into your context length. And this will work at training models to be able to process things near the end of your context window. But what it won’t work for is the kind of global coherence. And

um so there’s a number of different ways that people do this. One way is just to find um very long documents uh like books or something like this. There aren’t a lot of these uh but there are books. There are code bases and code bases tend to be coherent. So you can

just download a big codebase and that will definitely have more tokens. Um multi-document corpora of some kind. So, you know, like uh you can get a whole bunch of documents that are on the same topic and then at least you’ll have topical coherence between them. Um I’ve

also seen people train on patents because patents have like there’s lots of public data for them and you can kind of link link them together. A big source nowadays is agent trajectories because you know agents can make very very long trajectories if you

just interact with them for a little while. So that’s another one. And then you can also create synthetic tasks and a lot of the evaluations are synthetic tasks. So you can do that as well. Um and I I have some examples of how the models do this. Um a lot of the model

training companies don’t like to share these details with you because this is like if they release their um well so number one the closed models like anthropic open AAI you know those places uh uh Gemini they will not release even their architectural details the open

model companies have to release their architectural details otherwise you can’t run their model so what do they keep secret they keep their data mixes and their their training strategy secret so um this there’s a lot less detailed stuff that we can say. Um, but there’s

uh all of them explain all of the ones in here explain a little bit about their data mixtures and they are things like uh cleaned up sampled documents and video um documents, repository files, issue, PRs, uh long document QA, scientific papers and technical reports.

And then uh they also create a bunch of synthetic tasks. And typically they have a scaling strategy of like 8 64 256 1 million uh that that sort of stuff as well. Oh yes. Um [clears throat] one other

logistical detail that you might need to deal with um or might not need to deal with depending on the size of the model you’re training. like for example for your project if you want to start dealing with um uh larger models is context parallelism and basically what

this means is there’s a bunch of uh ways to do parallelism when training models I wrote context parallelism up here what are two other ways uh that you can parallelize models during model training or serving >> yes

>> data parallelism >> yeah you you said a more specific example of the second one but data parallelism and I would say model parallelism in general and model parallelism can have lots of varieties. It can have uh expert parallelism where

you have different experts on different machines uh tensor more generally tensor parallelism where you split um uh the uh the parameters and you can also have pipeline parallelism where you have different layers on different uh different devices. But context

parallelism starts becoming import more important for agents because when you train agents you want to train them on very long contexts. And so uh the way this works is you will compute like one part of the context uh the keys and values. So you send it to the next

device calculate the keys and values for the next for that part send it to the next device uh and etc etc. So um this is uh very useful. It’s really important if you start getting above like 32k 32k context or so. Um, but it’s also a pain because

you have the you have all the modeling people saying I want my I want my KI delta attention and then you have all the info people saying but wait I need to implement a context parallel kernel for KI delta attention and make sure it actually

works and you can back prop through it and all of these other things. So um I I personally had an experience where I was trying to train uh Quen uh for uh like a smaller quen model like 135 uh 35b and uh I I wanted to go do use context parallelism on it and it’s like there’s

no kernel for gated delta net anywhere on the internet and so um you need to uh you need to either go in and implement that kernel or decide to do something else. So um th those two can conflict some of the times. So be be aware I guess

um the the good news is for like standard attention and sliding window attention and stuff like that all of those will have uh kernels more easily implemented. The hard parts are where you’re like not doing a standard attention operation.

Okay. So the next part I’d like to talk about is prompt and KV caching. This is uh very important even if you’re not the person who’s implementing the language model, but also if you’re just a person who’s um like running agents uh yourself, but I’m

going to talk about uh the the technical details of how it’s implemented. So, you know, at least you you have a good uh idea about how that works. So um when we I I simplified this a little bit but when we’re generating an agent call we have an input uh we have an output uh

so this could be like the the messages and tool calls and then we have the tool result and so so as we calculate the input we do prefill and as we calculate the output uh we do uh basically decoding Right? And so during prefill and decoding, we have to

calculate the KV cache. This is all done in a batch. Uh this is done every time we generate a token. But after we have that, we have the uh key and value uh for each of the positions up until the end of the output. And so what we can do is we can throw that in a cache and then

we don’t need to calculate it anymore. The tool result is new uh because you know we don’t calculate over the tool result. So this becomes a new input and we generate an output and we get another tool result. And so in step two we need to calculate the KV cache over the new

input and we uh during prefill and the output but we can reuse the cache and then we do that over and over and over again. And so as you imagine um as you can imagine I just showed steps 1 2 3 but by the time we get to step 50 if we haven’t done anything tricky with our

output this is going to be 50 times larger than the new input in the in the output right so that’s pretty significant savings if we can cache this properly and here are some examples from actual papers uh that you know examine this and

basically What we can see is this is like the the token length on uh different GPUs and this is the amount of uh of compute that you need with caching and no caching. And so the upper one of the same color is uh or the lower one with the same color is caching and the

the upper one is no caching. So uh basically as the sequence gets longer the non-cashed one becomes quadratic and the cache one is is mostly linear. This also shows up in how much you pay to run an agent. And so I I pulled all of these numbers from uh the official

pricing. And basically you can see here that if you go through all of these uh all of these numbers including the commercial ones typically cashed tokens are onetenth of the price of uncashed tokens. And so what this means is you need to be very aware of uh whether your

tokens are being cached or not. Uh because if they’re not being cached uh you’re going to be paying a lot of money to be running your agents. Um, so does anyone have an idea of what a good uh cash hit rate is if you’re running an agent? Yeah.

>> 80 to 90%. >> Um, yeah. So I I heard 80 to 90% I heard like more than 95%. I my my impression is 90 to 95 is everything is fine. Like I’m I’m good with 90 to 95. If it’s less than that, you might be fine, but you might be might be losing out somewhere.

Um so, uh yeah, so basically you can you can cache most of your tokens. Um one other interesting thing is the output also tends to be five times more than the input. So um this is uh there’s a 50x difference between output and cached input which just reflects the amount of

uh uh the amount of cost uh savings that you can get from serving it. So why why is caching um so much cheaper is maybe kind of obvious. You just need to look up um you need to look up the outputs. uh you just need to look these up instead of like actually uh

performing the calculation over them. Uh especially the feed forward network calculation is pretty expensive in transformers and so being able to skip that is is uh particularly a win. Um but how do we actually manage that? So um the first thing is uh how do you hold

it in memory? And there’s two like most popular open-source uh two most popular open-source LLM serving libraries VLM and SGLANG and both of them are based on an innovation around the how you manage KV caching. Um so VLM’s main technical in innovation

and like of course they have a lot more stuff in there now. They’re like general purpose serving libraries, but like the main technical innovation that led to VLM was something called page detention. And basically the way it works is you have physical GPU memory blocks and you

rear you arrange them like kind of memory blocks or or disk blocks on uh like more typical uh like CPUbased machines. And so you might have uh block zero uh block one block two that you actually need to calculate and they put that you know in like block one and then

block seven and then block three or sorry uh block seven block one block three and then you have an index to each one of these uh as you’re generating. And the reason why they wanted to do this is they want to be able to generate in parallel many sequences in parallel

and when you finish generating um you can like evict that from your memory and so you can uh evict whole blocks at once and this makes the calculation more more efficient. So um this is uh one technical trick that they used to make this uh you know make parallel serving

work. the technical innovation that kind of led to SG lang in particular uh was something called RAIX attention and this is specifically focused around uh like re reusing already cached uh out uh inputs. And so what they do is they form

a a tree basically uh it’s a radics tree and they use this to look up prefixes that have been used already. And so this uh when it first came out this was specifically focused around uh chat bots like chatgpt uh because you know you can have

somebody start a conversation and then continue the conversation and so you’ll be caching some portion of the already done conversation. But for uh now it’s like much much more important for agents because agent conversations are much longer and much more frequent. So we

need this uh looking up more frequently. Um so another uh another thing that you need to think about um which is handled in uh newer versions of sglang for instance is let’s say you’re doing multi-achine serving. So you’re serving on um you know different machines uh you

know maybe you have a data center with like 64 nodes or 128 nodes or something like this. um every time a request comes in, you want to route it to the machine that has your KV cache that has the best KV cache for you um to be accessing and so that you actually have cache aware

routing uh route it to the appropriate place and then have the worker with the best uh KV cache match uh process set. I love getting all my uh my serving statistics from Open Router um because they have so much data and uh they cover all of the things I want to talk about,

but I believe they even have a ranking of providers by cash hit rate. [laughter] I’m sorry mist the data the data doesn’t lie but um you can see uh mistrol has the worst cache hit rate and then um you have other one other ones like Alibaba

cloud or dicard or silicon flow that have higher cash hit rates um this is not entirely reflective of the provider being bad or something like that it could be just the workloads that they’re running but there’s a number of things that like actually factor into

into this cache hit rate and um one of them one of them is how good is the cash routing. The other one is how fast is the cash eviction. Um so the providers will not hold your cash for you forever. Um they will uh you know evict your cash after a certain amount of time. Does

anyone know how long this is for like popular providers? Have you ever taken a look at that >> yet? Close to an hour. >> Close to an hour. I I think the long like I’m not I’m not like super up to

date on this, but I know anthropic used to be five minutes. And so it’s basically if you have a an active conversation going on with them and you don’t pause your conversation, you’ll stay in the cache. Otherwise, if you like stop the conversation, you get

evicted from the cache. I think you at least have an option to go up to an hour now. Um but I don’t know if you um but like it costs a little bit more if you go up to an hour or something. Um but yeah, ba basically they don’t want to keep this around because there this is a

huge amount of memory uh that they’re keeping in memory and so they want to use it if it’s working and and evict it if it’s not working. So um that that’s another policy that could change your cash hit rate. The bad news for us as users is the pricing doesn’t change no

matter how good the provider is at routing or um or cash eviction. So you uh you might be paying just because they evict you uh evict you too quickly which seems a little bit unfair. >> Um yeah >> do they use like cash layering where you

can store it on device in memory that’s like adjusted to device and then on this >> yeah so the that’s a great question. So the question was, do they use a cache layering where you have like a fast cache that’s stored in the device and then uh a slower cache that’s stored on

disk or something like that? I actually don’t know the answer to that question. That’d be a great uh like a great follow-up for us to do. Um the open source libraries also are probably not as sophisticated as the closed libraries because um if you look at all the people

on open router their job is to be really good at serving and so you know they’re some of them might be using open source libraries but they’ll be building stuff on top of them and that’s kind of their you know competitive differentiation. So um like I I think

the internal solutions that inference providers are likely to be you know a bit better than the ones that are available in open source too with respect to that. >> They usually offload to CPU from GPU. >> Yeah. Because your GPU memory is so

precious. So yeah. Yeah. >> Cool. Um so uh yeah. Oh, sorry. I skipped a important part. So, what does this mean? If you’re just designing a harness and you’re not deploying models or or anything like this, it means you need to be super super careful to not

break your cache unintentionally. So, remember that your cache only works if the entire prefix is exactly the same. And so there’s a bunch of things that we’ve thought about uh doing but we decided are a very bad idea because we don’t want to break the

cache. So one thing that’s completely forbidden you should not do is don’t change the system message every time you uh you take a step because that means like no caching essentially. uh you don’t have a single message that you’re able to cache. So one thing one very

logical thing that you might think of is okay I want to append the current time and date to the system message uh so that the agent knows what time it is. Do not do this because the time and date changes every step and so that will break your cache. So if you do need to

append it, append it to the final user message uh or the final uh tool call result that you get. um and uh like changing your um changing your tools uh dynamically in the middle of the conversation. You can do this. That’s okay. Um another thing is a very logical

thing that you might think of doing is routing to a different model every time you uh every time you take a step. So it’s like this step looks like a really good model for you know a really power a really good step for a really powerful model. um like uh like I want to do this

step on Fable and then I want to do the ne next three steps on um on a cheaper model to save money. But the thing is you’re still paying you’re still paying the you know 10x cost for Fable the next time you use uh Fable. So you really need to be very uh

frugal in the amount of switching you’re going to be doing if you want to be doing switching to save cost for instance. Cool. Um, so the final thing I’d like to talk about is context compaction. Um, how

many people know know about this? Use this in your everyday coding agent use. How many people are really excited when it says compacting context? That’s nobody good. So um the the basic idea behind context compaction is uh that you have uh before

and uh this is you know all your observations and actions that you might have had. Um then after what you tend to do is you tend to take your previous context and you crush it down into just the essence of the previous context. Um, and so the reason why people are

like not excited about using this, uh, perhaps obviously is if you do a bad job at this, your agent forgets things. It starts doing something completely different and, uh, maybe deletes your file system. um we can view compaction as essentially

trying to come up with a previous representation where your future action sequence um you take your history, you compact that, you get a working state and you want your future action sequence to be the same or you know similar or possibly better um if uh after

compaction than it was before. Um at the same time you also have a constraint which your state needs to be bounded. So it needs to be small enough that you’re happy to continue work. Um so if we look at what survives uh

compaction um there’s different ways of doing this. I’m going to cover the different ways that people do this, but a common strategy is to take some anchors and these can be the very first um like the very first things the user said when they started the session for

instance. Um, so, uh, one of the answers to the things that I heard agents forget when I asked the question, what do agents forget was sometimes they forget the things at the very beginning of the session? And that actually might be a design bug in

the agent harness that you’re using. Um, because uh, some good agent harnesses uh, like kind of intentionally avoid deleting the stuff at the very beginning of the session because they think it’s more important. Um, but it could also be, you know, impossible to avoid

depending on the situation. Um, also you want the stuff in the middle that’s encoded and put back into the context. And then the recent tail is often kept around. So often people will keep around the the most recent actions. Separately from this, I don’t see this a

whole lot, but um, you can also add the evidence to an evidence store. Actually, no, sorry. I I’ll correct this. Um, the evidence is typically added to an evidence store. Um, and at the very least, if you’re using a CLI coding agent, usually your full history is

saved on disk somewhere. And so, if you know where on disk, your coding agent has saved all of this history. Even if you lose stuff through compaction, you can tell the coding agent to go back and read the history, and it can, you know, uh, read the history for you and find

the the previous results. Um but there there are also uh possibly more structured ways of doing this and so then um you know other things can be discarded. Um so there’s a few ways to decide uh compaction policy. Um the first one is

the trigger and there’s essentially two ways of doing this. Um one is a token threshold. So you limit uh the token threshold. Uh so you can say I’m I’m only going to use up to 200k tokens. Once I go over 200k tokens, I’m I’m going to compact the context.

Another thing is um like when you get forced to do that by the provider. Um so that’s kind of like a hard limit on the number of tokens you can have. Um finally, you can also have a manual request. And so uh a lot of uh kind of coding agent toolkits for instance have

a slashcompact uh like command that you can use to compact the context. Um then based on that uh you’ll select you might protect the beginning and keep the recent tail and compact the older region. Um replace it with a summary. Um

that summary can be a structured summary. So sometimes uh they require a tool call uh that says this and this and this and this must be in the compaction summary. Sometimes it’s just free text. Um there’s also uh I think this is only open AI uh

possibly. But basically they um they compact into something encrypted uh that you cannot read. So you can’t see how they’re compacting because they don’t want you to steal their uh their precious compaction algorithm. Uh so uh so

there’s that. Um and then uh yeah, you can you can keep on on working on that. one of the risks here is uh sometimes you can compact over and over and over again and the first time it works uh but as you walk through more um you uh you

end up losing more and more. So again this is one of the reasons why people don’t like compaction because it can cause the agent to go off the rails. Um so uh ba basically the way we evaluate this is typically in a holistic state. So you can vary the compaction

algorithm and evaluate it against whatever benchmarks you’re evaluating against. I would like to warn you uh about this. I have a a kind of like story from when we were first developing open hands which is um we actually I think made one of the first like

production grade uh compaction algorithms. We called it compression then um and we evaluated it against bbench which is a very common you know benchmark for for coding agents and we worked pretty hard on it like as a research project for about a month or

something like that and we were like we found this great compaction strategy. It doesn’t reduce our score at all. it’s saving us a lot of tokens. And then we actually tried it in our um we tried it in our our system and it made these really silly mistakes like um sending a

pull request every time it compacted. So it forgot that it had sent a pull request to GitHub. And so uh you know we would get five pull requests for the same functionality because it had forgotten. Um or you know it would forget

user instructions that were sent in the middle of the conversation because we had only evaluated on single turn conversations that had a single user message and we were compacting away the multiple user messages. So it’s not just good enough to benchmark uh unless you

have really really robust benchmarks that represent every use case. You kind of need to test them in the wild as Cool. Um so we’re we’re getting close to time. Uh but yeah, basically uh what I covered today is uh you know agent loops grow very quickly. Uh we have a lot of

inference challenges in doing so. uh we can handle uh efficiency problems through architectures and um and caching and uh compaction is kind of the way we handle this uh once we expand beyond the level that we can handle. So um that’s all I have. Uh if there is any questions

happy to happy to cover them and next time we’re going to be talking about skills and memory. So this is like even longer context over your entire uh your entire time. Um, are there any questions before we all uh all leave or >> Yeah, great question. Um, the question

was, do we consider sub agents as a way to manage context? And yes, but we’re going to talk about sub aents later, so I didn’t cover them this time. Yeah.