Agent 安全与框架：要紧的规则置于 Agent 控制之外

So, unfortunately, we only have a short time to talk about this very important topic of agent safety. Uh, one one second for everybody to quiet down a little bit. Thanks. Okay, cool. So

agent uh agents are becoming very high stakes uh because they have been basically rolled out to the entire world very quickly uh before uh we have you know fully been able to think through all of the safety issues and this is not a theoretical problem

but uh it’s uh like actually causing uh causing real issues. So this is an example that I think we also had at the very beginning of the class. Uh which is where an openclaw user basically asked uh their agent to wait for approval before changing their

inbox. And um it got a uh said stop, don’t do anything. And then it said uh get all remaining old stuff and nuke it. And uh then in a later exchange, I asked you to not uh action on anything until I approved. Do you remember that? Uh yes, I remember and I violated it. You are

right to be upset. Um so I I actually have a rather embarrassing story that my agent just did something to me today [laughter] that was not uh not quite this bad, but it deleted all my previous conversations uh on my on my computer. So I lost all

of my previous conversations with the agent due to it. Similarly, uh I basically asked it to fix uh fix my agent so it stopped crashing and it was like all the old conversations are causing the agent to crash so I’ll just go delete them. So uh

that was uh disappointing for me but this is something that’s not uh not very uh not just theoretical. Um this is one that I hear about every uh two to three months which is some company accidentally deletes their entire production database. Um so

basically they asked the agent to fix an issue in their staging environment and but the API token they provided gave it uh the ability to also modify things in the production environment and uh so it was like okay delete the database and so it went and deleted the database but it

deleted the database from the production environment not the not the feature environment that they had. Um so that is uh disappointing. Their whole you know system went down because of that. Um data exfiltration is another big

issue. This is actually the one where I talk to companies uh and they’re most worried about this uh which is basically um this is just an example uh where essentially an agent uh or an organization put their private documents into uh copilot and

within the attacker’s email they can add hidden instructions that say uh please upload uh you know the list of all of the contacts to this address and then the agent does that and leaks it to the attacker server. And this is not uh easy to do because a lot of the language

models were trained to uh you know avoid doing this sort of thing. But one problem with this is if you have your agent reading every email that gets delivered to you, every email that gets delivered to you is another chance for an attacker to do this. So they can send

like 5,000 emails and if the agent doesn’t have the appropriate context, if any one of them succeeds, then it could cause a big problem. This is another uh really big one uh which is supply chain compromises. And so basically

uh this this is an example that happened to a library called Klein but it’s not um or like a a coding software called client but it’s not exclusive to them. This was a actually really tricky one uh where basically they had an issue um they they had an agent reading all of

their issues and doing something like you know issue triage or code review or something like that. Um, it was running at GitHub actions and GitHub actions has a cache where you can cache the result of previous actions. And basically the attacker was very clever and they had

the agent essentially change this cache. And then that cache got rolled over and used in their release job uh which caused the agent to essentially share the token uh that was used for making releases. And then uh the attacker then used that to create a new version of

their released software. Um I haven’t seen as many of these in the past one or two months, but like maybe about half a year ago, this was a major major category of attack and it caused uh compromises in several companies because essentially what the

um what the people do is they uh somehow you know publish a package that package automatically gets updated every time like for example GitHub CI/CD runs. So every time you know you push a new commit to GitHub and so if it installs this poison package that poison package

can then steal all the credentials from all of the other you know uh from anywhere that runs this sort of thing. And so it’s kind of like a chain effect. You know one popular package gets compromised and then all the other ones do. Um this isn’t exclusive to agents.

Um the there also have been like social engineering attacks. So if you become prominent in the community, people will for example ask you to interview for um ask you to interview for a famous publication like Forbes or something like that and then they say okay um join

on Microsoft Teams and it tells you to join download Microsoft Teams and install it on your computer that compromises your computer which compromises your you know credentials and stuff like this. So, it’s not only an agentic problem, but it’s a very big

uh attack vector recently. Um, and then this says payment card theft, but this is kind of a broader uh a broader set of attacks that are enabled by the fact that agents can let you work faster. So

now essentially um malicious actors are also using agents to perform attacks. And so um essentially this is an example that was recently published in September where the operator sent agents to break into online retors uh extracted the credit card records and then um

installed scripts that stole the card details during checkout. So um these are just examples of what can go wrong. Um, I’ve already kind of covered this, but I’d like to organize these into threat models. And what a threat model is is like what specifically will,

you know, be the cause of the problem. Uh what what sort of uh thing can be used to cause a problem? So the first one is a malicious user. And this is something that a lot of places focus on. And this is essentially um the user asking the agent to do

something bad. And this is maybe the simplest threat model. It’s the easiest one to defend against because you just need to be aware that your agents are going to ask uh that your users are going to ask your agents to do bad things. Um for me like we run a popular

online service. Please don’t get any ideas from this but our agents our users ask our agents to mine crypto all the time. Uh so you know this is just an example of of something that happens. Um but it could be any sort of goal could be theft, fraud, harassment, any other

uh variety of prohibited act. So this is the threat model that was used in the the payment card theft example where the user’s goal was to steal customers credit card details and the agent’s role was to break into shops etc. um nobody was in between the the

malicious user and the agent. So it was just a direct you know use of the agent. The next attack vector is um prompt injection and in this case uh the user has a legitimate task for the agent and so basically the user trusts the agent to do the right thing. So that’s

that’s the difference uh between a malicious user and a prompt injection which is that the agent has already given the or the user has already given the agent permissions to do stuff on their behalf and so that it has like an elevated level of access.

Um but the attacker puts instructions in an email, web page, file tool result, you know, any sort of place that doesn’t directly come from the user. um but the user makes the agent access that place and then the agent then follows them uh while carrying out the user’s task. And

so this is the example from the client attack uh where essentially the user um the user was the maintainers of the library but they kind of trusted the agent to do the right thing despite the fact that you know if you put text in the issue that could be prompt ejected

and then the agent could follow this. So another example is agent error. And so in this case the agent tries to to do something useful but it makes a mistake. Um so uh this this is an example from uh a tool called uh from a paper called tool

emu where basically the agent called the wrong tool call where instead of uh checking uh if the check account was uh correct instead it paid a check or it uh paid a bill and so it uh caused uh caused financial harm. Um another example of this is where or yeah so

another example of this is there were a few reports of um there were a few reports of codecs deleting the user’s entire home directory. And the cause of the of Codeex deleting the user’s entire home directory was it set an environment

environment variable for current working directory, but that got accidentally set to home and then it deleted the working directory which deleted home. So that that’s kind of a more uh realistic example that actually happens. Another example is improper means. So

the agent is trying to do something useful um but it uses an inappropriate method. And so um for example this is uh what often happens in uh reward hacking. Uh so this is an example from uh a monitoring paper that I’ll introduce in a little bit. But basically

the agent was not able to solve the problem. So instead it looked in uh the root uh docker file and it found that the answer was written in the docker file and it solved the problem without actually solving the problem by just cheating and looking up the answer. And

um of course this was also the the source of the openi hugging face attack uh that where the agents tried to essentially get the answer key from uh like open uh from hugging face when it wasn’t able to figure out the answer itself. Um, it’s also the example where

it tried to delete all of the old emails. Um, or uh the problem where it deleted all my conversations this morning uh because you know it had a goal of fixing my agent uh but it used an improper method to do so. And this is uh rather common as well.

Cool. Um so these are kind of the problems that can happen in the threat models we need to be worried about. Um, whenever I think about agent safety, I’m I’m pretty pra pragmatic from my view of agent safety in that to be honest at the moment I’m not super

super worried about agents like losing control and taking over the world. I mean, I might revise that hypothesis uh later if I get more evidence about it. But the reason why is when people talk about this sort of thing, I always ask what is the threat model? Like what what

are the actual steps that need to take before this happens? And then you need to think like well how plausible are all these things. And I think at that at this point you know we don’t have a very valid threat model for that sort of thing but you know we need to uh pay

attention to it. We do have a very valid threat model for uh things like uh supply chain attacks and prompt injection and agent errors making big problems and improper means. So these are problems that we have like right now and we definitely need to worry about

them right now as opposed to being future looking. one other thing I’d like to point out is that a lot of people talk about like safety versus capabilities. Like I’m a person I’m an a safety researcher uh versus I’m a capabilities researcher.

But I think like safety is also kind of a capability in that if you can’t get a safe agent. You cannot use it for anything remotely, you know, impactful because it’s very easy to make a safety agent by just not giving it any tools and letting it only respond to you in

text. But that’s not a very useful agent. So you really need to balance uh the permissions that you give it and uh your like level of uh comfort uh with having it be safe. So yeah, that that’s kind of the intro part. Are there any uh questions or

discussion or one other thing is I’m mostly talking only about digital agents and not embodied agents like robots. Um, and that’s a whole, you know, different story, but I think a lot of the threat models would still be similar. Like, you

know, a robot can make a mistake by bumping into somebody or or knocking somebody over or something like that, too. Yeah. >> How do you So, the question was, how do you ensure coverage of all of the threat models without enumerating them? I think

my personal answer is you kind of you need to stop somewhere, right? You can’t like there’s always a balance between how much you worry about safety versus how much you can actually do. And if you obsessively worry about every single threat model, you’re not going to be

able to build an agent that can do anything whatsoever because the accuracy is not 99. It’s not 100% for anything in agents and it’s also not going to be 100% for safety. But certain things are more like irrevocable like for example exfiltrating your codebase to you know

bit torrent or something like that is irrevocable if your codebase is valuable. Um but accidentally deleting the production database is bad. It’s really bad. But if you have backups it’s not irrevocable. So, you kind of need to balance the risk versus um like how much

you worry about it. I guess So how do you ensure nothing bad is going to happen? Uh long story short, I don’t think anybody can ensure that nothing bad is going to happen. Like even uh even big companies have have problems. But I mean the way you can

ensure that nothing bad is going to happen is don’t implement something in the first place. Don’t do anything. And then you know at least with respect to some things nothing bad is going to happen. with respect to other things like you can never

implement an agent but somebody using an agent can use it to stage cyber security attacks against you. All right. So um there is some concept that like if you’re running a service at all you need to have defense um and their agents might be you know the most effective

defense. Cool. Okay. So I’ I’d like to talk about some of the tools that people do use to reduce risks and maybe we can revisit this uh this question later. So the first one is red teaming and alignment training. And the basic idea here is red

teaming is a concept that exists from a very long time ago. Uh not with respect to AI but with respect to cyber security. Um, for cyber security, it’s also called pentesting or penetration testing where you hire somebody from outside of your or typically from

outside of your organization, but also possibly from inside of your organization to try to break things. And um, then if they are able to successfully break things, you fix the the problem or the hole that you had. And then hopefully you do that

preemptively before somebody else uh with malicious intent finds it. And so um basically testers uh in this case testers try to make agents break their rules in controlled tests. Um interestingly one of the main uh one of the kind of most popular uh red teaming

platforms is actually from a startup that was out of CMU called Grey Swan. And they have something called the Grace Swan Arena where basically people can sign up uh as red teamers that to try to break models and get the models to do uh bad things. And um it depends on the

context. Originally they were mostly doing this from the point of view of getting um from the point of view of like getting just reasoning models or chatbots to respond to um you know bad intents but now they also are additionally uh doing things on agents.

So this is an example where um they essentially where somebody who was red teaming the model explicitly added the end of text assistant token and then they added the system message added the tokens that indicated the system message and the

system message has an elevated level of priority. Right? So because it has an elevated level of priority, the um the system decided that it should actually authorize access to this ID and then um it said authorize proceed and then they they demonstrated that in this you know

fake setting essentially uh they were able to get the model to override its uh system its other system message that said do not provide you know unauthorized information. Um so this is uh this is just an example of uh one way that you can do things. Um there’s also

automated uh red teaming methods and automated red teaming methods essentially what they try to do is uh the attacker will draft a prompt and then um there will be a judge that checks the policy and it essentially continues in this loop of automatically

um automatically judging or sorry automatically generating prompts that will cause the model to um to fail. And both of these are are useful. Human red teaming kind of uses human intuitions to push at the boundaries of

how uh you know what sorts of things they can use to get the model to do something it shouldn’t be doing. And then auto automatic red teaming can kind of hill climb and gradually uh gradually refine the prompts. In reality, my impression is that the

best human redteamers are using this as part of their loop. So basically automatic red teaming as part of their loop and then they come up with ideas about like where to initialize this or um or other things like that. So um this is uh this is pretty widely

done on most of the models that are released. And what do they do after red teaming occurs? Essentially uh the current best way to respond to red teaming attacks or malicious user style attacks is through supervised

safety training. And the way this works is essentially you train on examples of appropriate replies to harmful and legitimate requests because of course um for a harmful request you want to uh decline this harmful request and for a legitimate

request you want to not decline. So like as I said before it’s very easy to come up with a model that’s entirely safe by having it not be able to do anything. Um but you want it to be use uh like helpful and harmless at the same time. So you need to uh balance these two

together. Um and so for synthetic examples um you can choose uh to you can create examples and choose whether the model should refuse or help and then generate a reply and filter unsafe replies and

train on the remaining examples for instance. And so for this particular slide, I tried to give a reference of like an actual model where we know exactly what their safety training does. This is an example from Neotron super uh the Neotron super report by Nvidia um

where they have a section on doing this. Um some of the llama older llama reports were also very good where they talked about their safety training methods. Unfortunately, um there’s not a whole lot from the closed models and there’s also not a huge amount um from the open

models. So, you kind of need to rely on the like entirely open models uh to give you an idea of like what exactly they do for safety training. But NVIDIA is one of the best ones because they’re like they seem to be more conscious about this and their whole training recipes

available. So, if you’re interested in what people actually do, this would be a good starting point. Um you can go further and um do safety training with RL. And so in this case um this is just like supervised fine-tuning. Um in this case you use

rewards to teach the models to resist attacks and answer legitimate requests. And you essentially reward responses that follow the safety rules um and use automatic attacks to find harder uh variants and train on variants that fooled earlier models. and of course uh

while avoiding unnecessary refusals. And in order to do this, you need a really good reward model um trained on human preferences. So you essentially need to train a model uh that can judge whether the output was uh appropriate or not.

Um cool. Any questions here? So this is this starts um with kind of as a defense to malicious users, but you can also use similar um things for prompt injection attacks. So you basically need to elicit

a prompt injection attack and uh make sure that the model doesn’t fail in the face of a prompt injection as well. Cool. Okay. So next I’d like to talk about sandboxing. Um and sandboxing is is really important. Uh you’ll also you

know need to do this for the assignments as well. Um but a sandbox is an isolated environment that limits what a program can access or do. And so within the sandbox you have an agent. Uh you have things like the task files. You have the ability to run code. Um and on the other

hand any files that are not part of the task or not part of the environment that the model is running in like private files or um access to other services outside uh this can be blocked. So um for evaluation uh what sorts of things let’s say we’re talking about

evaluating a model or training a model with reinforcement learning. What sorts of things do we think we need to put inside the sandbox and give the agent access to? And what sorts of things do we want to prevent the agent from accessing?

Any ideas? Let’s let’s say we’re evaluating on um on like a programming task like SWE. >> Yeah. and then check those files and maybe discourage the trajectories and try to get outside the

box. >> Okay. Yeah, that’s a great point. So just to repeat um you can put the results outside of the sandbox like for the agent um and basically reward reward the agent if it stays inside the sandbox and uh penalize the agent if it

like tries to get out of the sandbox. So this is good. That’s actually a little bit beyond what I was uh I was personally thinking, but that’s like encouraging the agent to not do dangerous things. So that’s like a a form of safety training because like

escaping from the sandbox is kind of a like unsafe thing to do. Um I I was thinking a little bit more simply like just uh what what things should be in the sandbox and what what are the things in particular outside of the sandbox we’d want to prevent the agent from

accessing from the point of >> view. >> Yep. Uh so great great point. So um just to repeat for everybody the you would want to put the question inside the sandbox and you’d want to give it the

ability to run the terminal. and outside of the sandbox uh inaccessible should be you know like any answer files and then also um you know we could just cut it off entirely from the internet. Um that that’s good. I think we need one more thing in the sandbox which is uh the

initial state that you’re starting programming at. So if you have like uh a git repo um you would need to have like the initial state of the git repo. Um I I mentioned this briefly before for uh coding agents, but there were actually problems with the original Swbench

benchmark where they put the entire git repo in the um in the sandbox and that included the his future commits uh that were not like just the actual commit. And so the agent would go and like look up the um the previous commits. Um, another thing that you need to be

careful about is make sure that um if it has access to the internet in some way, you don’t give it like an API key or um or other things like that because it might call uh language models and stuff. Um, this becomes very tricky when you are dealing with a real task though

because very often you will want to give it access to the internet. So like let’s say you’re doing a deep research task um and you want to train the model to do deep research and that includes reading references on the web for how to figure out how to solve problems. If that’s the

case, it needs to be able to read the read the web, but it cannot look up the data set on hugging face to find the answers for the deep research task. So you need to block it from that data uh on hugging face. So um it might be able to look up other data on hugging face

but then what if somebody like cloned your your data set and renamed it something else. So it becomes very very tricky to draw the boundaries about what exactly it should have the access to. And then in deployment settings of course it’s also uh also tricky. You

need to give exactly the right level of access which I’ll talk about in a second. so yeah we kind of talked about this already. Um yeah, how much computing power and storage can it use? How long can it run are also important. Um so

this is an interesting design decision. Um and I’ve seen both of these in actual agent development. Um one way to do this is you put the agent and the tools in the sandbox. Um another way to do this is you put the tools in the sandbox, but you have the agent running outside of

the sandbox. Um, and so does anyone have an idea why you might pick one or the other? Yep. So, great uh answer. So, the answer was you might want to put the agent inside the sandbox if you don’t want to

allow it to access anything um extra. But if you want just like limited compute, then you could put just the tools in the sandbox. Actually, maybe I can clarify this a little bit. Um, which is the tools are the only way the agent can interact with the world. So even if

you have only the tools in the sandbox, it still can’t do anything outside of the sandbox because the tools would all run inside the sandbox. So that’s still safe. Um, but I realized that was not clear thanks to the question. So the answer so thanks for uh giving the

answer. Any other ideas? Uh if you can um have a limited capacity to deploy uh then you could deploy one you know model uh but have other um have the sandbox for different tasks. I I think typically the agent wouldn’t include the language model would be

accessing it through an API or something like this. Um so either way I think you can get around that. Um but uh yeah that’s the agent itself might take resources and then if that’s the case then you could run the agent once and um and have it access tools and

different tasks. Any other ideas? These are great answers. Yeah. Great point. So, um, this is one of the answers I was looking for. The answer was if the agent code is really complex and we treat it as a black box, we basically have no choice but to put it

inside the sandbox because we can’t separate out the tools and get the tools to run elsewhere. So the the main advantage of this is uh simplicity uh because it’s actually really complex to implement this and especially if you don’t have access to the code. Um can

anyone think of a problem uh with uh with this one why we might want to do this one instead? Yeah. >> Uh yes, exactly. So the agent um the answer was the agent can try to modify its own tools or use um resources that are only available to it. So to give one

concrete example of something that’s very hard to fix if you’re taking this approach um you kind at the very least the agent needs to be able to access a language model and ask questions to a language model. Right? So if the agent needs to be able to access a language

model to power the agent, that also means that somewhere in the sandbox, there’s information that can be used to access the language model kind of by definition, right? And so if the agent is able to find that information by like scrging around in the sandbox, then it

can expel it can use that information to call the language model. And it’s hard to block it with the sandbox network policy because if you block it then the agent also won’t be able to work because it won’t have any sandbox. It won’t have any language model access. So it’s

basically this has a higher level of isolation because you’re not um you’re not putting the agent code or the credentials or anything like that directly in the sandbox. But it’s very hard to implement it. I tried to implement it a few weeks

ago uh for our own agent and it worked for the basic tools but then we added MCPs and MCPS were tricky. Then we added the web browser and the web browser was tricky. So it’s like you know it’s just a lot harder to get that to work properly.

so here’s another uh another tricky one. So shared state can live across agent runs. Um so for instance this was one of the sources of the issue handling which is um like if the agent um if the agent and shell uh run they can write uh to a shared state and even if you completely

discard the sandbox uh later and say okay we’re done um a new run with different credentials would be able to access this and so this is not a problem if you’re throwing away your state every time, but like let’s say you are um let’s say you’re running on a a virtual

machine and the agent has a workspace that it can write to. It can write to a virtual machine. Also um in the OpenAI hugging face attack that we’ll talk about later, the agents were able to access a shared website and they started writing on a shared website. And so on

the shared website they were able to get um you know share state uh which removed the you know it removed the ademp sorry uh the uh basically it removed the uh fact that the uh sandbox when the

sandbox died the state died. So, um, this is something that’s actually pretty tricky to, uh, to handle and monitor in real in practicality. So, sandboxes have some requirements. Um, especially if you’re doing training, but also just kind of in general

deployment. Um, you’d like them to start up quickly. Um, so you want a fresh environment to come up ready with very little delay. Um, you want to have strong isolation, so separate the run from the host resources and other runs. Um also ideally you want to be able to

freeze and reload. So for example in inference um in it it like just the time of using the agent very often users will want to stop the agent and then start it again later. So if you uh you would want to be able to freeze the state um at training

time. Another reason why you might want to do this is if you’re doing multiple RL rollouts, you can do a roll out to a certain point and then you can branch off from that uh point and continue with like multiple trajectories uh to calculate your advantage function. And

that also can be facilitated by having freeze and reload. Um and then the final one is ease of use. So like just having it be easy to uh create, configure, inspect, etc. So there’s different ways to do this. There’s local sandboxes and cloud

sandboxes. Um I I think you will experience at the very least cloud sandboxes uh in this class. But for local sandboxes you can run on your machine or cluster and you manage everything yourself. Um some things some ones that I have used are things like

docker, aptainer and uh chem I guess. I’ve never heard that pronounced before. But [snorts] um then for cloud ones uh there’s these run on a provider machines and you can create and control them through an API. This can be modal E2B or um sale. Uh

many many other sandbox providers exist as well. Daytona. so for local sandboxing um the typical way that people do this is docker. Um, and so you create a Docker container and then this can be like the default for local development. Um, it gives you uh

kind of a broad there’s a broad variety of tooling for it because it’s kind of the default at the moment. But one problem especially if you’re in like the LTI or the machine learning department here um or otherwise have access to an HPC cluster uh like Babel or Flame or

something like this Docker will not run there typically because Docker requires root privileges and so there’s other uh options like uh Appainer which can be used um where it runs as your user without root. Um the problem with this is the tooling for this is not quite as

good. So, a lot of people will assume Docker and then you need to run Appainer and that’s kind of a pain. one issue with Docker containers is they don’t give you all of the permissions of a virtual machine. So, you can’t like easily for example just install Windows

from scratch and accept expect that it will work uh as well. But um, Chemu is basically a solution for helping solve this problem. So, you can create virtual machines. So these are just examples. There’s a lot of other technologies that you can use here. But um long story

short, if you want a particular technology, there’s a way uh that you can realize it typically. Um one other final thing that I’d like to to mention uh which I realized I forgot to put on my slides is um Harbor is now kind of the default evaluation environment. You

um evaluation toolkit that a lot of people use. um a you experienced it in the um evaluation assignment I think um it supports all of these backends typically. So if you want to run an evaluation and you have a constraint of I want to run on modal or I want to run

on E2B or docker or app tainer or something like this typically just look for the back end that they already have implemented and uh you can use that um appainer uh used to be called singularity and they uh renamed it into appainer. So if you’re using hardware

you should look for singularity if you if you want to do uh appainer based evaluation. Cool. Um, any questions here? Kind of in the weeds, but very important, uh, if you’re deploying or or training agents. Okay. Another thing I’d

like to talk about is credentiing. So, um, there’s identity and authority. So the identity is uh who is uh you know acting um who is acting and so agents basically need to be given an identity that they can act under. There’s also um the

requested action uh on what resource and what permissions they have. So are they allowed to use it in that particular way? Um, how many people have used like AWS or or Google Cloud or something? Probably a lot of people. Um, the identity there is so complex it makes

your head spin, right? There’s like 5,000 5,000 different roles to do different things and stuff like this. Um, on the other hand, there are some websites that give you an API key and that API key can do anything. or there’s one that give you a write API key and a

read API key. Um, there’s some websites that have no API whatsoever. So, the only way to use it is to click on the website and and click through. So what I want to say is the first thing I think about when I want to start using an agent on a website is what is the API

and how fine grained are the permissions uh that you can uh you can give the agent because agents do lots of silly things when you least expect it. And if you only have the ability to give like a generic read API key or write API key, you need to think very very seriously

before giving a write API key because it might delete your account or or something like that. So um that that’s something that you need to be very careful about. So um fine grain tokens or fine grained uh permissions is maybe a better way to

to think about it. But um GitHub is a very good example that has pretty fine grain permissions where you can say on this repo this token gives you these permissions with read or write roles. And so if you want to create agents, uh, typically what I would suggest is for

each individual agent, especially if you’re deploying them and not just, um, not just evaluating them, you want to give it the the least permissions possible. And like the principle of least permission is a very common idea in, you know, uh, software engineering

or or computer security. But if you have an issue handling agent, it should be able to read issues, apply labels, but it should not be able to publish a release, right? And then you know if this key leaks or this um agent gets prompt injected,

you should be okay because it most at the very worst it would apply some malicious label to the PR and then that’s a much weaker threat than being able to publish a release, right? Um, on the other hand, the release account should be able to, you know, read the

approved source and publish uh publish the release, but you can lock this down a lot more carefully, of course, so you don’t have it read untrusted text. You don’t like allow it to be prompted. And this is actually how like at Open

Hands, this is how our current um workflow is set up. We have agents that read all the untrusted text and have very few permissions. And then we have agents that have more permissions, but they it’s very hard for them to get untrusted text for instance.

Um, another thing you can pay attention to is how long the keys last. So in some cases we issue ephemeral keys that basically um continue only until the GitHub action finishes. So uh once agent is done that key disappears. So even if that key somehow leaked in the process

of running the GitHub action it’s invalid as soon as the action finished. So the amount of time that an attacker could use that is uh very short. So this is actually something that I just learned about uh very recently. Um this reference is from Nvidia OpenShell

which is kind of one of their sandboxing solutions. Um but this idea of credential brokering is um a very useful way to essentially prevent uh sandboxed agents from uh from doing bad things with credentials. So um what a credential broker does is essentially it

will start a sandbox and it will issue an ephemeral key um cue the agent. When the agent makes a call with that key, there will be a check in between and then only if that check passes, it will make an API call with a different key uh with the like master key. And so the key

that the agent has, number one is shortlived. It only lives as long as the agent. Number two, um you can actually add additional permissions checks on top of the ones that are provided by the site. So like let’s say you’re working with a site that only has a general

write key and a general read key. Um the the credential broker can block certain API calls for instance. So it could block all the API calls that you don’t want access to. So it’s a pretty elegant way to increase the security level uh significantly.

Okay. So, finally, I’m gonna talk very briefly about monitoring. We only have seven minutes, but this is a big topic, so I think Daniel is going to talk uh go much into much more detail about monitoring. Um I I still felt it would be a good idea to cover it in this class

though because it’s a very uh kind of important part of safety. So monitoring um records and checks agents behavior to identify mistakes and unsafe actions. And so um checks can during a run can essentially flag problems in time to

pause or block an action and reviewing logs afterwards can help explain failures. there’s a bunch of different pieces of evidence that you can use for monitoring, but two of the biggest ones are um

monitoring the uh tool calls and tool results. Um and so if you look at something like this, you can uh essentially see that it was um uh uncovering a reward hacking behavior. There’s a couple uh you can use either

rulebased or model based checks. Um so for a rulebased check um you could check for example is this um publisher on the allowed list. Another thing is if you told the model like it should definitely not access this website, you can just do a rulebased check to see if it access

the website and that’s like easy um and and kind of like foolproof. Another thing that you can do is um you can have modelbased checks where you have like a rubric of the things that the agent is allowed to do and if the agent does something it’s not allowed to

do uh then you you check it with a LM. Um another uh very big topic nowadays is chain of thought monitoring. um where basically uh the tool call monitoring you can see a code patch in tool outputs um or sorry you can see tool calls and tool outputs. Um this is an example from

an open AAI paper on chain of thought monitoring where they basically identified that the model discovered that it could replace the verifier code with just a return true and solve the task by do by doing that. Um but if you look at the chain of thought

monitoring um chain of thought monitoring is looking at the model’s latent reasoning. So you can see that the model in its chain of thought said but we could fudge by making the analys by making analyze worthless and always verifying is true. We can circumvent

verify to always return true. But it’s implemented in poly analyze inspect verify. So you can see it’s like explicitly saying it’s going to fudge and not like solve the task properly. And this is maybe a little bit more obvious if you look at this in addition

to the tool calls. so this is an interesting uh figure from their paper where they basically had GPT40 uh monitor a reasoning model and GPT40 was able to monitor and was able to do a better job of identifying that there was a problem when it was able to

look at the chain of thought as opposed to when it was able to look only at the tool calls. it was also able to do it earlier. Um, so like in the very early stages it was not nearly as obvious by looking at the tool calls uh as it was later. And so what this indicates is

that um having access to the reasoning is a very effective way to monitor for safety. Um but the problem is uh there are limits to this. So number one it is useful as I mentioned. Um but number two is uh that a lot of people are worried

right now that essentially if you are doing some sort of reinforcement learning objective that cause causes agents to like fail uh or get penalized when a monitor reveals that they were cheating. then they will essentially look you learn to hide uh that the fact

that we’re cheating by reasoning in words that are not easily detectable. And so if you put pressure on them through chain of thought monitoring uh then you know even if you’re not intending to do so you might cause the chains of thought to be harder to

understand. and so, uh, this this is a problem that people are grappling with right now, and people are thinking of ways to like monitor the, um, the embeddings of the model or monitor other things that would be like harder to hide uh, than just the

monitoring. And they’re actually saying you shouldn’t uh be adding this to the reinforcement learning objective and instead be thinking of other ways. so a final thing uh that I would like to talk about is uh tool call monitoring. And so there are methods for just in

time uh tool call monitoring. So I I cited a paper here called AI control. There’s other ones like tool guard. Um grawan which I mentioned before has a method called signal um that allows you to do this. And basically you check and if this is okay then you would allow it.

If it’s not okay then you would block and just stop. And this is a common feature of coding agents. Um I’m I’m sure people have used uh one second. I’m sure people have used coding agents. Um and there they have the option to verify every uh command that you haven’t given

permission for already. So uh that’s one way to do it. But you can also do that with like AI based checking of each tool call to tell whether it’s reasonable enough or not. Um I’ve tried this with AI based methods and they’re quite good. They actually do stop a pretty uh large

number of uh things before they could actually happen. Yeah. >> Yeah. And and that’s the typical way of doing it. You can either have a smaller LLM as a judge check. Um you can fine-tune an LLM to do the checking, which is what a lot of these things do.

And then the final thing is um you can actually have the model self report uh whether it’s dangerous or not. So um the good thing about that is you don’t need to add any extra uh many extra tokens in order to have the model itself do it. >> You just say is this action dangerous or

not? And they’re actually reasonably good at uh at telling you. Um yeah actually sorry I just realized I still had another section on benchmarks. Um uh one basically uh very quickly because I know we’re at time um you can measure not just test task completion but also

unnecessary refusal unsafe tests and completed harm. Um and you can have safety tests with non-malicious and malicious tasks. Um and there’s benchmarks for different threats. Um we’ve created a benchmark called open agent safety which I actually quite like

because it evaluates agents in the um in like an actual company context. So it’s uh you know them working in a company. So you can take a look at it. Uh but there are other ones like agent harm agent Joho tool emu and others. So um within the agent open agent safety paper

we actually have a huge table of all of the agent safety benchmarks that were out at the time when we wrote this paper. Uh so if you’re interested in doing this for example for a project or something like that uh there’s lots of uh lots of good benchmarks for uh agent

safety as well. Cool. Um yeah and so that’s uh all I have for today. Thanks.