屏幕当接口：Computer Use 与 SFT 第一课

Okay, great. So, yeah, today I’m super excited to have JY Code as our guest lecturer. So, um JY is an expert on um on multimodal machine learning um in particular computer use agents. Uh so, JY um has experience both in academia here as a PhD student CMU uh working

with uh me and Russ Solakutnav but um also has a lot of industry experience. Uh so he worked at Google research before starting the PhD and um also he led the computer use agent team at Meta’s TBD lab for a year and a half. >> Yeah. And so kind of like laid the

foundation for um the strong agents that they’re releasing now. And yeah, he also um built uh some of the most uh impactful benchmarks in this area and also did work on um search for agents. So he’ll also be giving a lecture later on in the course and we’ve covered some

of his work already on multi-agent computer use and looking forward to uh you know hearing about the foundations of KUAS today. Thanks. >> Cool. Uh thanks a lot Daniel for the kind introduction. Um so today I’m very happy

to talk about something that I’ve uh personally been very excited about for for a couple years now. Um I think today it’s like taken for granted that we have uh that computer use is a is a core capability in like many frontier models. But uh as we will see by uh kind of

stepping a bit through the history of computer use agents, this was not always the case. Uh I think for a long time like um it was not very clear whether we would want computer use agents or how we would build strong computer use agents. Um so I think computer use agents or kas

are a little bit different from the agents that you’ve already covered in this class. Um so I think they’re a bit different I think of them as a little bit different from like sui agents and more textbased agents. Um so first of all what is a cooler? Um, a coupa is a

model is an agent that observes a sorry, it’s actually not. Um, okay, there we go. Here you >> uh I’m I’m think I think it’s fine. Yeah, sorry. So, AUA is a computer use agent. Um, it’s a little bit different from the agents that you’ve already seen

cuz what they take as input is actually the graphical user interface that you have in most applications. Um, so you might imagine this could be a web browsing agent. It could be an agent that operates on your desktop computer or it could be something that operates

on a mobile device such as your phone. Um, so all of these are broadly speaking computer use agents. Um, agents that operate through the same graphical user interface that humans use. Um, so they take in like screenshots as inputs and they execute actions uh of the same type

that humans do. So clicking actions, scrolling actions, swiping, tapping and and so on. Um so the key difference here is that they operate in the exact same space as humans which I think is a very uh it allows for very interesting problems and challenges to be solved in

the modeling and evaluation space. So broadly speaking uh kers operate in the following fashion. You have a user task um that is provided to your cooler. You also have some environment observations and typically this is a screenshot uh of whatever you have on

your current computer screen or mobile device. Um this can also be some other uh like textbased representations but broadly speaking most frontier course uh use like screenshot representations. So this is essentially exactly what the user would see if they were to attempt

the same task. Both of these inputs go into your cool which then produces a reasoning uh trace typically. So it’s like a chain of thought and it produces an action that it wants to execute next. And this action can be uh any any action that you

allow in your um in your environment. So it can be clicking, scrolling, typing and so on. We then process this action and we execute it in in some infrastructure that you have uh in your environment. Um so if it’s a click action, you actually

execute it on the computer and you uh get some new observation after executing this action. To give you a concrete example, um you might have a task like buy me a bloom map and you might have some initial environment observation for example from

your web browser. So these would go into input into your computer use agent model and your model might produce some reasoning trace of the form. I see a blue monk on the center of the screen. I will click it and it will then produce uh some like formatted action to be

executed. So in this case it wants to do a left click on the on the environment. um we will execute this action on the environment. So we will actually like click somewhere on the environment and we will get a new observation that happens after we execute this action. So

for example over here you get a new screenshot. Um in this case the agent has clicked into the blue mark. So you kind of have a new observation that shows the details of the blue mark. Your model then takes in this new observation and it produces a new action to be

executed condition on this updated observation. So it will it might then say like delivery details are below. I will scroll down to find them and it will produce this scroll action that we will again like execute on the infrastructure. Um you then get a new

observation and you basically repeat this until um your model decides it’s done. Right? So this can go on like for many many iterations depending on how complex your task is. Uh so yeah you start with an observation your model executes an action you get a

new observation um and this repeats until the end and um typically you get some reward either from the environment a human user or some other judge telling you whether the model did it successfully or not. Um so typically like this reward is just a 01 uh binary

reward but it can also be um a partial reward in some cases. Um so this is broadly speaking how a computer use agent works. It’s this like observe, reason, act loop that that repeats until uh your model exhausts its budget or it thinks it has successfully

completed the task. Um and today it seems like very simple uh and models are actually very capable as as we’ll see in a couple slides. Um but this was not always the case. So um maybe we can have a quick poll. So when do people think like computer use agents were first uh

like invented? Um do people think it was like before 2020? Okay, a few hands. Um say like 2020 to 2024 period is when we started building coolers. Cool. And who thinks it was like after 2024? Um so the the answer is actually that um

the first cooler paper came out in in 2017. So this was like uh quite ancient in in um AI standards, right? Uh it was actually a paper from OpenAI. So they had this project called um World of Bits where they built this very simplified toy environment. Um as you can see here,

it’s mostly like synthetic data. It’s very simplified environments that look maybe a bit like what you might see on a mobile device, but it’s very simplified. Uh not not realistic at all. Um and what they did in this paper is that they actually did like a bunch of RL

training. Um so this is not really similar to like modern LLM RL but it’s more like similar to RL that we did for um game agents in in the past. Um but this was I think this project was like ahead of its time. All of the models back then could not do this at all. Like

even this simplified uh mini world of bits task they were very bad at it. Um so this was the first cooler paper probably. Um in 2022 I think a bit more interest started picking up with the advent of uh language models. Um there was this other paper called web shop

that um also did uh kind of like simplified shopping environments but it was a lot more realistic than than mini world of bits as you can see in this visualization here. So it’s kind of clicking through this like simulated Amazon site. Um and it’s still a bit

artificial but it’s a lot better than mini world of bits. Um and at this point like language model agents started doing a bit better at these tasks. uh and we weren’t really training them in uh in in the way that we were for many world bits in 2022. In 2024 actually um there was

this I think of it as a seminal work uh from actually Graham’s lab at CMU called web arena. So it was this work that created a lot of simulated environments. Um and as you can see like all of these environments are a lot more realistic than web shop and mini world of bits. um

they contain like very realistic data often scripted from real world websites. So these environments were populated with actual data from like Amazon and Reddit and and so on. um so these are like very realistic data and environments. Um and I think they marked

a point where uh the models also started getting a bit better at these problems. In 2024 uh there was also this almost concurrent release of this benchmark called OS world. So I think this is one of the most uh standard computer use benchmarks these days. And this was also

using uh real apps and kind of like real data but um uh it tackled like desktop use rather than web browsing. So it’s a bit more general. You can use a lot of different apps on Linux, Windows uh and so on. Um so this is a bit different from web arena because web arena was web

browsing but OS world is uh like fullon desktop use. Um I think things also took a bit of a change like in 2025 to 2026. So this was the point where like coding agents got a lot better and at this point like a lot of people started generating

environments uh end to end. Um so for web arena and OS world what they did was they actually took open source implementations of these apps and they filled it with realistic data but all of the all all of the software was already written by humans and open sourced in

some manner. um my PC bench and a few other uh benchmarks around the 2025 to 2026 period actually use coding agents to generate uh the entire app and website um end to end. So shown here is actually a Gmail clone. It’s not the real Gmail, but it looks very close to

it. And this is actually generated completely by an LLM a coding agent. Um all of the data that is seated here is also completely synthetic. So, we’ve actually gotten to a point where a lot of like frontier cooler environments are completely generated and completely

filled with synthetic data and um I think they actually work very well. Uh sometimes they’re even better than real applications because um they’re a lot cleaner and they’re not as as noisy. Ka world was also another project that came from CMU that took like a real

world software. So, I think Kool World is very interesting because the software used here is a bit more of the long tail. So you have some software used for like astronomy and biology and other things of that sort. Um so it’s a bit it it departs a bit from like just

consumerbased apps into very specialized desktop software. Um all of the data here is also uh I think is realistic data. So it’s downloaded data from real world use cases um and is loaded onto onto onto these environments. Uh shown here is a demo very recently

from this month of GPT6 Astra. So GBT6 Astra was a huge leap in computer use. I think it basically like solved most of the computer use benchmarks. Uh it’s like better than human baselines at this point. Um it’s kind of a lot faster than previous models, but it’s still very

slow and very expensive. So this demo is showing GPT6 Astra operating like a CAD software. So it’s doing like some like 3D modeling. Um it’s actually sped up 30 times. So in in practice like this model is actually 30 times slower than whatever you see here. Um but but the

point is it’s very capable but it’s still very expensive and and slow. Um so we still have some research problems left but in terms of capabilities I think current models are are very good. Um so you can see like we progress from toy environments with miniw walk uh to

today 9 years later where we have GPT6 extra that is uh comparable or better than your median human at a lot of computer tasks. Um so very exciting very fast progress and we can think about like how we might measure these problems. Um if you think

about them from a research or academic standpoint uh benchmarks are one of the core ways in which we measure how fast these agents have been progressing. Um so we’ll go through an overview of like several computer use benchmarks and how they measure different axes of uh agent

performance. Um so if we think about how computer use might be measured uh it’s actually a bit I think it’s a bit more interesting um than some of the other agents that we’ve already covered in this course. Um so computer use is in some sense a bit less

of a verifiable task compared to like software engineering or math. Um in software engineering or math you kind of have unit tests or or um symbolic verifiers that tell you whether you successfully accomplish a task. for computer use is actually a bit different

because you can imagine that um successfully accomplishing a problem is not always um people do not even people do not generally agree on it. So it can be challenging to write like programmatic verifiers. Um so we have several ways of evaluating how computer

use agents do on some of these benchmarks. Um the most obvious one is whether the action was correct. So you can measure whether a click action was uh done on the correct item. Um you can also measure the outcome which is probably

the most endto-end fashion. So you give it a task like buy me a blue mark and you measure the outcome of whether it actually managed to um purchase the item or not. So this can be done through either like um checking the state if you have a simulated environment or you can

just look at the trajectory of the agent and decide whether it um provided enough evidence that it did the task. Um, and lastly, I think you also want to measure whether the model respected any constraints that might be inherent or explicit in the task. So, for example,

you might want it to navigate to the checkout page but not actually check out. Um, so you also have some other like uh verifiers to make sure that it did the task while respecting any userdefined constraints. So, I think evaluation is generally very challenging

for Kas. Um, there’s a few broad categories of evals that we’ll go through today. Um, so broadly speaking, I’ve categorized them into four different categories. So the first one that we’ll talk about is static evaluations.

Um, so let’s go back to our task. Add a blue mark under $20 to the cart and do not purchase. Static evaluations measure whether every action along the the trajectory of the agent is correct or not. So for example, you might start off on this screenshot and your agent would,

you know, try to attempt to click on this blue mark here. um we might have some ground truth reference rule like we want XY that the agent predicts to be in a target box. If it’s if the prediction is within a target box for example this red box here we give it a score of one.

Um otherwise we kind of give it a score of zero meaning that it did not perform the correct next action at this point. Um so in this case like for static evals uh we consider it to be correct if the current step produces the correct action or not. Um so this this because it’s

static you can’t really like roll out the agent end to end. Uh we just consider actions at every point um to determine whether it successfully accomplished the current action. One canonical example for benchmarks in this area is known as screen spot pro.

Um so screen spot pro is what we call a grounding benchmark. Um it tests the abilities of the models to ground actions in the screenshots and it consists of a variety of applications operating systems and about 1,500 different screenshot examples. The

evaluation in this case is the perpoint uh accuracy. So for example this task of constructing a UV spear mesh um you have this like uh Blender screenshot here. So this is a screenshot. It’s not an actual environment. So the agent does not have access to any interactive environment.

is just given this screenshot and the goal is to determine whether it produces an action that falls within this red bounding box over here. So if it if it predicts a click action that falls within this uh bounding box, we consider it to be a correct action. Otherwise, we

consider it to be an incorrect action and and we don’t give it any any reward. Another example of a static benchmark is mine to web um which tests a bit more of a variety of actions. So um you can think of this as like next action imitation in this case given a current

screenshot and a task. Um for example rent a car in Brooklyn central um I’m supposed to predict the next action that my agent will take. Uh so in this case the ground truth action is to click on the car tab and um no actual action is executed on this task. You’re given a

series of screenshots and your agent has to predict the next action condition on that screenshot. So once it predicts the action we score it and we provide it the next screenshot from our offline data set. So again, this is not actually executing any actions in the

environment. There’s no environment in this case. It’s a series of static screenshots and ground truth actions. Um, so as you can imagine, this kind of evaluation is very fast. It’s very cheap to run because you don’t actually have to support an environment or simulation.

You just have to provide it screenshots and measure whether the resulting predicted actions are within uh the ground truth set. Um but one problem as you can imagine with static evals um is that there’s a lot of uh false negatives uh in the sense that

for this screenshot and for this task there’s a bunch of different actions that you can click that you can perform that would be considered correct uh like for example I don’t really have to click the car tab in this case I could adjust the dates um to begin with and that

would also be considered a valid action but in static evals because you have a set of ground troop actions um we we actually would mark those as as incorrect. Um so that’s one of the main drawbacks of static evals. They are very fast but they don’t really measure the

thing that you care about uh which is the true end toend performance. So I think what we actually care about is whether the agent successfully did the thing at the end of the day. Um which brings us to our second type of evaluations which are end to end

Um so again like let’s let’s look at this example task like add a blue monk under under $20. um we might have an agent execute this task in an environment or on the real web and at the end we can uh check whether the agent did it correctly by using some uh

programmatic unit test like checks. So for example um an agent might have attempted attempted this task and at the end of the trajectory we can uh run these like unit test like checks to see whether it actually did it. So this is done in a simulated environment. So we

actually have like API access to all of the back end and state of the environment. Um so we can check whether the blue mark is in the cart uh whether the blue mark is less than $20 and whether the agent has not um actually place the order. So we check whether the

order is the order count is zero and we only give it a reward of one if all of these individual unit tests pass. Um so this this is actually like a programmatic check. You actually have an environment in this case. So the agent is doing like whatever it needs to do.

Um so we don’t we don’t check for the individual actions that the agent executes at every step but we do a check for uh whether it actually accomplish the task at the end based on these unit tests. So if your unit tests are comprehensive enough you could imagine

that this is a very good uh reward signal um because we don’t really supervise every single action of the of the model. Uh we just kind of like check whether it did it correctly or not at the end. So web arena was one of the first

benchmarks to do this. uh web arena was a simulated um web navigation benchmark consisting of a few environments. Um so you have 800 about 800 tasks on four different uh apps that web arena uh hosts and the agent will execute all of these 800 tasks and at the end you kind

of have a pro programmatic verifier as we discussed to check whether the agent did it correctly or not. So this is an example task. Um the goal is to create an empty repository called awesome LLM reading. So the agent will go onto this simulated gitlab uh repo. So importantly

this is a simulated environment. It’s something that you host locally. Uh you do not go on the real web at any point in time for for web arena. Um the agent has to try and accomplish this goal within the simulated environment and at the end of the task we run this

programmatic verifier which basically checks for a bunch of conditions. Uh so you you essentially check whether it managed to create this repository or not. Um and all of these verifiers are handwritten by students uh by grad students at CMU actually. Um so they uh

they hand wrote these verifiers for every single task in web arena and um at the end of the assuming you know the students did it correctly uh at the end of the execution um you would have a verifier that tells you whether the agent successfully did it or not. Um

there there’s still a few like uh false negatives and false positives because the verifiers are not comprehensive enough but on the whole it’s a very uh I think it’s a very nice benchmark. It definitely like led the way to people taking and evaluating coolers a bit more

seriously. Um so I think it’s important to contrast this with previous static evals because this actually measures the end to end performance. So you kind of just check for the outcome. you don’t care about the individual steps as long as the

agent did the task correctly. Um but you could also imagine that these this leads to some reward hacking. So the agent might uh you know try to accomplish the verifiers rather than actually doing the thing which is a common problem across most cool and and agent um evals and

training generally. Um but I think this was a huge step up step up in in agent evals. Um so they made sure that you did it in simulated sandboxes and they uh measured like the outcome rather than the individual steps. Uh visual web arena was a an extension

of web arena that kind of like treated it as a multimodal problem instead. Um so at this point when web arena was released most models were um handling the cooler problem or web browsing problem using accessibility trees which are kind of a simplified version of HTML

and visual web arena actually uh tried to convince people that you kind of want to operate using screenshot based approaches. So visual web arena is an extension of web arena that consists of 910 uh visually grounded tasks. So all of the problems in visual web arena have

to be solved using multimodal representations. Um and the evaluation is a bit similar. So you kind of have like programmatic checks as well. Um but you also have some multimodal based uh programmatic checks. So um there’s some problems that you can’t solve if you

rely purely on HTML. Another form of evaluation for these benchmarks is instead of using programmatic verifiers, you kind of use an a different LLM to judge whether the model did the task correctly. So given a trajectory of a model, you prompt a

different VLM or LLM. Um so a textbased LLM or multimodal uh vision language model uh you prompt it with the trajectory of the model and you ask it whether it completed the task successfully and this is also a very popular evaluation metric um because

it’s not always possible to write u verifiers for every single task that you care about. Um so in this case you again have the trajectory that the agent um completed and instead of a programmatic verifier you have a bunch of rubrics or or maybe you just even provide the task

to the to the LLM. Um so you provide the trajectory of the agent and you provide the task and the the LLM kind of tells you whether it thinks the agent completed the task successfully or not. Um so typically like these judges they produce a short chain of thought

followed by a judgment for example pass or fail or sometimes uh you assign like um floating point scores but typically the judge tells you whether you pass or fail or not. Um so this is a bit more general because you don’t need to write verifiers for every single task in your

benchmark. Um but it’s also a bit more prone to like false uh false positives and negatives because um you can imagine that your LLM judge can be easily fooled into thinking that a task was successfully accomplished when it was not. Um but in general this is a lot

more uh flexible and um several benchmarks that use this form of judging include web voyager which was one of the first benchmarks that tested the abilities of coolers to navigate on the real web. Um so this is very different from web arena because um

what they did is they actually came up with 15 different websites and 600 tasks on these websites and they told the uh the benchmark is to actually go on the real web on these websites to accomplish certain goals. Um so actually if you go on some of these websites in the

benchmark like there’s this recipe site called all recipes. If you look at some of the task that the agent is supposed to do you can actually see some traces on the real web of previous agents attempting this task. uh which is actually it’s not great but you can um

see some evidence of why we might not want to evaluate agents on the real web. Um but nevertheless like web voyager was still quite valuable because it tested the capabilities of these agents on non simulated environments. So it’s basically as close to a real world

evaluation as you can get. Um it gets agents to go and do these things on the real web. Uh for example like find the most start GitHub project for climate change data visualization. um they they uh deploy like who was to go on the real GitHub and do a bunch of search and uh

find the actual most start GitHub repo for this task. Um and in this case because it’s going on the real web you can’t really have programmatic verifiers because you can’t really uh query the real web for its state. Um so the eval here is the VLM judge that we talked

about. So the judge which is a different LLM gets the uh series of actions and screenshots that the agent has has taken and observed. Um it gets the original task. So it sees this task. It sees what the agent has done and it kind of predicts whether the agent succeeded or

not. So we we evaluate this problem with like VLM judges. Um no programmatic verifiers. Um so this is kind of like you can think of it as quite uh wild wild west evals. you deploy something on the real web, you run a judge and you um you know at the end of the day say

whether it’s succeeded or not. But I think a lot of people like this because it’s in some ways as close to a real world eval as you can get. Online mind to web was a was a different eval that followed up on web voyager. Um so it’s a bit more comprehensive. The

judge is a bit more stronger. So they show that the judge actually has about 85% agreement with humans. Um so it’s very close to uh being uh human level agreement for for judges. Um but essentially uh online Mind2 web also looks at real world evals. Um so they

expanded to about 136 websites and 300 tasks and they get LLMs to go on these 136 real websites to accomplish certain tasks. Um and they they show that uh it actually um using this judge is a bit more accurate compared to Web Voyagers judge. Um so they they have a bit more

of a complicated judge with different rubrics and and criteria. Um but essentially this also goes on the real web and it inherits it inherits a lot of problems that going on the real web um poses similar to web voyager. Um but again it’s a very nice eval because it’s

[snorts] in some ways as close to a real world eval as you can get. Um so this is completely automated but you have an LLM judging the task. You have an LLM attempting the task on the real world. Um and because uh the judge is very close to human eval

kind of have this automated pipeline of an LLM going on the real world doing something and you have a human-like judge determining whether it succeeded or not. So I think it’s a very nice benchmark um very automated uh but again I think people don’t really use this uh

recently anymore because it’s um essentially solved by modern models and I think as cool words have gotten more popular um people have also become a bit more careful about deploying them on the real world um because these website owners really don’t like having like

tons of agents ping ping their websites to to do a benchmark um yeah you can imagine that if your website got included in this benchmark and you started getting a lot more traffic, uh you probably also wouldn’t be very pleased about it.

Yeah. Um so one I think one important thing is that all of these evaluation metrics that we talked about are very um they are programmatic verifiers or they are LLM judges. Um so they are kind of like proxies uh to determine whether we successfully accomplish a task or not.

Um but human review meaning uh getting humans to judge whether the agent did something or not is probably still the gold standard. So uh these papers also ran like a human evaluation baseline where they got humans to review the trajectories of agents and um you get a

few humans per trajectory to determine whether the agent succeeded or not. Uh and at the end you uh kind of like get humans to vote whether the agent succeeded or not. Um, this is in some ways like still the gold standard of determining whether an agent

successfully accomplished a task, but this is very expensive to run. So, it’s not really possible to do this for every single task in your benchmark or every single um uh like debug or validation run that you want to do. Um, but this is something that most benchmark creators

still do. Uh, get humans to attempt the task or judge the task and and determine whether the agent is doing the right thing. Um, but this is very expensive. So it’s typically just done once or twice for ensuring that your agents or your benchmark is uh reasonable.

So the other type of evaluation that people typically do is um they’re also end to end evaluations but they’re done on like desktop computers or mobile devices instead of of the web. Um so the web is quite interesting because you can do a lot of things on the web. Uh I

think personally most of my time is spent on Chrome as well. Um but there’s a different set of tasks where uh require you to operate an actual virtual machine or an actual desktop and um there’s also several benchmarks that measure this in an end toend fashion. Uh

we’ve already talked about OS world which is one of the canonical cua benchmarks. Um so this is a real Linux virtual machine that you can spin up. It consists of about 369 tasks across nine different applications like um uh Libra Office, uh Chrome and and so on.

Um it’s 369 tasks across nine different applications. And for each of these tasks, they also have programmatic verifiers. So your agent has to um go into these virtual machines, attempt the task across multiple different apps, and at the end, we run some uh unit test

like checks to see if it succeeded or not. So some examples of task in OS world are for example uh you might want to update uh this Libra office uh calcul sheet um using receipts in a different folder. So in this case your agent has to navigate

to the folder find the receipts it has to enter like the appropriate uh values in this um workbook and it has to save the workbook. So it has to do this uh navigate across different apps um handle like entering items in in this uh Libra office and uh basically like accomplish

this task based off just an initial user instruction. After the agent thinks it’s done, we run all of these like unit tests that we have. So we check whether the the spreadsheet consists of the items that we think uh should be there and we

assign it again a 01 score of whether we think it succeeded or not. So kind of like very similar in spirit to web arena but um in this case it’s using uh real world apps. So these are apps that people actually use on a day-to-day basis and it’s operating again in a full

virtual machine. So it’s not just confined to to Chrome or to the web anymore. It’s uh operating over this virtual machine that consists of nine different apps. Um so it’s a lot more broad and you could imagine that um with desktop agents we can automate a wider

variety of tasks than if we just use web browsing agents. Uh Windows agent arena was a different benchmark that is kind of like inspired by OS World. Um but it’s a benchmark created by Microsoft to uh test the performance of agents on Windows

machines. Um so a lot of a lot of like productive work in the world actually happens on Windows machines and um I think this benchmark was quite nice because it measured the ability of agents to perform some of these tasks that are very common in enterprise

workflows or in um other companies. Yeah. So it’s uh again like you spin up a Windows virtual machine, you get an agent to attempt a task and at the end you have some uh verifiers to determine whether it did the task correctly or And again like because we’re using

programmatic verifiers um some of these are also very prone to having false positives or false negatives but um you know they have their merits uh as LLM judges do um but these these are essentially some of the core end to end eval benchmarks that that um people have

been exploring um from the 2024 to 2025 period. Yeah. Um most of these benchmarks are actually quite well solved by modern coolers. So for example the recent GPT6 release I think achieves like much higher than human performance on most of these benchmarks. So um the

community has actually um recently [clears throat] um put a bit less attention on these benchmarks that I just mentioned um and they’ve started to look at a bit more like longer horizon or um uh specialized benchmarks I would say. So long horizon computer use is

this other it it’s actually the same task but the problems that we care about span uh much longer horizons. So some of these tasks can take actual humans hours or even like two to three hours or even longer to complete. Um so now we’re really like stress testing the abilities

of kas to complete um things that humans don’t actually want to do or we’re getting them to complete like longer and longer tasks. A lot of the ones that we’ve seen in OS world the web arena uh typically take a human about uh 10 minutes or 20 minutes or less to

complete. Um, but some of the ones that we’ll see in the following slides can be several hours operating over potentially specialized software. So, we’re really getting to the point where we’re trying to get them to be better than like expert human use of the computer.

Um, a lot of these long horizon benchmarks, they follow a very similar evaluation strategy. So, you kind of have this action trace which represents what actions the LLM has taken uh on this environment. Um and you have a few different checks because it’s long

horizon. Um people tend to impose like more rubric based approaches. So the the version where we have a single outcome reward of whether it successfully did it or not is not so useful for long horizon because it’s very sparse. Um the rewards are very sparse. Um the agent can be

doing a lot of useful work in the interim but fail to achieve a perfect score. Um so we kind of have like a lot more fine grain rewards for long horizon computer use. Um there’s a few different checks here. So state check is the interim rewards that I mentioned. Um

these are kind of like rubric based rewards where they check whether the agent did um an intermediate task successfully and you kind of have a process reward where um this essentially checks whether the agent completed the entire task successfully end to end. Um

so this is kind of like the ultimate reward but we have some intermediate rewards to ensure that we have some non-zero signal for long horizon computer use. Um and of course you can always have a final verdict for whether the agent

successfully did it or not. Um but again because we’re looking at long horizon computer use uh this is very sparse. So often times like a lot of models get uh a zero on the final reward which is why you kind of want to have intermediate rubrics or um uh process rewards.

Yeah. And and importantly um as I mentioned like having a a good deliverable at the end is not always a great evaluation metric for long horizon computer use because it’s so long. Um so you kind of want to have some interim uh rewards to ensure that the model was

doing productive things in the meantime. Uh Odyssey’s was a benchmark that we built here at CMU that measures long horizon uh web browsing. So Odyssey’s consists of 200 tasks. Um we have uh rubrics for every task. So, we have a set of um things that we believe the

model should complete. Uh so, for example, for this task, um plan a trip to a Palm Springs wedding. Um there’s eight rubrics that the model needs to satisfy in order to perfectly accomplish the task. Uh for example, you need to find a a non-stop flight from Pittsburgh

to to Los Angeles. Um you need to uh find a drive time within the 9:00 a.m. to 4 p.m. window. and you have a bunch of these criterias that you need to satisfy in order to accomplish the task successfully. Um so Odyssey’s kind of like grades performance across many

different rubrics um allowing you to have partial rewards instead of just a single binary pass or fail um outcome at Yeah. So um this example I already mentioned we kind of have like eight rubric items um and the model needs to satisfy like each of the rubric items in

order to achieve a perfect score idea. Um, so this is kind of an in between of uh like it it’s still LLM uh verified for the most part, but you kind of have like fine grain LLM verification because you don’t just um produce a single pass or fail uh reward at the end.

Um OS World also came up with a follow-up called OS World 2 and in this benchmark they actually tried to tackle like hourlong workflows. So OS World 1.0 know was a lot of tasks that require just like minutes of of human time to complete and OS World 2 actually has

about 108 tasks that that on on median require a human 1.6 hours to complete. Um so this is very uh the some of these workflows are very complex. Um they require like doing uh CAD modeling or 3D modeling or photo editing. Um so we’re getting into the realm of like more

specialized um software use. Um uh on average most Frontier agents that they benchmark required about 300 calls to make. Uh so this also um typically it’s it’s a few hours of like just benchmarking an agent. Uh so I think this gets to a point where it’s a lot

more interesting because you can see these agents uh even like productively working for hours at a time. Um, so it’s very expensive to evaluate, but it’s quite magical to like start in Eval, come back an hour later and you see that the agent is still productively making

progress on some of these tasks. This is an example task uh from OS World 2 and um I don’t know if I don’t know about the median human, but I definitely uh would not be able to do this uh easily. So I think we’re getting to the point where you have like expert level

task. Um, so the task here is to build a a cat support bracket and export a model that matches this drawing. So you’re provided with this reference drawing on the left and the goal of the model is to be able to produce this thing on the right uh by operating some like um

computer aided design uh software. Um so I think this is this is actually very complex. Uh it requires like navigating 3D space. It requires knowing like what tools you have available in this environment and you need to be able to sustain work over like probably several

hours because um a human would also take uh on the order of an hour or at least like half an hour to complete this task. Um so it has a lot of very interesting things. Um this is a visualization of a GPT6 a GPT6 model trying to attempt one of the OS world two tasks. So again like

this is sped up 30 times. So it’s actually much slower in practice, but you can actually see like navigating between a terminal environment, going back to the CAT software and um actually like doing 3D modeling uh autonomously for for about I think an hour of time.

Um so it’s actually able to design like these 3D parts quite well. Uh and this is a a very huge leap from the models that we had even like two years ago. Um, so we’re getting to the point where these models are are very good at attempting many of these like long

horizon paths. Yeah. Um, so I’ll mention a few other benchmarks. So cool world long is another benchmark from CMU uh that tried to tackle like long horizon computer use. Um, so again this is uh very specialized environments across a

variety of tasks for example healthcare task, engineering tasks or like uh just architecture and design task. um consists of 200 of these types of tasks and most of them require like 500 or more LLM calls. Um so again this is a few hours of human work time. Um and

when this benchmark was released early this year most models scored a very very poorly on this benchmark but um I think recent models including GPD6 uh actually um achieved like close to 90% performance on this benchmark. So, uh, long horizon computer use and like

computer use in general is very, uh, very close to being solved or some might say even solved because these models can outperform humans at a lot of these, uh, general software use tasks. Um, so hopefully I’ve convinced you that uh, some of these benchmarks are useful

uh, or or most of them were useful at some point in time. Um and uh I think I think hopefully I’ve convinced you that they are very capable and have very interesting um uh capabilities. Uh they are very good nowadays at automating a lot of different types of tasks. Um so I

think the natural next question is like how do we how are these models built? How do we build these models? How do we train them? And how do we introduce these type of capabilities to them? Um, so we’ll talk a bit about like how how modeling typically works and also how uh

these models are trained. But I think I’ll pause for a few seconds for any questions about benchmarks because I know we covered a lot of different things. for something that has a twoar. >> Yeah. So the question is whether any of

these benchmarks assign um partial rubric rewards to intermediate steps. So for example, if the task is to add something highly rated to the card and you did do that but you have some like irrelevant items that you added, do you get penalized for that? Um I think

that’s a good question. Uh I think reward design is very tricky in a lot of these benchmarks. So um some tasks in web arena do have these kind of like penalization uh but a lot of the times like people design these benchmarks so that the answer is kind of unambiguous.

So the problems are a bit artificial in a sense. So for example web arena might say add the top rated item to my cart and if you do that then um you kind of get like a perfect score but uh an agent I I’ve seen like agents reward hack by adding a ton of things to the cart um to

try and accomplish the goal. So it depends a lot on the benchmarks. I think typically people don’t really penalize it at such a fine grain manner but it’s also because at the point where these benchmarks were created the agents were so bad that I think some of the reward

scores were a bit more generous to begin with. Um so I think it’s some benchmarks do have these things but it’s not I wouldn’t say it’s like super common. Yeah, I think people tend to air on the side of like giving false positives than false negatives.

>> Yes, >> this is not exactly like a benchmark question, but do you think there is like benefit in looking at other not? >> Yeah. So the question is whether there’s benefits in using other modalities other than just pixelbased representations

like screenshots. I think there definitely is. Um open AAI has said that um at least previous models and I assume Astra as well they use like the accessibility representations of the computer when available. So Mac OS has this like uh accessibility

representation that essentially is for like visually impaired users. um it’s kind of like XML like representations where they determine where they tell you like this area is a description of a certain item like this area is my chrome window and this area is my dock and so

on. Um so the models at least GPD6 do use these representations when available. I think there is a lot of value in it um especially for uh if you want your model to be more robust. Um, I’m personally I personally really like the pure like screenshot based

representations because it’s very I think it’s very general. So you train a model that takes in just screenshot representations. You can deploy it on any device without having access to the underlying internals of the of the device. So you can train a model on like

Windows and it presumably gen generalizes a bit more easily to like a Linux or Mac computer um because they have different like underlying accessibility representations. for example. Um but yeah, I think models definitely use alternative

representations. Yeah. Yes. >> Yeah. My question was around the rubrics itself. >> Yeah. >> I’m assuming rubrics are different for each dots. So are they one is that expert generated rubrics or is that like

a synthetic rubric that you generate and then in terms of reward are they how what are ways to mitigate that from the rubrics? >> Yeah. So the question is um whether rubrics are generated by humans or I guess synthetically generated by LLMs

and um whether reward hacking happens with rubrics and how you mitigate that. Um so at least for Odysseies most of the rubrics are LLM generated but human verified. Um so one common way of like generating these rubrics is actually uh related to the previous question where

you give a model like privilege information. So you give it more information than it has at evaluation time. uh for example like some hints or some uh HTML representation and you kind of get it to do better than it would in a realistic eval scenario and based on

that you can uh generate like more accurate rubrics um because you have like privilege information. Um so for Odysseies and I think for a lot of uh industry benchmarks rubrics are typically these days like LLM generated by by human verified. Um sometimes not

even human verified but verified by a different LLM uh that that has like potentially different information. But um yeah, I think I think this is quite reasonable these days to do like synthetic generation, but I personally still feel that a human has to uh look

at everything that you put into a benchmark at some point in order to make sure that that it’s um uh legitimate. Um I think reward hacking definitely happens a lot for um training these models. Um yeah, I think we can talk a bit more about that in the training

section, but essentially I think reward hacking is a hu huge problem. I would say that rubric based rewards actually mitigate reward hacking a bit better than outcome based rewards. Um because you kind of have like these different criteria that you need to satisfy. Um if

for example like if you just want to buy like the top rated item then you can reward hack by adding every single thing on the site to your cart for example. Whereas you could imagine that rubric based rewards uh check for this. Yeah.

>> Yes. >> Do you have any sense about whether these academic benchmarks are used for evaluation or also do people train on them whether academia or industry? >> Yeah. So the question is whether um people train on these benchmarks in

academia or industry. Uh well you’re not supposed to train on these benchmarks because you’re kind of like uh testing on the thing you’re training on. But I think uh it’s kind of like common in industry to generate tasks that are very similar to your benchmarks. Um so it’s

very common for people to uh produce tasks that are like OS world like or web arena like and train on those things instead. Um so that it’s like in distribution but you’re not really training on the same test set. So I think that’s very common. Um cool world

also actually has an explicit training set. Um so for some benchmarks they have like train validation and and ideally test splits um that people can train on but um you’re not supposed to train on the the actual eval but it’s uh I think people do find like proxy signals for

essentially doing that. Yeah unfortunately cool yeah maybe the last question and we can move on. Yeah >> I was curious what representation What is the action space? >> Yeah. Um,

>> yeah. So, the question is what the action space is actually for KUS. Um, I I think we’ll cover this very quickly. So, maybe we can just uh do that in a couple slides. Yeah. But essentially, um, it depends on a model by model basis and it also depends like what

environment you want to deploy the model in. Um, yeah. So, how how do we um how is a KUA represented? As you might have guessed by now, as is with most models in AI these days, a cool is basically an LLM prompted and accepting inputs in a

certain format. Um, so more specifically, it’s a it’s typically a vision language model because most models take in um screenshot representations. So you have a goal that’s typically represented in natural language. Like again um sorry to use the

same example multiple times, but the goal might be like buy me a blue monk that’s less than $20. And you might have an initial screenshot that is based on what your environment initially has. So you have a goal and a screenshot. Uh this goes into your your policy which is

essentially an LLM. Um and your LLM produces an action. Um so typically this is represented as like text. So your LLM produces some structured representation in text of what action it wants to produce. So this might be a click action with X and Y coordinates. Uh again we

then execute this action. um we add it to the to the context that we feed into the LLM at the next step. Um we also get a new screenshot after we execute this action. So we get screenshot one after executing action one or screenshot zero. We again concatenate this to the context

that we already have. Um we feed this again to the LLM and it produces the the next action that we are supposed to execute. We then execute this action again on screenshot one um and get a new screenshot and we concatenate this to the context and we we we again feed it

to the LLM. So, um, you repeat this until until your LLM thinks it’s done or until, uh, you run out of budget, I guess. Yeah. Um, so you have this loop. Um, it’s typically called a react loop, reasoning and action. Um, and at every step, you add in you you concatenate the

action that you produce the new screenshot to your ever growing context and you feed this back into the LLM. Um, so it’s an interle like vision language interle uh, action history. you have natural language in the form of goals, actions. You also have like pixel inputs

or potentially other metadata. Um, and this essentially is like concatenated into this long sequence, this long multimodel sequence that you feed into your language model. Um, so this grows and grows. Um, there’s different ways that people used to handle like longer

and longer context, but essentially, uh, you feed in like the context, um, to the LLM at every step and it generates the next action at that point in time. Um, of course, like we don’t actually feed in the actual pixel inputs. Uh, typically for most language models, you

would tokenize this in some way. So, you would tokenize like your natural language to become uh text tokens. You would convert your screenshots into like visual tokens or or some like pixelbased representations. And the nice thing about modern language models is that um

this is typically just like concatenated directly. So, what it looks like to the model is that you have a bunch of embeddings uh that you feed into it at every single step. the model produces an action that you then tokenize and you uh concatenate it to the context. Um so you

kind of just repeat this growing the number of tokens that you feed in at every step. Um and the model takes in that uh conditions on the context at every point. Um produces the next action and you kind of repeat this. So it’s a very um nice and simple representation I

think uh and and it works surprisingly well. Yeah. So you typically have like text that you convert into token embeddings. You have screenshots that you convert into visual embeddings. And all of this is just interled and concatenated and

passed to your language model. Um and and these days your language model is a is a transformer. Um obviously that produces actions for the next turn. Yeah. So the question earlier about how like what action space we usually use. Um different models have different

action spaces. uh if you squint they like generally look similar. So they typically have a set of actions that they can produce like click actions, scroll actions um or whatever other actions you have in your uh environment, right? Um so different models produce

them in slightly different ways. For example, you might produce them as like pi autogy outputs or you might produce them as just like two calls or XML. Um and you would have to like process this and pass it into something that you can actually execute on your environment.

Yeah. Um, there’s a question in the back. Yeah. So the question is why do we use screenshots rather than HTML? I think HTML is also a valid well HTML is very messy but people have done like simplified versions of HTML as

representations. Um I guess one reason that most so most recent models do not use um or they primarily rely on screenshots. They sometimes still use metadata but uh to generalize it broadly most recent models use uh screenshots as inputs. And I think the reason for it is

that it’s a lot simpler. Um so it’s it’s a lot simpler to like take a screenshot of your computer rather than trying to process the uh underlying uh like accessibility metadata. um you you can like get HTML easily on the web but you can’t really do it as easily for like

desktop computer use right um so for that reason I think people like prefer screenshots a lot more because it’s a lot uh uh more simplified and I think somewhat surprisingly to some the screenshot based representation has achieved like very very strong

performance so it seems almost like unnecessary to use HTML or other structured representations um so I guess that that’s kind of like um Why? Uh it’s very simple and it and it works somewhat surprisingly well. >> Yeah, great question. I think um this

was something that you know people did not really accept as readily even one year ago. Um but yeah, the multimodal models are very good. So it’s uh surprisingly easy to do this. >> Uh yes, another question.

>> So if it to pick a button or something. So the loading screen as well. >> Yeah. So um there’s a bunch of like different design choices you can do with

regards to like how you can handle these edge cases. Um sorry the question is to the question is like whether whether you would take a loading screenshot immediately or what do you do when when you execute an action. So I think computer use is annoying and also

interesting because it encounter you encounter a lot of these like age cases and infrastructure problems compared to like math or or coding models. Um it this can be resolved in a couple of ways. I think the most common way that people do this is that you typically

like sleep or wait for a few seconds after executing an action. So you would execute an action and you sleep for like 3 seconds before you take the screenshot and execute the next thing. Um so if you do see a loading screen at that point in time, models are typically trained to

wait again for another um specified amount of time. Um yeah, there’s more complex infra infra um there’s like more complex ways of designing models where you kind of like wait until the screenshot changes, but I think the most simple and common way to deal with it is

you just sleep. Yeah. Yeah. Thanks a lot for the questions. So um this is essentially at a high level like how almost all frontier coolers are trained uh to the best of my knowledge. Um sorry this is not how they’re trained. This is how they are modeled

and we’ll soon like look a bit at how they are actually trained to do these uh um quite amazing things. Cool. So um with training a model uh essentially like most computer use agents are based on pre-trained uh vision language models. So training

happens at a few different stages like obviously when you deploy GPD6 as a computer use agent uh you’re not deploying I guess it’s not obvious but you’re not deploying a specialized version of GPD6 that can only do computer use. So it’s kind of a general

model that can have computer use as one of its capabilities and I think this is like increasingly the way that people do it. Um so computer use specific training happens at a few different stages. Um at the pre-training stage where models are trained uh with like large scale

internet um data uh core specific data is typically injected in a few different formats. So you have kind of like your grounding data which is um data that encourages the model to learn to ground information in in screenshots. Um so this data typically looks at like the

form of um you have a screenshot and you kind of have like a series of bounding boxes followed by coordinates. So what the model is learning to do here is that it’s learning to um uh just like learning grounding abilities. So it’s learning like if I want to click on the

button I have to produce like these XY coordinates. So it’s just learning like the most basic uh capabilities like learning localization, grounding and so on. Um you might also have some data in the form of like immediate actions. So this is actually very similar to the

static evals that we talked about. Uh you might have these screenshots, you might have some like highle task and the goal here is to predict the next action that you will you will produce. Um so this is again like very noisy data. It might even be like synthetically

generated. But the goal here in pre-training is just to introduce uh some form of like localization and grounding ability which the kua needs to do later on in order to um be able to produce like accurate click and um other types of actions. Yeah.

Um after pre-training we typically have a post-training phase. Um so nowadays like this is typically mixed in with other model capabilities but uh to simplify it we can assume that we only care about like cooler for now. Um and in this post- training phase there’s

there’s two different stages. So the first stage is that you typically do supervised fine-tuning um to train the model to mimic a human demonstration of of a computer use task. So you have a bunch of like human demonstrations that you can collect or otherwise like uh

scrape from somewhere. Um, and this typically looks a lot like the trajectories that we already have. So the human is given a screenshot, the human does something and you record that action. So it might be a click action that the human does, you record that

using some like specialized software that you have. Um, so you have a bunch of these like human demonstrations that essentially uh some humans um record humans doing cool like tasks. Um, nowadays it’s also very common to make in synthetic data. So maybe you prompt

your agent in a certain way or you prompt a stronger model to attempt the task and you kind of like keep the trajectories that you think are good. Um so this is like synthetically generated by a different model or the same model. Um and you also kind of mix this in. Uh

so this is typically a bit lower quality and noisier than human data. But you can generate this at scale. So you don’t need humans to actually produce this data. So it’s actually generally still quite useful to have um this like synthetically model generated data in

your post- training stage. Um the more interesting part I think is after this uh post- training supervised fine-tuning stage your model has some semblance of a cooler uh capability. So you can after after SFT you can typically prompt your model and it will

do a decent job at most of these cooler tasks. I think the more interesting stage is um nowadays it’s is it wasn’t very common like one year ago but nowadays it’s very common for kas to undergo an a reinforcement learning stage. So what happens here is that you

kind of drop your cooler in a simulated environment or real world and you give it a thing to do. Um it it it will do that thing. So it will like try to buy a blue mark and at the end you will assign a reward to it either from like rubric based rewards, programmatic verifiers or

like um just llm as a judge. You assign a reward to it in some way and you kind of do like reinforcement learning to reinforce the successful trajectories. Um so this is kind of interesting because it’s like uh there is no human demonstration for these tasks. You kind

of just let the model loose in a sandbox and it um tries a bunch of things uh learns from its uh successes and failures and it you you repeat this until until it’s able to do this successfully. Um yeah so I think people are very excited

about RL these days. Um but the other stages are also very important to ensure that the model has a strong foundation for doing RL. Yeah. So let’s talk a bit more in detail about the various stages. So in the pre-training phase as I mentioned you

kind of have these like screenshots and the goal of this pre-training data is just to learn uh very basic capabilities. So you probably learn like grounding abilities. um given this screenshot uh typically your model is trained to predict the next token uh for

very long description that essentially boils down to like predicting the coordinates of items on the page. So you might have this like car label and the model has to predict the xy coordinates or the xy uh width and height of this bounding box. Um so it’s kind of like

very noisy data but data that you can imagine getting in very large quantities at scale. Uh so for example like this screenshot data it comes from the United website and you can easily generate like all of the bounding box and grounding data by just running a simple like

JavaScript uh script on the website. So you can extract like all the bounding boxes. So you can collect this kind of grounding data very easily at scale which is why it’s a great fit for pre-training where you want like a a lot of uh noisy but large volumes of data.

The other type of data that people sometimes collect is um action prediction. Uh so you can collect for example in the same way from HTML pages you can get like the element name or element uh meta of certain things on the page and um you can provide this as as

the input to the cool and has to predict again the xy coordinates of this. So this is a bit more like semantic understanding where you can provide like a HTML element and it has to predict uh what it wants to do or what it wants to click on. Um so again this is like very

low-level capabilities. It’s really just teaching the cooler how to understand screenshots and how to ground uh various items on the page. Yeah. So you basically learn like screenshot understanding. Uh you learn localization. So sometimes you learn

like where items are on the page. uh you can also learn like the relationship of items with each other. So we know for example like the car button is to the right of the hotel button and uh this might be introduced in your pre-training data in Subway. Um but this is typically

generated at scale and mixed into the early stages of frontier model pre-training um just so that our cooler has a bit of an easier time later on in in learning to ground and learning to produce actions. uh after pre-training the next stage is

uh I think people typically call it post- trainining um so again like we are ignoring all the other data that goes into post- trainining and focusing on cooler data here um this typically comes in two forms so you have like human data where you pay a bunch of humans uh to

annotate trajectories so you might give someone the task like find a birya taco recipe I could make in under three hours and a human here like actually attempts this task by navigating on a web browser um so over here like this example is from momo

Momo web mix um where they collected like 36,000 human trajectories. So they collected 36,000 demonstrations of people doing tasks similar to this like going on the web and doing things and and um at every point you get the screenshot observation and you get the

thing that the human did. Uh so you end up with a bunch of like screenshot action screenshot action trajectories. They also generated a bunch of um synthetic trajectories that we’ll we’ll talk about in a bit. Um but some open- source examples of like cooler data are

Momo web mix and also agent net. So both of both of these papers essentially like collected human demonstrations uh of these tasks. So it’s a very nice way to um train your cooler to like imitate human behavior. Um so condition on screenshot is trained to predict the

next action in the sequence. Um, of course, increasingly so these days we can generate some like synthetic data for for this part of the training as well. Um, so Momo Momo web actually did this thing where they uh gave a model the accessibility tree of the

website. So I think this was um uh addressed in some of the previous questions but uh some of these models do a lot better when you give them metadata. So they have a model that is trained to use like accessibility trees and they provide the accessibility tree

which is essentially a simplified HTML representation uh to the model and they let this model like roll out taking actions in the accessibility tree space and they collect the same set of trajectories. Um so you kind of have this model that is using like privilege

information that it might not have at actual eval and inference time. So it’s uh doing a lot better than it would have if it just had screenshot inputs. Um, so you roll out this model, you collect like 105,000 synthetic traces in this way and you can

also um uh not not all traces are retained. So you can kind of like look at the traces and keep just the ones that you think are higher quality based on some metric like using a different LLM to judge it or using a human to filter them out. Um, so traces are

collected in this way either with human demonstration or or synthetic data. Um, and one analogy I really like with the synthetic data version where you have privilege information is that um, I actually don’t know if it’s true, but I heard that Tesla actually deploy deploys

their cars um, during training with LA. So they have these cars that have LA installed and they do like self-driving and obviously if you have like 3D LA information, you do a lot better. Um, and they collect those trajectories and they discard like the LAR information

and they train their model to just use like the pixelbased information. So you kind of have a teacher that is using a lot better information than you would have at inference time. Um and I think for a lot of cooler applications, people have attempted similar ways of injecting

like metadata or extra information um in order to collect better traces. Um so I think it’s a very nice way of introducing synthetic data. Yeah. Question in the back. >> Yeah. So the question is whether um if you do like pre-training does the

coordinate uh representation generalize? So if you train it on a Mac, does it generalize to a tablet or maybe even like computers with other resolutions? Right? Um and I think typically the way that we collect like coordinate data is that it’s normalized. So uh coordinates

aren’t usually represented in absolute uh formats. So they’re represented in like um zero to a thousand. Zero zero zero meaning like top left of the screen and thousand thousand meaning like bottom right of the screen. So they’re like completely normalized and in my

experience it transfers quite well even when the absolute resolutions are different. Yeah. Cool. Um question >> more efficient way of doing it like perhaps a

like more efficient ways if you have >> instead of getting the screenshot through it just stoing like values what certain items are in some database you can just >> there are boxes just like a description of what the box does

>> yeah I think that’s a That’s also a reasonable way to do it. I think models these days are not trained as uh like monolithic coolers. Like if you try using like GPD6 Astro, it doesn’t always just do computer actions. It will interleaf like bash calls and it

will like write code instead. So I think models like typically try to solve it in the most efficient way possible. Um so they definitely like do use that kind of data at training and also at inference. Um yeah.

Oh, okay. Okay, cool. Uh, okay. I’ll cover a bit more quickly then. Yeah, basically we have a bunch of like um uh open- source data that we can train models on these days. So, a lot of um different examples and the final stage of training is

actually uh post- training RL. So as I mentioned in this stage you kind of like deploy the model in a simulated environment or if you are feeling um you know extra spicy you can also deploy on the real web but people don’t really do that nowadays. Um so you typically like

deploy the model it takes actions in a simulated environment and at the end it produces you get a reward from the environment or from some other judge and you update the policy based on the rewards. So uh broadly speaking like if you get a positive reward you upweight

the actions that happen in that trajectory a bit higher and if you get a zero or negative reward you um downweight those actions. So your model essentially is trained to maximize the reward in an environment and in this case it’s like doing it completely

autonomously. So no human demonstrations. Um the tasks that the model is attempting might come from humans but uh typically it’s like operating in an autonomous fashion. Yeah. And um hopefully it’s obvious why we need simulated environments for RL. A

lot of the tasks that we care about, we don’t really want to execute them repeatedly on the real world. For example, you can only book uh an airport ride on Uber so many times before you get banned or you run out of money, right? So you kind of want like

simulated environments to do reinforcement learning in. Um and there’s several uh benchmarks that uh kind of like tackle the reinforcement learning building uh paradigm. Um so I think I won’t go through these in detail but uh you can

reference them on the slides. Um but essentially I like to show this graph which shows that the reinforcement learning part of post training is actually very important for uh introducing like strong cooler abilities. Um so you start off with a

success rate after SFT of about.5 and if you do like reinforcement learning at the end you climb to about like 6 or or more. Um so it’s a very important part of ka training and uh anecdotally it also introduces some abilities that is hard to do through human uh

demonstration alone. So it’s much easier to train a model to operate over long horizons um using reinforcement learning because you typically don’t have human demonstrations of very long uh tasks. It’s very expensive to collect those. Yeah. Um so to summarize yeah we covered

like how coolers are trained. Uh you typically have a few training stages like pre-training um post- training where you do like behavior cloning on human demonstrations and synthetic data and you also have like the final cherry on top where you do the model uh

reinforcement learning phase um of running it in simulation and getting rewards in the environment and hopefully by the end of this you get a strong cool. Um but at a high level this is like the pipeline that most labs use to train cooler models these days.

So to summarize um kas are a bit different from other agents. I think they are uh kind of like really nice because they operate directly on the same interface as humans. Um it’s very tricky to evaluate coolers and people have been doing it for for a while. Um

but essentially we want to target the right metrics and level of behavior. Um cooler models are designed so that they predict actions from interled visual history. So you kind of have this interleaf context of screenshots and actions that you previously took and the

model has to condition on this to produce the next action. Uh models these days are trained in multiple stages. So you have pre-training with large scale noisy data. You have SFT where you try to behavior clone human or synthetic

demonstrations and you have a final RL phase where you uh let the model um roll out itself in simulated environments and you reward it for good behaviors and penalize it for bad behaviors. Um essentially that’s uh probably the summary of all I have time to cover off

cover today but I encourage you to visit the slides for some uh remaining takes on what we can still work on for Kers. Um, essentially I think speed is still an issue. Personalization is a very ripe area for research. Uh, infrastructure and UX is a big concern. And, um, I’m

also very excited about multi- aent systems. Yeah. Uh, thank you for listening and happy to take questions for remaining time or Yeah. God bless.