How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere

Hi everyone. My name is Nuha, I'm a senior researcher at Cohere and associate industry member at Mila and an adjunct PI at the Allen Institute to begin. So excited to tell you today about how to optimize your agent team and get frontier coding at 70% lower the cost. This is a work that I did at Cohere. All right, so traditionally there are two ways to make an agent stronger at inference time. First, you can let it think longer, meaning you scale compute sequentially with longer chain of reasoning. And second, you can sample more. You run many independent attempts in parallel and then you aggregate them. But as agents take on longer and longer task, they can navigate large repo, they can edit the code, they [snorts] can debug their own work over hours. So what can happen here is that the context window may blow up. And so we can see here that, you know, the context window eventually fills up and the longer it runs, the longer you wait. And sampling more doesn't either fix it because each attempt here still has to do the whole task on its own.

Right? So what if we split the work? So here in this case the key the lead keeps the task and hands out subtask and each worker starts with a fresh context so it can read files, it can run tests, it can dig through logs, etc. etc. and then it can send back only a compact result. So here the lead keeps the full picture and it can finish faster if the task is decomposable and agents can work on independent pieces. And so here we're talking about a new axe for scaling inference time compute. You work as a team. You give a task to a team of coding agent and watch them do the task for you. But then scaling the team gets expensive very very quickly. And so as you add more agents, your token budget can blow up. And it can actually and it raises a harder question actually. So first, how does coordination actually happen? How should we organize the team so that agent can work together etc.

And so, you know, this is the question that we set out to answer. How should a team be orchestrated to be both effective and efficient? By effective, I mean it actually solves more of the task so we get higher reward. And by efficient, I mean it doesn't burn through your budget getting there. All right. So, since these models now are extremely capable, I was wondering what if we throw an extremely challenging coding task as a society of agent and see what they can do. And so, we would expect them to automatically form clean groups, split up the work neatly, and build it together, communicate nicely, and coordinate, etc. etc. But that's not what not quite what happened actually. In reality, what happened the agent preferred to solve the task on their own. So, each went off and build their own version. Even when we asked them to communicate, they mostly discussed things, sometimes that didn't matter, and sometimes they kept also stepping on each other's toes. They also preferred to communicate back to the lead, etc. etc. So, here we were wondering, okay, how should we design a team so that, you know, we can actually get the best out of it in terms of reward and in terms of cost. And hopefully in terms of latency. And so, you know, we set out to figure out what a multi-agent harness should actually look like. And like we were wondering about the team design. So, there are many design choices and each one changes how well the team does and how much it cost. So, first the organization, right?

How do you structure the team? Who leads? And at which stage of the work, planning implementation or verification do actually workers come in? Um second thing uh is communication. Should agent talk to each other directly or only report to the lead? Third is the team size. How many workers is worth it? Like should it be 1 2 16 100 etc. And fourth, the models, right? Which model should play? Which role do we need, you know, a strong model everywhere, or, you know, can cheaper ones do the same work, right? And finally here that something that we cared about is the state of the software engineering, right? Should should, you know, we should allocate more workers towards planning, towards implementation, towards verification, etc. And so, we actually framed team design as a compute allocation problem. So, what we cared about is reward per added dollar over the solo agent, right? So, starting from a solo agent, we ask where does the next dollar of inference buy the most reward? This is what we cared about. And so, here we actually define a metric called efficiency, and it's actually a function four things. First, the stage, it's the the stage at which the worker is at, whether it's implementation planning or verification, or all of them together, we call it We also define that as a function of the worker model. Is it like a very capable model? Either is it a very, you know, you know, less capable model? The number

of workers, the lead model, etc. Um, and all right. So, basically, if we To answer this question, we actually design our multi-agent harness as a top-down hierarchy. We build deliberately a minimal harness, a mini suite agent, and it's a hierarchical, as I mentioned. Here, we have a lead, and based on the task, it decides dynamically how big that team should be and um, you know, who should do what. And so, here we made sure that every part of the harness is exposed. So, we could change one choice at a time. We keep everything else fixed and see how each one moves reward and cost. Right? And so, here more closely, how does a team of coding agent do work? You can see here every agent works through a bash tool in one shared workspace. So, basically, the lead receives the task and hands out work with a tool we call delegate agent. And then the worker sees only its subtask and reports back to the lead. And it can also communicate directly to the other worker. And so, here basically, the lead can decide based on the task, oh, actually, I need to just, you know, power an implementation worker or the verification worker, etc.

And so, as I mentioned also, the lead decides how many workers to sprout, how to combine, also consolidate the results. It also has the capability to resume a worker, to kill a worker, and to tap in and, for example, supervise and tell the worker that you have made [snorts] some errors here, fix it, etc. And so, yeah, and you may be wondering, "Okay, what is the lead doing while the other workers are working?" We actually have two variations. One is what we call the blocking lead, the other one is asynchronous lead. So, the blocking lead actually hands out subtask and waits until every worker reports back. It doesn't do anything, you know, in between. Uh whereas an async lead, it keeps actually working while, you know, the results keep coming from the other workers. It can supervise what other workers are doing, it can send them feedback, etc., etc. And so, how did we test this? We test this on a benchmark called program bench, which is basically a reverse engineering task. So, basically, the agent starts from an empty workspace with only the original program executable, which it can run but not read. And it also receives some documentation. So, here the goal is to basically probe the program to discover its interface, to discover the edge cases, the behavior, and then write a new implementation that behaves in the same way. And then, the task here were extremely challenging. They range from SQLite, sorry, uh for example, the PHP interpreter, and for example, you know, on the leaderboard on the official

website, on the task FFmpeg, the best model only passes 10% of the test. So, this is also done by, you know, frontier model. So, it was quite challenging, and it was a good test data for for our, you know, investigation. And so, the question here is that does a team beat a solo agent, yes or no? The answer is yes. It actually can beat a single agent, but at four to seven times the cost. And we can actually see here in the figure that you know, with a minimal prompt, uh a team with a blocking lead, actually, beats the solo agent by almost 10 points, but it costs about uh costs about 4 to 7 times as much. Now, you can see that the blocking actually performed better than the asynchronous, but costs higher, and it actually takes more time to complete. So, what we noticed here is that we noticed that the async lead doesn't know when to wait, when its workers are still exploring. It starts building on its own guesses before the workers finding come back, and then when the reports finally arrive, it has already moved on. And so, either it ignores them, or it has to go back and redo the work.

And sometimes it just decides the task is done while the workers are still actually running. So, basically, it decides, "Okay, done. It's time to submit the work." And so, the work of these workers is simply lost. But then, you know, this problem never occurs with a blocking lead because it will wait until the workers report back, and then it actually consolidates the work and do the submission. All right. So, the next thing I want to emphasize on is the prompt minimal. So, I I mentioned, okay, in the earlier results, I we used the minimal prompt, which is an earlier prompt, and I can't emphasize on this enough. This is how our agent operates. We give them prompt and instructions, and essentially, they do what we ask them. And so, the first one, we tried to go very, very minimal. We just told the lead about the task, you know, that it should create you know, sub workers and agents and it should inform them about the task, and should tell them about the tools. We gave also the lead tools. We wanted to see why we started minimal because we wanted to see how the agent behave on their own. Do they need a lot of guidance, or is it is a minimal prompt enough, et cetera. And so, this is very minimal. We also were wondering, okay, what happens if we give it a very heavy prompt, meaning like, you know, what the model should exactly do. We tell it exactly about different tracks. Say, follow exactly this plan, or try to, you know, delegate agents, and you know, create agent throughout the whole process of the implementation from exploring and planning to implementation, create also adversarial

workers, verify the task, etc. etc. And actually that helped a lot. We can see here, you know, compared to the minimal prompt, you know, we can see that the solo agent alone jumps almost six points just from better instructions and telling them, "Okay, don't step on each other's toes. I don't, you know, there is this folder, write in this folder, etc." So, lots lots of guidance and lots of prompt tuning, which is not ideal cuz we spent too much time there. And now these agents are supposed to be smart enough to figure out how to do it. And so, we can actually see that the teams, both, you know, async and blocking, both go up and we can see that the async lead now catches up with the blocking one, right? So, once we tell the lead how to coordinate, it knows when to wait. So, this actually really helped address the problem that I talked about earlier. But again, you know, we have a team that works better than the solo, but it's really really expensive. It costs six to seven times as much as the solo agent. So, we were wondering, now that a team beats a solo agent, but it costs more. So, if a team is a new way to spend inference time compute, where should the next dollar go, basically? Can we actually optimize our agent setup so that actually we get very good reward but at a very, you know, much cheaper cost. So, to answer this question, actually we took the same lead and gave it workers at only one stage of the software engineering process. So, the first setup that we considered here,

the workers are only responsible for the planning and exploration, right? So, basically here the lead will give them the task and then each of the workers will go and try to prompt the binary to discover, okay, what kind of plan I need to do, and then it will report the plan to the lead and the lead will take care of the implementation verification. The second one is, you know, workers are only responsible for implementation and so the lead is the one responsible for coming up with a plan, distributing that, splitting the work between workers. they will do the implementation, report back with the code actually to the to the lead. And so, basically and then the lead does the verification on its own. Now, the third actually set up that we had, the lead is responsible for both the planning and the implementation. And then here the workers are responsible to do the verification, which is the final stage and then they will report back to the lead and then the lead will decide whether to submit or not. And so, Okay, we wanted to see where exactly extra agent will pay off the most and here are the conclusion of our results like extra agent pay off most in program bench in planning. So, here you can see okay, going from solo with minimal prompt to solo with PIV. PIV is the heavy prompt. Actually, we get like a jump already and then we go with a team of heavy prompting, we get extra, you know, increase and gain in performance.

But then now when we allocate more agent towards the planning, right? We can see that we get almost the same performance as the PIV but at half the cost. So, Actually, the conclusion here is that, you know, the extra agent pay off the most in planning and um at at at the task that bottlenecks actually uh the the the task, which is planning in this case because you need to figure out how exactly what what's the binary behavior and come up with an detailed plan on what should you implement first, etc. And so, here yeah, I Actually, we computed even, you know, the efficiency average that I discussed earlier and we can see that, you know, it dominates the PIV actually on the frontier. So, each planning dollar buys almost 1.6 more reward than any other stage that we have here, whether it's implementation or verification. And actually, we we we noticed that, you know, you know, doubling down on the number of agents in the implementation stage in the implementation or verification stage doesn't actually really help. We can see that it just increased the cost but we are actually getting the same performance as the solo, so it's not worth it. And so, again, like program Bench only exposes a black box executable, so discovery here is the bottleneck. So, planners paralyze this probing and it brings and they bring lots of benefits, right?

And so, yes. So, this is to explain why planning workers pay off and we were wondering why not, for example, verification or implementation. And, you know, as I mentioned, Program Bench will give you this binary and its documentation. So, before you write anything, you have to work out what the program actually does. So, discovery is the real bottleneck here. And you can also think about this in terms of context, right? So, we have one agent that's exploring, you know, alone files uh in its own context with probing and also still misses edge cases, right? And on the other hand, you know, we have several workers each explore different part of the program in a fresh context and each one hands back a compact report to the lead. So, basically here the lead, you know, learns more than it could do alone. And so, now, you know, this is also to explain better this phenomena and you can and we actually looked at our data. And we came up with these nice figure based on what we saw and we specifically looked at uh when the lead delegates, right? So, here we have planning, we have implementation and verification and we have all the stages as well. And so, every row that we can see here in the figure is one run uh and actually time goes left to right, right? So, the yellow is planning, the blue is implementation, the green is verification and every dot here that you see in the figure is the lead sending work to a worker. And so, with planning guidance, all the delegation happens in one burst at the very, very first of, you know, the the work before any code

is written. So, basically here the the workers go explore and then they report back and then the lead builds. With a full um PIV guidance, where, you know, um there is no um specific guidance on where exactly the the the number of workers should be allocated, we get the same early burst, but then, if you You here, actually the lead keeps delegating all the way through. Another 830 calls and they buy nothing actually. They just increase the cost as we saw. It was around $500, but it didn't actually improve in performance. So, it's the same reward for planning we can see, but it's 2.4 times more reward per call that we we actually measured. And so, what pays off here is is one early burst of discovery and not delegating all the way through. Right? So, why implement workers fall short? Here, first of all, as I mentioned, the plan was the bottleneck. And so, if you don't do it right, it's not going to work out. And so, here we have only the lead that explored. It had limited capacity to discover all, you know, the the edge cases and the full binary. And so, we had a weak foundation and then basically each worker got like a narrow slice. It doesn't have visibility over the entire task and basically they were already doing errors by implementing each feature on their own. That's one problem. And on top of that, they were building on a weak foundation. And also, we noticed that

so, here you might ask me that, you know, why not give workers more context? We did. But, we noticed that when you they have more context, actually they didn't respect the boundaries of their specific task. They get excited and then they want to build extra features that nobody asked them for it. So, they just step on each other's toes, etc. The other issue that we noticed is that, you know, they are supposed to communicate together to figure out who's doing what to avoid conflicts and duplicated work, but they didn't do that. They actually they worked alone and 84 of messages go to the lead directly and nobody sees who's doing what. And sometimes here what happens is the lead gets overwhelmed and it doesn't know doesn't communicate well where the problem is to the other workers, etc. So, it can be actually quite challenging. And so, what happens here is that when, you know, implementation is not fully correct, you know, the the the lead pays for the gap. It actually tries to, you know, make up for that duplicated work and does lots of stitching. And so, here what happens is that the latency goes up, and also the reward goes down, and you know, it's also expensive.

Uh Okay, so basically so far I've been talking about program bench, but we also run experiments also on commit zero. And what I want to say here is that the result like are are different, and the hierarchy and the the allocation of the team is different according to the task. Because the bottleneck for program bench was planning. And so, allocating more agents there really really helped, but for commit zero it was not the case. The commit zero the bottleneck was not planning, because basically you're given the skeleton. So, the docs are known and the unit tests are actually known to the workers. So, the plan is basically given to you. And the the bottleneck was for the you know, for the reward to go up was the implementation. You need to get the implementation right. And so, here when we doubled the number of agent on the implementation stage, this is where we saw the biggest improvement. I'm not going to present results on this. Please check out the paper for for for these results. You'll find them in the appendix. So, the conclusion here is that you need to allocate more agents at the software engineering stage that bottleneck your task.

All right, so planning already gets us the same reward at half the cost. So, we were wondering, can we go even cheaper than this? Can we cut the cost even further? And so, here we did some experiments, and we focused for example on three GPT-5.6 models, Luna, Terra, and Sol in a this increasing order of capability and cost. And so, like the question here is that what would happen if we keep Sol as the lead, like the strongest model, and then use the cheap models as the workers? Right? So, remember here we used to have this. This is all the P here is the all Sol, you know, model. By the way, here the the the Sol the basic This is the version like of P where the lead dynamically decides on the size of the team and it ended up on average creating and spawning five agents, right? And so, basically here, we have this. We go from PIV, we actually allocate more agent towards P.

And basically, when we replace the sole, you know, sub workers with cheap ones like Luna, we actually get almost, you know, same performance but 60% less the cost. Like just we lose a little bit on performance, but it's still good because we're actually maximizing reward compared to the solo and then we're we're getting less cost. And so, you know, now the other way around. We were wondering like can the lead be cheap, right? We just even get a cheaper lead. And so, already here, we can see that okay, you know, going from like a cheap lead like solo, Luna, Terra, sole, we keep increasing in performance and also in cost. And so, can we have like this homogeneous team like Luna plus Luna, Terra plus Terra. Already we can see the scaling law the same as the model gets more capable, we get better results. But then, actually, you know, we noticed that when we have a strong lead sole plus Luna, we can get the same performance as sole plus sole, but you know, at much lower the cost.

But the other way around like cheap workers can't make up for a weak a weak lead. Basically, the lead sets the ceiling and the workers set the price. Here, Terra with Luna workers still feels sole working alone 51% versus 57%. So, essentially here, the lead sets, you know, basically this also matches Meta's Meta reasoning work recently. Like they just mentioned that a better controller raised what the same workers achieved. All right. So, so far the lead decided how many workers to spawn as I mentioned and it did on average five workers. Now, we were wondering what would happen. We want to scale ourselves the number of agent at each stage. So, we will impose the team size. So, here, uh going to, you know, work with Sol as a lead, Luna as planners because this was the best um reward cost um um uh relation and we go from two to 16 planners, right? Uh and so, here I can actually look at the cost. You know, the 16 planner team is 38% cheaper than all Sol planning team and actually 70% um cheaper than workers at every stage here for the PID, which is great >> [snorts] >> cuz here we and it seems we we you do some budget constraint, we couldn't like scale up the the number of agent further, but it seemed like going up and up and up.

Obviously, there is a ceiling to it, but it's really helping and the good news is that it's not that expensive. It's not that it's not more expensive than the five Sol team uh all allocated towards planning. Now, we also uh try to see whether uh scaling up the number the size of the team for the other variants like implementation and verification and it didn't help at all. Uh we already saw that uh you know, allocating teams um agents toward in these stages is not helping because of, you know, the foundation was weak and other considerations. And so, the last thing we were actually curious about, how do agents communicate, right? And so, we gave every agent uh a send message tool to reach any other agent actually in the sandbox and we were curious whether communication uh happens directly or, you know, it happens through the lead and whether communication will lead to better performance or worse. And so, basically, the first thing is that are they communicating together? Not really.

They are actually essentially communi- communicating through the lead. Like 84% of all messages went from a worker to the lead. Worker to worker, you can see that actually very little, less than one message per task. So, even when they can talk to each other, they actually behave like a hierarchy. Like, do your piece, you report up. Expect the lead to communicate back to the right sub-agent, etc. And you can also see this pattern consistent across all models. The lead takes much more talks Sorry, talks much more to workers than they talk to to each other, and this is also true for Cloud Opus 4.7. Also, we did the experiments on Gemini, etc. And so, also, as I mentioned, we were curious about the reward. Does it, you know, does extra communication between workers help or not? In our experiments, we didn't find it helps, and I will explain why. Um so, first, already you can see, you know, on the left you can see that about half of Luna messages never even get through, like, because they're malformed and the harness reject them, etc. Um and so, we notice also in the middle figure that Sol writes about three to four times as much message text as as Luna. And under the PIV guidance, stronger teams, actually, we can see that they tend to send more messages overall, with leads steering their workers more, which is good. So, we can see that, you know, as the you know, capability of the model so, they message rather more. But then, on the right figure, you can see that more talking doesn't mean more reward. For example,

for the Sol and Tora Terra, adding messaging changes nothing, basically, beyond noise. And Luna teams actually lose four to seven points. So, it just seems distracting to them. And we noticed that communication mostly carries status report to a lead, you know, who already receives each worker results, anyway. So, it it was just like an overhead. There was nothing important to be discussed. And I want to be clear here. I don't want to say that peer messaging is not necessary, and we should not do it. I do think it's necessary, and I do think as we scale the number of agents, they should be capable to communicate efficiently together, because it doesn't make sense to all the time communicate through your manager for you to work with your team members. No, it doesn't work like that. I think that it's also a product of our of the harness that we built, and also of the, you know, the task itself, because basically the lead was splitting the work and making sure that each work is independent of the others and so they give them that the lead gives them directly the task and so they are not also aware of other you know, agent in the same sandbox and so we gave them the tool only to communicate together but they had didn't have visibility who exists there and so they have to do extra efforts and maybe a message word would have solved the problem and so would incentivize them more to sort of communicate and to figure out oh, this actually agent is doing something similar to what I'm doing let me communicate or I need more help let me reach out to someone else.

Anyways, we leave this to future work and so the other thing that we also were curious about is like you know, whether other you know, models frontier models know how to lead. So we also run five other models family with the same harness with identical tools and so here I'm talking about office 4.8.4.7.5 also Gemini family. We noticed that 5.6 teams delegates often and it gained up to 7 points in reward. Cloud on the other hand actually behave much you know, the same gaining almost 6 points but then office 4.7 rarely delegated and actually gained very little. High crew and Gemini you know, delegated as really lead as 5.6 but then gained nothing and so what I want to say here the conclusion is that you know, two things have basically to to go right. The lead has to choose to delegate. This is important because it's actually deciding on who's going to do the work and it has to actually make use of what comes back and at which stage of the task do also delegate. All right, so here the takeaways three things to remember. The first thing is that teams are now a real new axe for instance and compute but the catch here is that they cost several times more. We got up to 10 points which is you know, great but still it's very very expensive. Now you know, we tried our best to optimize our teams of agent and we were

successful to some extent. Still some work to be done, but essentially here the conclusion is that, you know, where you put agent matters more than how many. So, you need to find the bottleneck and scale the team there. This is what matters the most. And finally, the team does not have to all be strong. Like, a strong lead with cheap workers can get you near frontier results at actually a fraction of the cost. And so, what's next here? These are open questions that we are so curious and excited about. First, here we're excited about this hierarchy versus self-organization. We, in our work, we tested up to 16 workers in a hierarchical lead-worker structure, but then how should teams of thousands of agent be organized, communicate, and spend compute? Can agents design the team themselves given task, budget, and deadline? We think so. And so, but that requires, you know, careful investigation of the problem. Now, the second thing is that we think that there must be alternative forms of communication. As I mentioned, we observed 84% of messages going from workers to the lead and only 4% between workers. And so, as I think project outgrow, like coding project, you know, they can outgrow grow one lead, will effective communication here emerge, right? Will scale or like, should it be trained, right? How should the lead adapt to the task? It's not scalable to just

for each task that we have, we spend, you know, a huge amount of time optimizing the prompt and giving very detailed instruction to the lead. That should not be the case. The lead should figure out how to actually, you know, organize the team according to the task that it has. Um now, finally, uh multi-agent safety and security. Um you know, uh this is I can't emphasize on this enough, especially given what happens with the Penny Lane incident. You know, right now we have teams of agents that can run autonomously 24/7 with no supervision. They're quite capable and we're not talking anymore about the single agent, but we talking about teams of agent and even swarm of agent. And so, here the the key question is that how do we secure our systems and our infra? And we need to urgently basically figure out ways on how to align these models.

All right, with that I actually would like to thank you for making it to the end. I want to give a special shout out to the first author here, Kenan and Daniel, for their hard work on this. And for more information, please check out the paper and the blog using these QR codes. And thank you so much for listening.