You can be ambitious without the huge token bill. Here's how.

agents keep getting better and better agents keep getting better and better and better, but they can still make your and better, but they can still make your and better, but they can still make your bill go up. At Dreamforce, I got to ask bill go up. At Dreamforce, I got to ask bill go up. At Dreamforce, I got to ask Mark Beni off a question about how we Mark Beni off a question about how we Mark Beni off a question about how we get more people to adopt AI. And later get more people to adopt AI. And later get more people to adopt AI. And later that day, I was on a panel talking about that day, I was on a panel talking about that day, I was on a panel talking about the cost that we're all facing when we the cost that we're all facing when we the cost that we're all facing when we roll out agents of scale. By the end, roll out agents of scale. By the end, roll out agents of scale. By the end, you're going to know how to look at your you're going to know how to look at your you're going to know how to look at your own workflows and ask which work should own workflows and ask which work should own workflows and ask which work should still exist before you end up still exist before you end up still exist before you end up effectively paying rent for all of these effectively paying rent for all of these effectively paying rent for all of these agents. I'm excited about this one agents. I'm excited about this one agents. I'm excited about this one because I want people to be able to use because I want people to be able to use because I want people to be able to use agents that are bigger. I want them to agents that are bigger. I want them to agents that are bigger. I want them to be able to do more ambitious work be able to do more ambitious work be able to do more ambitious work without feeling like they're carrying without feeling like they're carrying without feeling like they're carrying the burden of a massive financial the burden of a massive financial the burden of a massive financial question mark when they set up AI. We're question mark when they set up AI. We're question mark when they set up AI. We're finally getting to the point where we finally getting to the point where we finally getting to the point where we have that capability. Now, we have a have that capability. Now, we have a have that capability. Now, we have a chance right now to rethink how jobs get chance right now to rethink how jobs get chance right now to rethink how jobs get done, how work gets done, and ultimately done, how work gets done, and ultimately done, how work gets done, and ultimately we have a chance to make the capability we have a chance to make the capability we have a chance to make the capability to do that kind of ambitious work with to do that kind of ambitious work with to do that kind of ambitious work with AI affordable for companies of all AI affordable for companies of all AI affordable for companies of all sizes. So stay with me because the most sizes. So stay with me because the most sizes. So stay with me because the most interesting part of the Dreamforce interesting part of the Dreamforce interesting part of the Dreamforce conversation, at least to me, was what conversation, at least to me, was what conversation, at least to me, was what happens when you go back to the business happens when you go back to the business happens when you go back to the business outcome, when you wipe the whiteboard outcome, when you wipe the whiteboard outcome, when you wipe the whiteboard clean and give yourself permission to clean and give yourself permission to clean and give yourself permission to change how work gets done to get to the change how work gets done to get to the change how work gets done to get to the same business value you've been looking same business value you've been looking same business value you've been looking to achieve all along.

to achieve all along. to achieve all along. >> The changes that you build, the model >> The changes that you build, the model >> The changes that you build, the model that you need, what you're paying for, that you need, what you're paying for, that you need, what you're paying for, all of that comes back and reasons back all of that comes back and reasons back all of that comes back and reasons back from the business value. This year, from the business value. This year, from the business value. This year, Dario Amade, Sam Alman, Jensen Hong, and Dario Amade, Sam Alman, Jensen Hong, and Dario Amade, Sam Alman, Jensen Hong, and others all shared the keynote stage and others all shared the keynote stage and others all shared the keynote stage and they talked about AI safety. They talked they talked about AI safety. They talked they talked about AI safety. They talked about growth, they talked about models, about growth, they talked about models, about growth, they talked about models, they talked about adoption, they talked they talked about adoption, they talked they talked about adoption, they talked about how to get AI in more people's about how to get AI in more people's about how to get AI in more people's hands safely. So, let me start with that hands safely. So, let me start with that hands safely. So, let me start with that adoption question. I was at the media adoption question. I was at the media adoption question. I was at the media Q&A and I asked Mark how he thought Q&A and I asked Mark how he thought Q&A and I asked Mark how he thought about getting different kinds of people about getting different kinds of people about getting different kinds of people to use Agent Force and really the to use Agent Force and really the to use Agent Force and really the question on a lot of our minds which is question on a lot of our minds which is question on a lot of our minds which is how do we help broad swaths of companies how do we help broad swaths of companies how do we help broad swaths of companies actually get excited about get actually get excited about get actually get excited about get passionate about and start adopting AI passionate about and start adopting AI passionate about and start adopting AI in ways that are useful. You have the in ways that are useful. You have the in ways that are useful. You have the early adopters already, right? You've early adopters already, right? You've early adopters already, right? You've got the people in the middle who will got the people in the middle who will got the people in the middle who will probably use Chad GPT a bit here and probably use Chad GPT a bit here and probably use Chad GPT a bit here and there and then you've got people who there and then you've got people who there and then you've got people who would really frankly prefer not to be would really frankly prefer not to be would really frankly prefer not to be using AI or don't understand why it's using AI or don't understand why it's using AI or don't understand why it's valuable or maybe they're going to fight valuable or maybe they're going to fight valuable or maybe they're going to fight changing the software. Every company of changing the software. Every company of changing the software. Every company of any reasonable size has these three any reasonable size has these three any reasonable size has these three kinds of people. So when Mark answered kinds of people. So when Mark answered kinds of people. So when Mark answered the question, the part of his answer the question, the part of his answer the question, the part of his answer that really interested me was how he that really interested me was how he that really interested me was how he pivoted back to the interface and how pivoted back to the interface and how pivoted back to the interface and how interfaces are changing in the age of interfaces are changing in the age of interfaces are changing in the age of AI. When you get agents into a tool like AI. When you get agents into a tool like AI. When you get agents into a tool like Slack, for instance, you're getting it Slack, for instance, you're getting it Slack, for instance, you're getting it into the places where people already into the places where people already into the places where people already work. And I thought that was really work. And I thought that was really work. And I thought that was really perceptive because we spend a ton of perceptive because we spend a ton of perceptive because we spend a ton of time in AI assuming that the person on time in AI assuming that the person on time in AI assuming that the person on the other side is excited to discover a the other side is excited to discover a the other side is excited to discover a new product. But most of us have a job.

new product. But most of us have a job. new product. But most of us have a job. We have a customer waiting. We have We have a customer waiting. We have We have a customer waiting. We have something we need to get out the door. something we need to get out the door. something we need to get out the door. We have no time left. If the AI can help We have no time left. If the AI can help We have no time left. If the AI can help get us where we need to go, right in the get us where we need to go, right in the get us where we need to go, right in the moment, where we are inside the work moment, where we are inside the work moment, where we are inside the work that we're already doing, then you've that we're already doing, then you've that we're already doing, then you've removed a giant reason to not try AI. removed a giant reason to not try AI. removed a giant reason to not try AI. Now, that does sound like a product Now, that does sound like a product Now, that does sound like a product detail. I get that it sounds small, but detail. I get that it sounds small, but detail. I get that it sounds small, but follow it through. Someone tries an follow it through. Someone tries an follow it through. Someone tries an agent because it's already there. Maybe agent because it's already there. Maybe agent because it's already there. Maybe it's living in Slack. It handles it's living in Slack. It handles it's living in Slack. It handles something useful. Well, now why don't something useful. Well, now why don't something useful. Well, now why don't you try it again? It kept a small you try it again? It kept a small you try it again? It kept a small promise for you. Try it again. And this promise for you. Try it again. And this promise for you. Try it again. And this time, you might ask for a little bit time, you might ask for a little bit time, you might ask for a little bit more. You might get a little bit bolder. more. You might get a little bit bolder. more. You might get a little bit bolder. Your colleague sees it. Your manager Your colleague sees it. Your manager Your colleague sees it. Your manager wants to know whether the whole team is wants to know whether the whole team is wants to know whether the whole team is going to be able to use it. And this is going to be able to use it. And this is going to be able to use it. And this is how you start to change. This is how you how you start to change. This is how you how you start to change. This is how you start to move beyond a little group of start to move beyond a little group of start to move beyond a little group of people who are going to find every new people who are going to find every new people who are going to find every new model anyway, right? the 5% and this is model anyway, right? the 5% and this is model anyway, right? the 5% and this is how you get into the whole company and how you get into the whole company and how you get into the whole company and that was the larger shift I was trying that was the larger shift I was trying that was the larger shift I was trying to understand at Dreamforce. I don't to understand at Dreamforce. I don't to understand at Dreamforce. I don't mean that every enterprise has mean that every enterprise has mean that every enterprise has immediately figured this out because immediately figured this out because immediately figured this out because they have Slack. We're all going on this they have Slack. We're all going on this they have Slack. We're all going on this journey together. But in the journey together. But in the journey together. But in the conversations I've been having, we're conversations I've been having, we're conversations I've been having, we're getting into what happens when people getting into what happens when people getting into what happens when people start to go from agents are a novelty to start to go from agents are a novelty to start to go from agents are a novelty to agents are reliable. Agents are agents are reliable. Agents are agents are reliable. Agents are trustworthy. Agents are something you trustworthy. Agents are something you trustworthy. Agents are something you can depend on. [clears throat] can depend on. [clears throat] can depend on. [clears throat] How do you make that work reliable? How How do you make that work reliable? How How do you make that work reliable? How do you connect it into the company's do you connect it into the company's do you connect it into the company's actual information? How do you pay for actual information? How do you pay for actual information? How do you pay for it when a successful experiment becomes

it when a successful experiment becomes it when a successful experiment becomes something a ton of people now want to something a ton of people now want to something a ton of people now want to use every day? So after the Q&A, I was use every day? So after the Q&A, I was use every day? So after the Q&A, I was on a panel hosted by Jes, the EVP of on a panel hosted by Jes, the EVP of on a panel hosted by Jes, the EVP of Agent Force at Salesforce, and we were Agent Force at Salesforce, and we were Agent Force at Salesforce, and we were talking together. There were folks from talking together. There were folks from talking together. There were folks from PWC and Pepsi there as well talking PWC and Pepsi there as well talking PWC and Pepsi there as well talking about AI implementation. My job was just about AI implementation. My job was just about AI implementation. My job was just to talk set macro context and the to talk set macro context and the to talk set macro context and the context I gave starts back in January. context I gave starts back in January. context I gave starts back in January. That's the point I keep coming back to That's the point I keep coming back to That's the point I keep coming back to this year when I think about how models this year when I think about how models this year when I think about how models have become so much more capable. have become so much more capable. have become so much more capable. Capable enough that a whole bunch of Capable enough that a whole bunch of Capable enough that a whole bunch of product work has suddenly looked really product work has suddenly looked really product work has suddenly looked really practical. A whole bunch of business practical. A whole bunch of business practical. A whole bunch of business work has suddenly looked really work has suddenly looked really work has suddenly looked really practical. You can give a model tools practical. You can give a model tools practical. You can give a model tools and it can use them extremely reliably. and it can use them extremely reliably. and it can use them extremely reliably. You can give it information. it can do You can give it information. it can do You can give it information. it can do something with it. Notice it needs something with it. Notice it needs something with it. Notice it needs another piece. Keep going. And unlike another piece. Keep going. And unlike another piece. Keep going. And unlike 2025, 2024, you don't have to break 2025, 2024, you don't have to break 2025, 2024, you don't have to break everything into tiny questions. You everything into tiny questions. You everything into tiny questions. You don't have to sit there kind of lugging don't have to sit there kind of lugging don't have to sit there kind of lugging the answer, copy paste hell from one the answer, copy paste hell from one the answer, copy paste hell from one place to another. And once that begins place to another. And once that begins place to another. And once that begins to work, of course, people use it more.

to work, of course, people use it more. to work, of course, people use it more. That's natural, right? So, continuing That's natural, right? So, continuing That's natural, right? So, continuing the story, we wanted adoption. useful the story, we wanted adoption. useful the story, we wanted adoption. useful models give people a reason to adopt. models give people a reason to adopt. models give people a reason to adopt. But that same capability that unlocked But that same capability that unlocked But that same capability that unlocked for all of us in January changes the for all of us in January changes the for all of us in January changes the size of the job. You used to ask for an size of the job. You used to ask for an size of the job. You used to ask for an answer. Now you're asking the agent to answer. Now you're asking the agent to answer. Now you're asking the agent to investigate a customer problem, to look investigate a customer problem, to look investigate a customer problem, to look through an account, to find relevant through an account, to find relevant through an account, to find relevant history, to check product information, history, to check product information, history, to check product information, and to come back with something that you and to come back with something that you and to come back with something that you can act on. The agent may make a whole can act on. The agent may make a whole can act on. The agent may make a whole series of model calls along the way, series of model calls along the way, series of model calls along the way, right? It's not going to be just one right? It's not going to be just one right? It's not going to be just one thing. It reads more. It uses more thing. It reads more. It uses more thing. It reads more. It uses more tools. It follows up on what it finds. tools. It follows up on what it finds. tools. It follows up on what it finds. There's way more work inside that one There's way more work inside that one There's way more work inside that one request. Now, 100x a,000x more tokens. request. Now, 100x a,000x more tokens. request. Now, 100x a,000x more tokens. So, you have more agents running and now So, you have more agents running and now So, you have more agents running and now you have way, way more tokens per run. you have way, way more tokens per run. you have way, way more tokens per run. And those things double multiply, don't And those things double multiply, don't And those things double multiply, don't they? Just to make the relationship they? Just to make the relationship they? Just to make the relationship really obvious, imagine 10 times as many really obvious, imagine 10 times as many really obvious, imagine 10 times as many runs with, you know, 100 times as many runs with, you know, 100 times as many runs with, you know, 100 times as many tokens per run. you're you're already at tokens per run. you're you're already at tokens per run. you're you're already at a thousandx just there. [clears throat] a thousandx just there. [clears throat] a thousandx just there. [clears throat] >> And those are kind of madeup numbers, >> And those are kind of madeup numbers, >> And those are kind of madeup numbers, but they're kind of not when you look at but they're kind of not when you look at but they're kind of not when you look at actual token consumption. You can get to actual token consumption. You can get to actual token consumption. You can get to a very large increase very quickly a very large increase very quickly a very large increase very quickly without anyone doing anything without anyone doing anything without anyone doing anything particularly strange in the middle of particularly strange in the middle of particularly strange in the middle of 2026. So, we're using a system that got 2026. So, we're using a system that got 2026. So, we're using a system that got better and the costs add up, right?

better and the costs add up, right? better and the costs add up, right? There are frontier models right now that There are frontier models right now that There are frontier models right now that are charging $50 per million output are charging $50 per million output are charging $50 per million output tokens. tokens. tokens. output specifically just getting the output specifically just getting the output specifically just getting the tokens out input cached input those are tokens out input cached input those are tokens out input cached input those are all more prices over the top right but all more prices over the top right but all more prices over the top right but even if we don't dive into the pricing even if we don't dive into the pricing even if we don't dive into the pricing table and get lost there you can see table and get lost there you can see table and get lost there you can see that the problem with sending every that the problem with sending every that the problem with sending every little decision to the most expensive little decision to the most expensive little decision to the most expensive intelligence you can buy is that you intelligence you can buy is that you intelligence you can buy is that you can't control your costs long term and can't control your costs long term and can't control your costs long term and this is where to be very honest with you this is where to be very honest with you this is where to be very honest with you I start to get a little bit frustrated I start to get a little bit frustrated I start to get a little bit frustrated with where the conversation goes with where the conversation goes with where the conversation goes sometimes um because people will often sometimes um because people will often sometimes um because people will often paint it as oh the greedy engineers are paint it as oh the greedy engineers are paint it as oh the greedy engineers are using all the tokens. The greedy using all the tokens. The greedy using all the tokens. The greedy creators are using all the tokens. Okay, creators are using all the tokens. Okay, creators are using all the tokens. Okay, but what models are we giving them? How but what models are we giving them? How but what models are we giving them? How are we enabling them? If you want people are we enabling them? If you want people are we enabling them? If you want people to use AI, you want them to find better to use AI, you want them to find better to use AI, you want them to find better bigger things they can tackle and bigger bigger things they can tackle and bigger bigger things they can tackle and bigger things they can go after. Well, then we things they can go after. Well, then we things they can go after. Well, then we need to equip them with a system that need to equip them with a system that need to equip them with a system that sets them up to do that. Because if sets them up to do that. Because if sets them up to do that. Because if every ordinary request takes a really every ordinary request takes a really every ordinary request takes a really expensive route through a really expensive route through a really expensive route through a really complicated system, that's a decision.

complicated system, that's a decision. complicated system, that's a decision. That's a design decision that someone That's a design decision that someone That's a design decision that someone made about the AI harness and made about the AI harness and made about the AI harness and infrastructure. So you can't hand that infrastructure. So you can't hand that infrastructure. So you can't hand that large decision, that giant design, that large decision, that giant design, that large decision, that giant design, that setting to a person and then act setting to a person and then act setting to a person and then act surprised when they adopt AI and it surprised when they adopt AI and it surprised when they adopt AI and it suddenly costs you the company money. suddenly costs you the company money. suddenly costs you the company money. And I want to stay with that for a And I want to stay with that for a And I want to stay with that for a minute because the obvious response is minute because the obvious response is minute because the obvious response is to go shopping for a cheaper model. And to go shopping for a cheaper model. And to go shopping for a cheaper model. And there's a ton of value in that. We there's a ton of value in that. We there's a ton of value in that. We talked about that a lot on the panel talked about that a lot on the panel talked about that a lot on the panel actually. But before you decide which actually. But before you decide which actually. But before you decide which model should do the work, I want to know model should do the work, I want to know model should do the work, I want to know why we're doing the work in the first why we're doing the work in the first why we're doing the work in the first place. This was a point I made on the place. This was a point I made on the place. This was a point I made on the panel. And that's where the conversation panel. And that's where the conversation panel. And that's where the conversation got a lot more interesting. Think about got a lot more interesting. Think about got a lot more interesting. Think about the old inter office envelope. Maybe for the old inter office envelope. Maybe for the old inter office envelope. Maybe for some of you, you've never seen an some of you, you've never seen an some of you, you've never seen an interoffice envelope. The white hairs interoffice envelope. The white hairs interoffice envelope. The white hairs are showing here. Uh but there was a are showing here. Uh but there was a are showing here. Uh but there was a little interoffice envelope. It had little interoffice envelope. It had little interoffice envelope. It had brown uh sort of a brown envelope with a brown uh sort of a brown envelope with a brown uh sort of a brown envelope with a little bit of string that wraps around little bit of string that wraps around little bit of string that wraps around two little circles. Someone puts a two little circles. Someone puts a two little circles. Someone puts a document in it. They write their name on document in it. They write their name on document in it. They write their name on the front. They send it to another the front. They send it to another the front. They send it to another department. This really happened, guys, department. This really happened, guys, department. This really happened, guys, for those of you who are younger. Uh the for those of you who are younger. Uh the for those of you who are younger. Uh the person opens it. They do their part of person opens it. They do their part of person opens it. They do their part of the document processing. They cross out the document processing. They cross out the document processing. They cross out their name and they send it down to the their name and they send it down to the their name and they send it down to the next department. By the time it gets next department. By the time it gets next department. By the time it gets back to you, the front of the envelope back to you, the front of the envelope back to you, the front of the envelope is like a history of process in the is like a history of process in the is like a history of process in the company. Now, picture that same process company. Now, picture that same process company. Now, picture that same process on the screen. The envelope became an on the screen. The envelope became an on the screen. The envelope became an email, then it became a ticket. Somebody email, then it became a ticket. Somebody email, then it became a ticket. Somebody still has to summarize what the previous still has to summarize what the previous still has to summarize what the previous person said, put it into the format the person said, put it into the format the person said, put it into the format the next team expects and explain why it next team expects and explain why it next team expects and explain why it matters. Now, there may be very, very

matters. Now, there may be very, very matters. Now, there may be very, very good reasons why those steps exist, but good reasons why those steps exist, but good reasons why those steps exist, but some of them exist because at the time some of them exist because at the time some of them exist because at the time that the process was designed, the that the process was designed, the that the process was designed, the people or the systems couldn't share people or the systems couldn't share people or the systems couldn't share what they needed any other way. Now, what they needed any other way. Now, what they needed any other way. Now, fast forward to 2026. You can put an fast forward to 2026. You can put an fast forward to 2026. You can put an agent in every one of those places. An agent in every one of those places. An agent in every one of those places. An agent can read the request. An agent can agent can read the request. An agent can agent can read the request. An agent can reformat the request. An agent can reformat the request. An agent can reformat the request. An agent can prepare the handoff. An agent can check prepare the handoff. An agent can check prepare the handoff. An agent can check the handoff. And it looks incredibly the handoff. And it looks incredibly the handoff. And it looks incredibly modern, doesn't it? There's AI modern, doesn't it? There's AI modern, doesn't it? There's AI everywhere in your company all of a everywhere in your company all of a everywhere in your company all of a sudden. But here's the kicker. You still sudden. But here's the kicker. You still sudden. But here's the kicker. You still may be passing the same envelope around may be passing the same envelope around may be passing the same envelope around the building. You're just doing it with the building. You're just doing it with the building. You're just doing it with agents. You've made each stop faster agents. You've made each stop faster agents. You've made each stop faster without asking why the envelope has to without asking why the envelope has to without asking why the envelope has to visit all of those desks along the way. visit all of those desks along the way. visit all of those desks along the way. This is what I was getting at in the This is what I was getting at in the This is what I was getting at in the panel when I talked about starting with panel when I talked about starting with panel when I talked about starting with a blank sheet of paper. That's what I a blank sheet of paper. That's what I a blank sheet of paper. That's what I meant. If you run a business, your job meant. If you run a business, your job meant. If you run a business, your job is to go back to the handful of things is to go back to the handful of things is to go back to the handful of things that actually create value. I tend to that actually create value. I tend to that actually create value. I tend to think in terms of like five to eight think in terms of like five to eight think in terms of like five to eight major value streams at the CEO level.

major value streams at the CEO level. major value streams at the CEO level. And your business might divide them up And your business might divide them up And your business might divide them up differently. But what I'm looking at is differently. But what I'm looking at is differently. But what I'm looking at is what is the complete path by which you what is the complete path by which you what is the complete path by which you win a customer or deliver what you sold win a customer or deliver what you sold win a customer or deliver what you sold or keep that customer or collect the or keep that customer or collect the or keep that customer or collect the money. Right? Ring the cash register. money. Right? Ring the cash register. money. Right? Ring the cash register. Start there and ask what has to happen Start there and ask what has to happen Start there and ask what has to happen for that result to be extraordinary, for that result to be extraordinary, for that result to be extraordinary, really good. And give yourself really good. And give yourself really good. And give yourself permission to start with that great permission to start with that great permission to start with that great result you want and draw the right path result you want and draw the right path result you want and draw the right path to get there. And it may be a very to get there. And it may be a very to get there. And it may be a very different path from where you're at different path from where you're at different path from where you're at today. Because if you start with every today. Because if you start with every today. Because if you start with every department's existing process, every old department's existing process, every old department's existing process, every old handoff is going to show up from one of handoff is going to show up from one of handoff is going to show up from one of your teams as, "Oh, it's a new your teams as, "Oh, it's a new your teams as, "Oh, it's a new requirement." requirement." requirement." And in that situation, nobody's going to And in that situation, nobody's going to And in that situation, nobody's going to argue to delete it. It's already on the argue to delete it. It's already on the argue to delete it. It's already on the page and everyone's going to be talking page and everyone's going to be talking page and everyone's going to be talking about the wrong thing, which is hiring about the wrong thing, which is hiring about the wrong thing, which is hiring the agent to perform the job as it was the agent to perform the job as it was the agent to perform the job as it was inherited. So, let's make this concrete inherited. So, let's make this concrete inherited. So, let's make this concrete with a customer asking for a quote. now with a customer asking for a quote. now with a customer asking for a quote. now is an imagined example, but it's one is an imagined example, but it's one is an imagined example, but it's one I've certainly lived and a lot of other I've certainly lived and a lot of other I've certainly lived and a lot of other folks have lived. The request comes in, folks have lived. The request comes in, folks have lived. The request comes in, somebody summarizes it for sales ops, somebody summarizes it for sales ops, somebody summarizes it for sales ops, somebody checks the account, somebody somebody checks the account, somebody somebody checks the account, somebody translates it into a product translates it into a product translates it into a product configuration, somebody puts that configuration, somebody puts that configuration, somebody puts that information into the pricing system.

information into the pricing system. information into the pricing system. Then the quote goes back through all Then the quote goes back through all Then the quote goes back through all those people who need to see it, right? those people who need to see it, right? those people who need to see it, right? And maybe product eventually signs off And maybe product eventually signs off And maybe product eventually signs off on it. That's where I was. And the on it. That's where I was. And the on it. That's where I was. And the customer eventually gets an answer. customer eventually gets an answer. customer eventually gets an answer. Well, what did we need at the end? We Well, what did we need at the end? We Well, what did we need at the end? We needed the customer to get an accurate needed the customer to get an accurate needed the customer to get an accurate quote that we're authorized to offer. We quote that we're authorized to offer. We quote that we're authorized to offer. We needed the company to have a record that needed the company to have a record that needed the company to have a record that that quote was generated. And we needed that quote was generated. And we needed that quote was generated. And we needed to ask about anything that we could not to ask about anything that we could not to ask about anything that we could not resolve along the way. And those are resolve along the way. And those are resolve along the way. And those are real requirements. But the summary that real requirements. But the summary that real requirements. But the summary that only existed because one team couldn't only existed because one team couldn't only existed because one team couldn't read the other team's system, that kind read the other team's system, that kind read the other team's system, that kind of work doesn't have a job anymore. So of work doesn't have a job anymore. So of work doesn't have a job anymore. So if you go through that process, the if you go through that process, the if you go through that process, the challenge is to ask yourself what does challenge is to ask yourself what does challenge is to ask yourself what does that final quote look like and then how that final quote look like and then how that final quote look like and then how can we get to that final quote tackling can we get to that final quote tackling can we get to that final quote tackling the complexity we need to tackle with a the complexity we need to tackle with a the complexity we need to tackle with a minimum of handoffs. You can drop minimum of handoffs. You can drop minimum of handoffs. You can drop traditional enterprise steps out traditional enterprise steps out traditional enterprise steps out entirely because the reason for those entirely because the reason for those entirely because the reason for those steps has gone. You never spend the steps has gone. You never spend the steps has gone. You never spend the tokens on them at all. Zero is the best tokens on them at all. Zero is the best tokens on them at all. Zero is the best cost, isn't it?

cost, isn't it? cost, isn't it? And notice what else changes. The agent And notice what else changes. The agent And notice what else changes. The agent doesn't have to read a summary that we doesn't have to read a summary that we doesn't have to read a summary that we didn't produce. It doesn't have to ask didn't produce. It doesn't have to ask didn't produce. It doesn't have to ask another agent what that summary meant. another agent what that summary meant. another agent what that summary meant. Like there's so many fewer points of Like there's so many fewer points of Like there's so many fewer points of failure here. It doesn't have to failure here. It doesn't have to failure here. It doesn't have to reconcile multiple versions of the same reconcile multiple versions of the same reconcile multiple versions of the same request that drifted as they moved request that drifted as they moved request that drifted as they moved through different departments with all through different departments with all through different departments with all these different agent interactions. So these different agent interactions. So these different agent interactions. So the workflow itself can get shorter. You the workflow itself can get shorter. You the workflow itself can get shorter. You get to use the model's ability to get to use the model's ability to get to use the model's ability to understand the whole problem instead of understand the whole problem instead of understand the whole problem instead of asking it to reenact every single asking it to reenact every single asking it to reenact every single handoff people used to make like a bunch handoff people used to make like a bunch handoff people used to make like a bunch of theater kind of going along. And that of theater kind of going along. And that of theater kind of going along. And that this is why we need business people this is why we need business people this is why we need business people involved in agent design systems. An involved in agent design systems. An involved in agent design systems. An engineer can make the step cheaper engineer can make the step cheaper engineer can make the step cheaper absolutely and they can build a really absolutely and they can build a really absolutely and they can build a really great agent to perform whatever work is great agent to perform whatever work is great agent to perform whatever work is needed. But the engineer may not have needed. But the engineer may not have needed. But the engineer may not have the authority to say that sales ops no the authority to say that sales ops no the authority to say that sales ops no longer needs that particular document. longer needs that particular document. longer needs that particular document. Somebody who owns the business result Somebody who owns the business result Somebody who owns the business result has to be able to have that conversation has to be able to have that conversation has to be able to have that conversation across departments and say I want across departments and say I want across departments and say I want simplicity. I want this to be easier.

simplicity. I want this to be easier. simplicity. I want this to be easier. >> Otherwise, every department gets a >> Otherwise, every department gets a >> Otherwise, every department gets a faster version of its own process and faster version of its own process and faster version of its own process and the customer is still waiting for the the customer is still waiting for the the customer is still waiting for the envelope to come back and we're just envelope to come back and we're just envelope to come back and we're just passing it around the building, right? passing it around the building, right? passing it around the building, right? But we're doing it with agents and But we're doing it with agents and But we're doing it with agents and calling it progress. Of course, the calling it progress. Of course, the calling it progress. Of course, the discount approval doesn't disappear just discount approval doesn't disappear just discount approval doesn't disappear just because the AI is smart. The price still because the AI is smart. The price still because the AI is smart. The price still has to be right, right? You still need a has to be right, right? You still need a has to be right, right? You still need a record of what you offered. Looking at record of what you offered. Looking at record of what you offered. Looking at the whole process lets you separate out the whole process lets you separate out the whole process lets you separate out all of those particular requirements all of those particular requirements all of those particular requirements from the admin work that kind of grew up from the admin work that kind of grew up from the admin work that kind of grew up like coral kind of accumulating and like coral kind of accumulating and like coral kind of accumulating and growing into a reef around the actual growing into a reef around the actual growing into a reef around the actual value. And when you remove some of that value. And when you remove some of that value. And when you remove some of that admin work, you know what's great about admin work, you know what's great about admin work, you know what's great about it? It's good for us as people. The team it? It's good for us as people. The team it? It's good for us as people. The team gets time back. Whether that becomes gets time back. Whether that becomes gets time back. Whether that becomes more sales or or better service or a more sales or or better service or a more sales or or better service or a less overloaded team, it's going to less overloaded team, it's going to less overloaded team, it's going to depend on what you do with the time. But depend on what you do with the time. But depend on what you do with the time. But it's worth trying for. It doesn't mean it's worth trying for. It doesn't mean it's worth trying for. It doesn't mean it's an automatic reduction in payroll it's an automatic reduction in payroll it's an automatic reduction in payroll or anything dramatic like that. In fact, or anything dramatic like that. In fact, or anything dramatic like that. In fact, I had another conversation at Dreamforce I had another conversation at Dreamforce I had another conversation at Dreamforce that helped connect the dots on this for that helped connect the dots on this for that helped connect the dots on this for me. I spoke with Rohan Kumar, me. I spoke with Rohan Kumar, me. I spoke with Rohan Kumar, Salesforce's president and chief Salesforce's president and chief Salesforce's president and chief platform and engineering officer about platform and engineering officer about platform and engineering officer about what he's been seeing as he's actually what he's been seeing as he's actually what he's been seeing as he's actually been rolling out agents with customers, been rolling out agents with customers, been rolling out agents with customers, talking about a lot of what we've just talking about a lot of what we've just talking about a lot of what we've just been talking about in this video. One of been talking about in this video. One of been talking about in this video. One of the things we both really resonated with the things we both really resonated with the things we both really resonated with was the idea that what do you do with AI was the idea that what do you do with AI was the idea that what do you do with AI once you're past the AI can do my email once you're past the AI can do my email once you're past the AI can do my email stage? People have tried the email stage? People have tried the email stage? People have tried the email assistant. They've had the experience of assistant. They've had the experience of assistant. They've had the experience of getting to draft faster with whatever getting to draft faster with whatever getting to draft faster with whatever tool they have, co-pilot. Right now,

tool they have, co-pilot. Right now, tool they have, co-pilot. Right now, they want to know what comes next. And they want to know what comes next. And they want to know what comes next. And his emphasis is really been around his emphasis is really been around his emphasis is really been around building with specific customers and the building with specific customers and the building with specific customers and the work that they actually need to get work that they actually need to get work that they actually need to get done. How do we embed with them, right? done. How do we embed with them, right? done. How do we embed with them, right? How can Salesforce dig in with them, How can Salesforce dig in with them, How can Salesforce dig in with them, figure out what's working, and figure figure out what's working, and figure figure out what's working, and figure out how to scale? He's also been out how to scale? He's also been out how to scale? He's also been thinking a lot about how you thinking a lot about how you thinking a lot about how you debottleneck product and technical work, debottleneck product and technical work, debottleneck product and technical work, which I love nerding out on. Um because which I love nerding out on. Um because which I love nerding out on. Um because think about it, if you run an think about it, if you run an think about it, if you run an engineering org, if AI helps you produce engineering org, if AI helps you produce engineering org, if AI helps you produce code faster, you're still going to have code faster, you're still going to have code faster, you're still going to have to merge it. You're still going to have to merge it. You're still going to have to merge it. You're still going to have to check it. You're still going to have to check it. You're still going to have to check it. You're still going to have to get it deployed in ways that yes, to get it deployed in ways that yes, to get it deployed in ways that yes, customers can use. And speeding up one customers can use. And speeding up one customers can use. And speeding up one of those activities doesn't mean that of those activities doesn't mean that of those activities doesn't mean that your whole org can suddenly move at that your whole org can suddenly move at that your whole org can suddenly move at that speed. You have to be really thoughtful speed. You have to be really thoughtful speed. You have to be really thoughtful and logical about getting faster. You and logical about getting faster. You and logical about getting faster. You have to follow the work far enough to have to follow the work far enough to have to follow the work far enough to see where it gets stuck next. The value see where it gets stuck next. The value see where it gets stuck next. The value of an agent isn't exhausted by making of an agent isn't exhausted by making of an agent isn't exhausted by making the existing activity faster. If it the existing activity faster. If it the existing activity faster. If it changes what one system can understand changes what one system can understand changes what one system can understand and do, it should give you a reason to and do, it should give you a reason to and do, it should give you a reason to reconsider the rest of your process, reconsider the rest of your process, reconsider the rest of your process, which is exactly kind of where Rohan and which is exactly kind of where Rohan and which is exactly kind of where Rohan and I were talking back and forth. And I I were talking back and forth. And I I were talking back and forth. And I think that's a much more serious think that's a much more serious think that's a much more serious conversation. It's a much more demanding conversation. It's a much more demanding conversation. It's a much more demanding conversation for companies than just conversation for companies than just conversation for companies than just buying access to a better model. But buying access to a better model. But buying access to a better model. But it's also a much more fruitful it's also a much more fruitful it's also a much more fruitful conversation. It means looking at work conversation. It means looking at work conversation. It means looking at work that people are used to doing and asking that people are used to doing and asking that people are used to doing and asking whether that work is just busy work now.

whether that work is just busy work now. whether that work is just busy work now. And once you've done that, thinking And once you've done that, thinking And once you've done that, thinking through your model choice starts to make through your model choice starts to make through your model choice starts to make a lot more sense because the remaining a lot more sense because the remaining a lot more sense because the remaining work is not all equally difficult. Some work is not all equally difficult. Some work is not all equally difficult. Some of it is a calculation with a known of it is a calculation with a known of it is a calculation with a known rule. Some of it is interpreting what rule. Some of it is interpreting what rule. Some of it is interpreting what the customer meant. And some of it is a the customer meant. And some of it is a the customer meant. And some of it is a messy exception, right? where the messy exception, right? where the messy exception, right? where the information conflicts and somebody needs information conflicts and somebody needs information conflicts and somebody needs to investigate. There's an edge case to investigate. There's an edge case to investigate. There's an edge case involved. Why? Why would we expect all involved. Why? Why would we expect all involved. Why? Why would we expect all of the different use cases I just of the different use cases I just of the different use cases I just described to need the same amount of described to need the same amount of described to need the same amount of intelligence? They don't. They don't. intelligence? They don't. They don't. intelligence? They don't. They don't. And this came up in my conversation with And this came up in my conversation with And this came up in my conversation with the head of agent forces engineering JSH the head of agent forces engineering JSH the head of agent forces engineering JSH as well. There's this continuum of work as well. There's this continuum of work as well. There's this continuum of work between where the answer follows a rule, between where the answer follows a rule, between where the answer follows a rule, where it's relatively deterministic and where it's relatively deterministic and where it's relatively deterministic and where the work may need interpretation, where the work may need interpretation, where the work may need interpretation, where it might be more probabilistic. where it might be more probabilistic. where it might be more probabilistic. When people first get excited about AI, When people first get excited about AI, When people first get excited about AI, they tend to push everything into the they tend to push everything into the they tend to push everything into the model and hey, it's amazing. It's model and hey, it's amazing. It's model and hey, it's amazing. It's miraculous. It's fantastic. It generates miraculous. It's fantastic. It generates miraculous. It's fantastic. It generates tokens. Let's have it do all the things, tokens. Let's have it do all the things, tokens. Let's have it do all the things, right? But a model can do all of those right? But a model can do all of those right? But a model can do all of those things and it can actually be incorrect things and it can actually be incorrect things and it can actually be incorrect if you're not basing it on the right if you're not basing it on the right if you're not basing it on the right data. A model can be more correct data. A model can be more correct data. A model can be more correct cheaper with an open-source open weights cheaper with an open-source open weights cheaper with an open-source open weights option if it's calling the right data.

option if it's calling the right data. option if it's calling the right data. If you're able to just use the model to If you're able to just use the model to If you're able to just use the model to coordinate work to call ordinary coordinate work to call ordinary coordinate work to call ordinary software like a CRM to do all of the software like a CRM to do all of the software like a CRM to do all of the parts that are already settled, you can parts that are already settled, you can parts that are already settled, you can save tokens. you can get more save tokens. you can get more save tokens. you can get more predictable work at the same time. And predictable work at the same time. And predictable work at the same time. And in the quote example we talked about, a in the quote example we talked about, a in the quote example we talked about, a pricing tool could apply pricing rules pricing tool could apply pricing rules pricing tool could apply pricing rules and you wouldn't have to use an LLM to and you wouldn't have to use an LLM to and you wouldn't have to use an LLM to get probabilistic pricing. The model get probabilistic pricing. The model get probabilistic pricing. The model doesn't need to rediscover arithmetic in doesn't need to rediscover arithmetic in doesn't need to rediscover arithmetic in that case, right? And all the model may that case, right? And all the model may that case, right? And all the model may need to do is understand the customer's need to do is understand the customer's need to do is understand the customer's language well enough to identify the language well enough to identify the language well enough to identify the product, recognize what's missing, and product, recognize what's missing, and product, recognize what's missing, and explain the answer that was derived from explain the answer that was derived from explain the answer that was derived from a tool call back to the to the CRM. And a tool call back to the to the CRM. And a tool call back to the to the CRM. And for a lot of routine enterprise work, for a lot of routine enterprise work, for a lot of routine enterprise work, the amount of intelligence required to the amount of intelligence required to the amount of intelligence required to do that kind of job, you don't need to do that kind of job, you don't need to do that kind of job, you don't need to worry about the frontier models anymore. worry about the frontier models anymore. worry about the frontier models anymore. We're we're so far past that. If you can We're we're so far past that. If you can We're we're so far past that. If you can solve a, you know, millennium prize solve a, you know, millennium prize solve a, you know, millennium prize problem like Navier Stokes, you are well problem like Navier Stokes, you are well problem like Navier Stokes, you are well past the intelligence level you need to past the intelligence level you need to past the intelligence level you need to update your CRM.

update your CRM. update your CRM. you are well past needing to figure out you are well past needing to figure out you are well past needing to figure out which model to use to figure out what which model to use to figure out what which model to use to figure out what products someone mentioned in an email. products someone mentioned in an email. products someone mentioned in an email. You need a model that does that job You need a model that does that job You need a model that does that job reliably with the right information, the reliably with the right information, the reliably with the right information, the right tools around it. And I'm so right tools around it. And I'm so right tools around it. And I'm so excited. I'm I'm excited about the fancy excited. I'm I'm excited about the fancy excited. I'm I'm excited about the fancy models, right? I'm excited about models models, right? I'm excited about models models, right? I'm excited about models that can help with extraordinary that can help with extraordinary that can help with extraordinary scientific problems. I believe we are scientific problems. I believe we are scientific problems. I believe we are early on the intelligence curve. There's early on the intelligence curve. There's early on the intelligence curve. There's a lot of good intelligent work to be a lot of good intelligent work to be a lot of good intelligent work to be discovered up ahead. I just don't need discovered up ahead. I just don't need discovered up ahead. I just don't need to pay for that level of capability to pay for that level of capability to pay for that level of capability every time a customer asks an ordinary every time a customer asks an ordinary every time a customer asks an ordinary business question. That's why good business question. That's why good business question. That's why good openweight models are so interesting to openweight models are so interesting to openweight models are so interesting to me along with cheaper hosted models me along with cheaper hosted models me along with cheaper hosted models because there's a lot of work where a because there's a lot of work where a because there's a lot of work where a less expensive model set up well is just less expensive model set up well is just less expensive model set up well is just frankly enough. Open weights aren't frankly enough. Open weights aren't frankly enough. Open weights aren't automatically cheaper to operate and a automatically cheaper to operate and a automatically cheaper to operate and a small model isn't automatically the small model isn't automatically the small model isn't automatically the right one, but often times that's the right one, but often times that's the right one, but often times that's the case. the opportunity is there to fit case. the opportunity is there to fit case. the opportunity is there to fit the system to a task whose requirements the system to a task whose requirements the system to a task whose requirements you actually understand. And so I think you actually understand. And so I think you actually understand. And so I think we need to stop asking ourselves which we need to stop asking ourselves which we need to stop asking ourselves which model wins everything. I think that's model wins everything. I think that's model wins everything. I think that's the wrong question. That's a 2025 the wrong question. That's a 2025 the wrong question. That's a 2025 question. We need to start asking which question. We need to start asking which question. We need to start asking which model can do this job well. On the model can do this job well. On the model can do this job well. On the panel, JS responded to the blank sheet panel, JS responded to the blank sheet panel, JS responded to the blank sheet and the model routing ideas by talking and the model routing ideas by talking and the model routing ideas by talking about classifiers and evals. Everyone's about classifiers and evals. Everyone's about classifiers and evals. Everyone's favorite subject, right? A classifier is favorite subject, right? A classifier is favorite subject, right? A classifier is the part of the system that recognizes

the part of the system that recognizes the part of the system that recognizes what kind of request has arrived and what kind of request has arrived and what kind of request has arrived and sends it to the appropriate place. This sends it to the appropriate place. This sends it to the appropriate place. This is how you actually make the distinction is how you actually make the distinction is how you actually make the distinction I'm talking about. We just talked about I'm talking about. We just talked about I'm talking about. We just talked about making this useful in practice. The making this useful in practice. The making this useful in practice. The standard customer request would be standard customer request would be standard customer request would be classified and sent to an open weights classified and sent to an open weights classified and sent to an open weights model. the unusual case, the edge case model. the unusual case, the edge case model. the unusual case, the edge case where the facts don't fit or the where the facts don't fit or the where the facts don't fit or the consequences require more care. That consequences require more care. That consequences require more care. That goes to a fancier model, doesn't it? My goes to a fancier model, doesn't it? My goes to a fancier model, doesn't it? My ambition here is to reserve the frontier ambition here is to reserve the frontier ambition here is to reserve the frontier model for the hardest problems. That's model for the hardest problems. That's model for the hardest problems. That's where a lot of us think this is going. where a lot of us think this is going. where a lot of us think this is going. And and in that situation, we want And and in that situation, we want And and in that situation, we want workflows and classifiers that allow us workflows and classifiers that allow us workflows and classifiers that allow us to support that reliably. In fact, we to support that reliably. In fact, we to support that reliably. In fact, we may put the smart model in the may put the smart model in the may put the smart model in the classifier just to do that. And that's classifier just to do that. And that's classifier just to do that. And that's something to work on. That's something something to work on. That's something something to work on. That's something to test on. It's not like finding 99% of to test on. It's not like finding 99% of to test on. It's not like finding 99% of the enterprise work is easy. It's the enterprise work is easy. It's the enterprise work is easy. It's actually quite challenging because it actually quite challenging because it actually quite challenging because it has to be the right 99%. has to be the right 99%. has to be the right 99%. But if you can identify the ordinary But if you can identify the ordinary But if you can identify the ordinary work reliably, you can give a lot of work reliably, you can give a lot of work reliably, you can give a lot of people access to super useful people access to super useful people access to super useful intelligence at a very very different intelligence at a very very different intelligence at a very very different cost than we assume AI costs. It becomes cost than we assume AI costs. It becomes cost than we assume AI costs. It becomes transformative.

transformative. transformative. And then the expensive model is And then the expensive model is And then the expensive model is absolutely there when the job calls for absolutely there when the job calls for absolutely there when the job calls for it. There there is a catch here though it. There there is a catch here though it. There there is a catch here though and this is a big part of what we and this is a big part of what we and this is a big part of what we discussed. You can't assume that the discussed. You can't assume that the discussed. You can't assume that the setup around the model should stay the setup around the model should stay the setup around the model should stay the same when you change the model. We were same when you change the model. We were same when you change the model. We were talking about thin harnesses and thick talking about thin harnesses and thick talking about thin harnesses and thick harnesses on the panel. Now the harness harnesses on the panel. Now the harness harnesses on the panel. Now the harness is and Jo advocated for this. I I fully is and Jo advocated for this. I I fully is and Jo advocated for this. I I fully agree. It's the whole arrangement around agree. It's the whole arrangement around agree. It's the whole arrangement around the model. It's not it's not a narrow the model. It's not it's not a narrow the model. It's not it's not a narrow definition of harness. It's a rich definition of harness. It's a rich definition of harness. It's a rich definition of harness. The instructions, definition of harness. The instructions, definition of harness. The instructions, the tools, the information it can use, the tools, the information it can use, the tools, the information it can use, the checks, the way it moves through a the checks, the way it moves through a the checks, the way it moves through a job. A less capable model often needs job. A less capable model often needs job. A less capable model often needs more of that harness structure to do its more of that harness structure to do its more of that harness structure to do its job predictably. And when you think job predictably. And when you think job predictably. And when you think about the quote example, you may end up about the quote example, you may end up about the quote example, you may end up giving that entire model a very specific giving that entire model a very specific giving that entire model a very specific job. Identify the product, the quantity, job. Identify the product, the quantity, job. Identify the product, the quantity, what the customer hasn't told us. The what the customer hasn't told us. The what the customer hasn't told us. The software is able to deterministically software is able to deterministically software is able to deterministically check what fields in the CRM are check what fields in the CRM are check what fields in the CRM are present. It's able to call a pricing present. It's able to call a pricing present. It's able to call a pricing tool. It's able to enforce an approval tool. It's able to enforce an approval tool. It's able to enforce an approval requirement. The model has useful work requirement. The model has useful work requirement. The model has useful work to do there, but the surrounding system to do there, but the surrounding system to do there, but the surrounding system has to have a harness that makes the has to have a harness that makes the has to have a harness that makes the path to that data really really clear.

path to that data really really clear. path to that data really really clear. It frankly it implies a thicker harness It frankly it implies a thicker harness It frankly it implies a thicker harness >> and the thickness comes from structure. >> and the thickness comes from structure. >> and the thickness comes from structure. It doesn't have to mean a gigantic pro. It doesn't have to mean a gigantic pro. It doesn't have to mean a gigantic pro. Whereas if you're going on that Whereas if you're going on that Whereas if you're going on that difficult exception, that 5% that 1% of difficult exception, that 5% that 1% of difficult exception, that 5% that 1% of cases, I want to give the frontier model cases, I want to give the frontier model cases, I want to give the frontier model freedom. And I found this by experience freedom. And I found this by experience freedom. And I found this by experience working with frontier models myself. If working with frontier models myself. If working with frontier models myself. If you have a challenging problem for it, you have a challenging problem for it, you have a challenging problem for it, you don't want to limit it. You want to you don't want to limit it. You want to you don't want to limit it. You want to be able to say, "Here's the customer be able to say, "Here's the customer be able to say, "Here's the customer problem. Here's the evidence. Here are a problem. Here's the evidence. Here are a problem. Here's the evidence. Here are a few tools, general purpose tools. Here's few tools, general purpose tools. Here's few tools, general purpose tools. Here's what a good result has to satisfy. Now, what a good result has to satisfy. Now, what a good result has to satisfy. Now, please work through it. Figure it out." please work through it. Figure it out." please work through it. Figure it out." And that's a thin harness, right? If I And that's a thin harness, right? If I And that's a thin harness, right? If I force the model to follow every little force the model to follow every little force the model to follow every little reasoning step I wrote for a weaker reasoning step I wrote for a weaker reasoning step I wrote for a weaker model, I get in the way of the IQ points model, I get in the way of the IQ points model, I get in the way of the IQ points I'm paying for. So, I want a thinner I'm paying for. So, I want a thinner I'm paying for. So, I want a thinner harness around the smart model doing harness around the smart model doing harness around the smart model doing investigative work, while I want lots of investigative work, while I want lots of investigative work, while I want lots of like structure, a good thick harness like structure, a good thick harness like structure, a good thick harness around the open weights model that I'm around the open weights model that I'm around the open weights model that I'm using at scale to run my business using at scale to run my business using at scale to run my business process. Ultimately, the setup needs to process. Ultimately, the setup needs to process. Ultimately, the setup needs to evolve with the model, just as the evolve with the model, just as the evolve with the model, just as the business process needs to evolve with business process needs to evolve with business process needs to evolve with what the system can do. Today, if you what the system can do. Today, if you what the system can do. Today, if you want to get into the everyday token want to get into the everyday token want to get into the everyday token waste inside a setup like that, I've put waste inside a setup like that, I've put waste inside a setup like that, I've put together 15 different changes and a together 15 different changes and a together 15 different changes and a token saver skill guide on the Substack token saver skill guide on the Substack token saver skill guide on the Substack going with this video. And it goes going with this video. And it goes going with this video. And it goes beyond what I can cover here. It covers beyond what I can cover here. It covers beyond what I can cover here. It covers things like carrying forward the things like carrying forward the things like carrying forward the accepted result and selecting the accepted result and selecting the accepted result and selecting the information the model needs. uh and information the model needs. uh and information the model needs. uh and really the dream force conversation really the dream force conversation really the dream force conversation pushed me to think about the same

pushed me to think about the same pushed me to think about the same problem at the scale of the business problem at the scale of the business problem at the scale of the business process. How much work are we carrying process. How much work are we carrying process. How much work are we carrying forward simply because it's what the old forward simply because it's what the old forward simply because it's what the old system needed. And this is where eval system needed. And this is where eval system needed. And this is where eval becomes a part of the larger story. You becomes a part of the larger story. You becomes a part of the larger story. You can't you can't tell whether a can't you can't tell whether a can't you can't tell whether a redesigned process works just because redesigned process works just because redesigned process works just because the agent sounds pleased with itself. the agent sounds pleased with itself. the agent sounds pleased with itself. Really, you can't always tell just Really, you can't always tell just Really, you can't always tell just because the human is happy. You can't because the human is happy. You can't because the human is happy. You can't tell when the cheaper model's worth it tell when the cheaper model's worth it tell when the cheaper model's worth it because it produced something that because it produced something that because it produced something that looked approximately right. You have to looked approximately right. You have to looked approximately right. You have to actually know. You have to actually actually know. You have to actually actually know. You have to actually know. And one of the metaphors I loved know. And one of the metaphors I loved know. And one of the metaphors I loved about EVELs, it came out of the panel about EVELs, it came out of the panel about EVELs, it came out of the panel discussion was this idea that we as discussion was this idea that we as discussion was this idea that we as humans all have annual evaluations at humans all have annual evaluations at humans all have annual evaluations at work, right? We evaluate people. We ask work, right? We evaluate people. We ask work, right? We evaluate people. We ask whether they've done their job well. Why whether they've done their job well. Why whether they've done their job well. Why would we give an agent real would we give an agent real would we give an agent real responsibility in the company and never responsibility in the company and never responsibility in the company and never evaluate it? We would, right? Obviously, evaluate it? We would, right? Obviously, evaluate it? We would, right? Obviously, an agent doing thousands of jobs needs an agent doing thousands of jobs needs an agent doing thousands of jobs needs feedback much more often than once a feedback much more often than once a feedback much more often than once a year. But regardless of the metaphor you year. But regardless of the metaphor you year. But regardless of the metaphor you use, that idea, the idea of evaluating use, that idea, the idea of evaluating use, that idea, the idea of evaluating an agent needs to become familiar to an agent needs to become familiar to an agent needs to become familiar to you. It's an important part of what you. It's an important part of what you. It's an important part of what we're talking about here. The agent has we're talking about here. The agent has we're talking about here. The agent has a job and we need to find a way to know a job and we need to find a way to know a job and we need to find a way to know whether it did that job well. And with whether it did that job well. And with whether it did that job well. And with the quote example, we've been following the quote example, we've been following the quote example, we've been following through. That means checking that the through. That means checking that the through. That means checking that the actual price is correct against the actual price is correct against the actual price is correct against the pricing system, checking that the pricing system, checking that the pricing system, checking that the approval happened, checking that the approval happened, checking that the approval happened, checking that the record changed. It also means record changed. It also means record changed. It also means recognizing when asking the customer a

recognizing when asking the customer a recognizing when asking the customer a question was the right next action even question was the right next action even question was the right next action even though the quote wasn't finished yet, though the quote wasn't finished yet, though the quote wasn't finished yet, which is a little bit more of a which is a little bit more of a which is a little bit more of a complicated use case. And yes, some complicated use case. And yes, some complicated use case. And yes, some parts will need judgment, too. Did the parts will need judgment, too. Did the parts will need judgment, too. Did the explanation actually address what the explanation actually address what the explanation actually address what the customer was worried about? someone who customer was worried about? someone who customer was worried about? someone who knows the work has to help define the knows the work has to help define the knows the work has to help define the eval in that situation. That's super eval in that situation. That's super eval in that situation. That's super normal. A a technically valid document, normal. A a technically valid document, normal. A a technically valid document, does that actually contain the right does that actually contain the right does that actually contain the right answer or is it technically valid and answer or is it technically valid and answer or is it technically valid and still incorrect? And this is why I keep still incorrect? And this is why I keep still incorrect? And this is why I keep saying eval are a human skill as much as saying eval are a human skill as much as saying eval are a human skill as much as an agent skill. In fact, I think one of an agent skill. In fact, I think one of an agent skill. In fact, I think one of the most important things that we can do the most important things that we can do the most important things that we can do is teach people how to write evals. It's is teach people how to write evals. It's is teach people how to write evals. It's such a scalable skill in 2026 such a scalable skill in 2026 such a scalable skill in 2026 because the agent has to be able to use because the agent has to be able to use because the agent has to be able to use those checks, those eval to learn when those checks, those eval to learn when those checks, those eval to learn when something didn't work to make a useful something didn't work to make a useful something didn't work to make a useful correction. If it's missing information, correction. If it's missing information, correction. If it's missing information, it needs to know what's missing. If the it needs to know what's missing. If the it needs to know what's missing. If the work is complete, it should know it's work is complete, it should know it's work is complete, it should know it's complete. Otherwise, you end up paying complete. Otherwise, you end up paying complete. Otherwise, you end up paying for a very long run and you don't even for a very long run and you don't even for a very long run and you don't even know what's worth it, right? You don't know what's worth it, right? You don't know what's worth it, right? You don't know if it's valuable. So wrapping it know if it's valuable. So wrapping it know if it's valuable. So wrapping it all up, by the time I was putting my all up, by the time I was putting my all up, by the time I was putting my notes together, you know, after the notes together, you know, after the notes together, you know, after the first day or so of Dreamforce, the cost first day or so of Dreamforce, the cost first day or so of Dreamforce, the cost conversation ended up being connected to conversation ended up being connected to conversation ended up being connected to a much larger story for me. Ultimately, a much larger story for me. Ultimately, a much larger story for me. Ultimately, models have gotten useful enough that models have gotten useful enough that models have gotten useful enough that more of us have wanted to start using more of us have wanted to start using more of us have wanted to start using them. They've gotten capable enough that them. They've gotten capable enough that them. They've gotten capable enough that they can run longer and do more. That's they can run longer and do more. That's they can run longer and do more. That's one of the big themes of the conference one of the big themes of the conference one of the big themes of the conference here this year. And now that same

here this year. And now that same here this year. And now that same capability is giving us the chance to capability is giving us the chance to capability is giving us the chance to remove work and to redraw the path to a remove work and to redraw the path to a remove work and to redraw the path to a business result. So the consumption is business result. So the consumption is business result. So the consumption is growing in a two-fold way. More people, growing in a two-fold way. More people, growing in a two-fold way. More people, more agent tokens, but our response to more agent tokens, but our response to more agent tokens, but our response to it can compound as well. A shorter it can compound as well. A shorter it can compound as well. A shorter process leaves less work to pay for and process leaves less work to pay for and process leaves less work to pay for and matching the remaining work to the right matching the remaining work to the right matching the remaining work to the right model gives us that double savings and model gives us that double savings and model gives us that double savings and we can make the work cheaper. Again, I we can make the work cheaper. Again, I we can make the work cheaper. Again, I don't have a universal percentage to don't have a universal percentage to don't have a universal percentage to promise you because frankly, your promise you because frankly, your promise you because frankly, your business is going to have to figure this business is going to have to figure this business is going to have to figure this out for itself, but what I would suggest out for itself, but what I would suggest out for itself, but what I would suggest is look beyond the model build. is look beyond the model build. is look beyond the model build. Ultimately, if your team can serve more Ultimately, if your team can serve more Ultimately, if your team can serve more customers, if you can get them a correct customers, if you can get them a correct customers, if you can get them a correct answer sooner, if you can spend less answer sooner, if you can spend less answer sooner, if you can spend less time repairing handoffs, and those are time repairing handoffs, and those are time repairing handoffs, and those are pieces of real business value, right? pieces of real business value, right? pieces of real business value, right? Well, in that case, even if the AI bill Well, in that case, even if the AI bill Well, in that case, even if the AI bill ends up growing a bit because you're ends up growing a bit because you're ends up growing a bit because you're using AI so much, still perfectly using AI so much, still perfectly using AI so much, still perfectly reasonable, a ton of value there. Now, reasonable, a ton of value there. Now, reasonable, a ton of value there. Now, if the AI bill grew because six agents if the AI bill grew because six agents if the AI bill grew because six agents were writing reports nobody was reading, were writing reports nobody was reading, were writing reports nobody was reading, different situation. So, I want you to different situation. So, I want you to different situation. So, I want you to look beyond the token leaderboard. Both look beyond the token leaderboard. Both look beyond the token leaderboard. Both of the examples I just gave to you can of the examples I just gave to you can of the examples I just gave to you can look like lots of tokens on the look like lots of tokens on the look like lots of tokens on the dashboard. And this takes me all the way dashboard. And this takes me all the way dashboard. And this takes me all the way back to the beginning and to Mark's back to the beginning and to Mark's back to the beginning and to Mark's answer in Q&A about getting AI into the answer in Q&A about getting AI into the answer in Q&A about getting AI into the places that people already work. I want places that people already work. I want places that people already work. I want that to succeed. I want ordinary people that to succeed. I want ordinary people that to succeed. I want ordinary people to have access to these systems without to have access to these systems without to have access to these systems without feeling like every question is a feeling like every question is a feeling like every question is a financial mistake. But once we make financial mistake. But once we make financial mistake. But once we make agents easy to use, the people designing agents easy to use, the people designing agents easy to use, the people designing the systems have a responsibility to the systems have a responsibility to the systems have a responsibility to make that use sensible for the business.

make that use sensible for the business. make that use sensible for the business. Give people tools that let them complete Give people tools that let them complete Give people tools that let them complete the job and then don't risk slap them the job and then don't risk slap them the job and then don't risk slap them when they use them. Give ordinary work when they use them. Give ordinary work when they use them. Give ordinary work an affordable path. Let the model spend an affordable path. Let the model spend an affordable path. Let the model spend its intelligence where understanding is its intelligence where understanding is its intelligence where understanding is actually required. That's the part of actually required. That's the part of actually required. That's the part of Dreamforce I want you to take back to Dreamforce I want you to take back to Dreamforce I want you to take back to your own work. The conversations I had your own work. The conversations I had your own work. The conversations I had were getting into what comes after that were getting into what comes after that were getting into what comes after that first useful experience, that aha moment first useful experience, that aha moment first useful experience, that aha moment with AI, after the email draft, right? with AI, after the email draft, right? with AI, after the email draft, right? After somebody tries an agent and says, After somebody tries an agent and says, After somebody tries an agent and says, "Okay, wow, this could actually help." "Okay, wow, this could actually help." "Okay, wow, this could actually help." That's when the old process becomes a That's when the old process becomes a That's when the old process becomes a choice you have to look at again. So, if choice you have to look at again. So, if choice you have to look at again. So, if you've got a workflow in front of you you've got a workflow in front of you you've got a workflow in front of you right now, I'm challenging you to look right now, I'm challenging you to look right now, I'm challenging you to look at it. Look all the way through to the at it. Look all the way through to the at it. Look all the way through to the person waiting for the results. Ask why person waiting for the results. Ask why person waiting for the results. Ask why each piece of work has to happen for each piece of work has to happen for each piece of work has to happen for that person to get what they need. You that person to get what they need. You that person to get what they need. You may find that the problem deserves a may find that the problem deserves a may find that the problem deserves a much more capable model, but I'm betting much more capable model, but I'm betting much more capable model, but I'm betting you, you're more likely to find that a you, you're more likely to find that a you, you're more likely to find that a cheaper model with good tools can handle cheaper model with good tools can handle cheaper model with good tools can handle it. And you may find that a whole it. And you may find that a whole it. And you may find that a whole stretch of work that you thought was stretch of work that you thought was stretch of work that you thought was essential to get the envelope onto the essential to get the envelope onto the essential to get the envelope onto the desk. Maybe it's maybe it's actually desk. Maybe it's maybe it's actually desk. Maybe it's maybe it's actually not. And now you have the ability to not. And now you have the ability to not. And now you have the ability to stop a bunch of those processes. So stop a bunch of those processes. So stop a bunch of those processes. So those are my takeaways from the first those are my takeaways from the first those are my takeaways from the first day or so of Dreamforce here. I'm day or so of Dreamforce here. I'm day or so of Dreamforce here. I'm excited to go back into it, continue excited to go back into it, continue excited to go back into it, continue having conversations, and I'd love to having conversations, and I'd love to having conversations, and I'd love to hear from you in the comments. What are hear from you in the comments. What are hear from you in the comments. What are you seeing at work with agents that you seeing at work with agents that you seeing at work with agents that allows you to save tokens, you to save allows you to save tokens, you to save allows you to save tokens, you to save process? How are you cutting your token process? How are you cutting your token process? How are you cutting your token bills, but still using AI more often?

bills, but still using AI more often? bills, but still using AI more often? I'll see you next time.