Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

Hello everyone. Um, I would like to walk Hello everyone. Um, I would like to walk you through that how we can transition you through that how we can transition you through that how we can transition in AI agents from an experiments to the in AI agents from an experiments to the in AI agents from an experiments to the real enterprise environment. So, before real enterprise environment. So, before real enterprise environment. So, before I begin, a quick introduction about me. I begin, a quick introduction about me. I begin, a quick introduction about me. My name is Sharish Gokar. I'm a senior My name is Sharish Gokar. I'm a senior My name is Sharish Gokar. I'm a senior staff software engineer at Intuit Credit staff software engineer at Intuit Credit staff software engineer at Intuit Credit Karma where I built and own the AI Karma where I built and own the AI Karma where I built and own the AI analytics infrastructure and services. analytics infrastructure and services. analytics infrastructure and services. Um, and beside that, um, I'm advisor, Um, and beside that, um, I'm advisor, Um, and beside that, um, I'm advisor, uh, author, researchers, um, and also a uh, author, researchers, um, and also a uh, author, researchers, um, and also a program committee members for various program committee members for various program committee members for various organizations where I mentor the young organizations where I mentor the young organizations where I mentor the young researchers as well. researchers as well. researchers as well. All right. All right. All right. So, let's, uh, walk through the, um, So, let's, uh, walk through the, um, So, let's, uh, walk through the, um, agenda here. So, first we would like to agenda here. So, first we would like to agenda here. So, first we would like to talk about what are the challenges we talk about what are the challenges we talk about what are the challenges we have when we build an AI agents and have when we build an AI agents and have when we build an AI agents and deploy into the production systems and deploy into the production systems and deploy into the production systems and how we, uh, can build a different how we, uh, can build a different how we, uh, can build a different framework to address that, uh, different framework to address that, uh, different framework to address that, uh, different challenges here. challenges here. challenges here. Um, then how the architecture looks like Um, then how the architecture looks like Um, then how the architecture looks like for it. for it. for it. Um, then, um, how you can build your own Um, then, um, how you can build your own Um, then, um, how you can build your own MCP servers, MCP servers, MCP servers, um, and MCP gateway. um, and MCP gateway. um, and MCP gateway. And then some monitoring on top of that And then some monitoring on top of that And then some monitoring on top of that gateway as well. Um, then further we're gateway as well. Um, then further we're gateway as well. Um, then further we're going to talk about what are the going to talk about what are the going to talk about what are the different type of agent framework you different type of agent framework you different type of agent framework you can have, um, and then, can have, um, and then, can have, um, and then, um, some additional framework around um, some additional framework around um, some additional framework around managing the memories, tracking and managing the memories, tracking and managing the memories, tracking and eval, um, how you can block the PII eval, um, how you can block the PII eval, um, how you can block the PII data, data, data, uh, going before the LLMs, and then uh, going before the LLMs, and then uh, going before the LLMs, and then context optimizer to save your token, context optimizer to save your token, context optimizer to save your token, right?

right? right? So, without delay, let's, um, let me So, without delay, let's, um, let me So, without delay, let's, um, let me walk you through the challenge. Um, of walk you through the challenge. Um, of walk you through the challenge. Um, of course, like there are many agents, uh, course, like there are many agents, uh, course, like there are many agents, uh, uh, generally get failed in productions uh, generally get failed in productions uh, generally get failed in productions because their foundation controls are because their foundation controls are because their foundation controls are missing. Um, and the key reason behind missing. Um, and the key reason behind missing. Um, and the key reason behind is that, is that, is that, um, you know, like, oh, most of the um, you know, like, oh, most of the um, you know, like, oh, most of the people are missing the gateway to make people are missing the gateway to make people are missing the gateway to make sure that agent can, you know, sure that agent can, you know, sure that agent can, you know, communicate to the gateway to get communicate to the gateway to get communicate to the gateway to get authenticated first with the right auth authenticated first with the right auth authenticated first with the right auth providers so that they can redirect to providers so that they can redirect to providers so that they can redirect to the right MCP servers, right? Second, the right MCP servers, right? Second, the right MCP servers, right? Second, that, um, MCP servers are missing the that, um, MCP servers are missing the that, um, MCP servers are missing the right tool to execute your query. right tool to execute your query. right tool to execute your query. Third, do we have like enough tracking Third, do we have like enough tracking Third, do we have like enough tracking and evaluation system to make sure that and evaluation system to make sure that and evaluation system to make sure that every query when you are sending to the every query when you are sending to the every query when you are sending to the agents, whether they've been tracked or agents, whether they've been tracked or agents, whether they've been tracked or not, and are you evaluating the quality not, and are you evaluating the quality not, and are you evaluating the quality response or not, right? Do we have any response or not, right? Do we have any response or not, right? Do we have any in memory management system or not in in memory management system or not in in memory management system or not in terms of let's say you are doing the terms of let's say you are doing the terms of let's say you are doing the interaction with agent for long or for interaction with agent for long or for interaction with agent for long or for many days, many days, many days, is it able to recall that what you have is it able to recall that what you have is it able to recall that what you have been asked in the past, right? And then been asked in the past, right? And then been asked in the past, right? And then there are other framework, of course, there are other framework, of course, there are other framework, of course, like do you have a context of the like do you have a context of the like do you have a context of the measures to save the token, or do you measures to save the token, or do you measures to save the token, or do you have inner production in place to make have inner production in place to make have inner production in place to make sure that your any secrets should not go sure that your any secrets should not go sure that your any secrets should not go into LLM or tracking or tracing or any into LLM or tracking or tracing or any into LLM or tracking or tracing or any memory DB or so, right? So, a lot of lot memory DB or so, right? So, a lot of lot memory DB or so, right? So, a lot of lot of open questions. So, in order to of open questions. So, in order to of open questions. So, in order to resolve this, resolve this, resolve this, we are not going to, you know, like we are not going to, you know, like we are not going to, you know, like build a magical agent here in this demo.

build a magical agent here in this demo. build a magical agent here in this demo. We're going to talk about like different We're going to talk about like different We're going to talk about like different framework that you can build to address framework that you can build to address framework that you can build to address this. this. this. So, first is the agent framework. So, first is the agent framework. So, first is the agent framework. This is where like we use the This is where like we use the This is where like we use the like we choose to use the LangGraph like we choose to use the LangGraph like we choose to use the LangGraph framework, which is provided by framework, which is provided by framework, which is provided by LangChain community. LangChain community. LangChain community. This is help us in building the This is help us in building the This is help us in building the task-specific node based on your task-specific node based on your task-specific node based on your questions, and which is easy to plug in questions, and which is easy to plug in questions, and which is easy to plug in any like any like any like marketplace plugins or skills and marketplace plugins or skills and marketplace plugins or skills and prompts to make sure that agents prompts to make sure that agents prompts to make sure that agents respond efficiently based on your query. respond efficiently based on your query. respond efficiently based on your query. Further, Further, Further, the second part is we are also building the second part is we are also building the second part is we are also building the MCP gateway to make sure that the MCP gateway to make sure that the MCP gateway to make sure that whenever agent make the call or any whenever agent make the call or any whenever agent make the call or any service account from that agent make the service account from that agent make the service account from that agent make the call, make sure that they are securely call, make sure that they are securely call, make sure that they are securely get authenticated to that MCP gateway, get authenticated to that MCP gateway, get authenticated to that MCP gateway, and then get redirected to the right MCP and then get redirected to the right MCP and then get redirected to the right MCP servers. servers. servers. Third is the customized MCP, which means Third is the customized MCP, which means Third is the customized MCP, which means that you are building your own MCP that you are building your own MCP that you are building your own MCP servers with the right tool in place to servers with the right tool in place to servers with the right tool in place to execute your query. And then finally execute your query. And then finally execute your query. And then finally some audit on top of what actually some audit on top of what actually some audit on top of what actually operation you're performing, right? And operation you're performing, right? And operation you're performing, right? And then we talk about some additional then we talk about some additional then we talk about some additional framework around framework around framework around you know like how you can block the PII you know like how you can block the PII you know like how you can block the PII secrets, how you going to store the secrets, how you going to store the secrets, how you going to store the tracking eval into the cloud, tracking eval into the cloud, tracking eval into the cloud, and how you going to and how you going to and how you going to process the memories, especially any process the memories, especially any process the memories, especially any threads and checkpoints on every threads and checkpoints on every threads and checkpoints on every context, and then also how you can context, and then also how you can context, and then also how you can optimize overall your context to make optimize overall your context to make optimize overall your context to make sure that you don't don't burn too many sure that you don't don't burn too many sure that you don't don't burn too many tokens tokens tokens when you run your query. All right?

when you run your query. All right? when you run your query. All right? So, we talk about now the agent So, we talk about now the agent So, we talk about now the agent infrastructure. So, the Firstly, we'll infrastructure. So, the Firstly, we'll infrastructure. So, the Firstly, we'll go through the local development. The go through the local development. The go through the local development. The way the approach should be is that you way the approach should be is that you way the approach should be is that you know like whenever you build an agent, know like whenever you build an agent, know like whenever you build an agent, right? First you have to register your right? First you have to register your right? First you have to register your agent into some system. You usually tell agent into some system. You usually tell agent into some system. You usually tell that hey look, this is my the use case that hey look, this is my the use case that hey look, this is my the use case and these are the use case going to use and these are the use case going to use and these are the use case going to use for this particular vertical or system, for this particular vertical or system, for this particular vertical or system, and you feed all those information and and you feed all those information and and you feed all those information and generate a ID for your agent, right? generate a ID for your agent, right? generate a ID for your agent, right? Once you have that, then you can have a Once you have that, then you can have a Once you have that, then you can have a dev portal, internal portal where you dev portal, internal portal where you dev portal, internal portal where you can ingest all the details saying that can ingest all the details saying that can ingest all the details saying that hey look, this is my agent use case, hey look, this is my agent use case, hey look, this is my agent use case, this is my agent ID, this is the this is my agent ID, this is the this is my agent ID, this is the framework I'm going to use either framework I'm going to use either framework I'm going to use either LangGraph, LangChain or so. These are my LangGraph, LangChain or so. These are my LangGraph, LangChain or so. These are my language I'm going to use either Python language I'm going to use either Python language I'm going to use either Python or TypeScript, or TypeScript, or TypeScript, and then you can choose and then you can choose and then you can choose you know like your skills or prompts, you know like your skills or prompts, you know like your skills or prompts, and then you submit your information, and then you submit your information, and then you submit your information, right? To this. Once you submit this, right? To this. Once you submit this, right? To this. Once you submit this, the dev portal is going to create a the dev portal is going to create a the dev portal is going to create a repository, which is going to contains repository, which is going to contains repository, which is going to contains not just the agent framework like your not just the agent framework like your not just the agent framework like your LangGraph agent framework, but is also LangGraph agent framework, but is also LangGraph agent framework, but is also going to contains going to contains going to contains your memory framework, your tracking your memory framework, your tracking your memory framework, your tracking eval framework, right? and then your eval framework, right? and then your eval framework, right? and then your context optimizer framework, and your context optimizer framework, and your context optimizer framework, and your PII blocking PII blocking PII blocking framework as well.

framework as well. framework as well. So, So, So, the reason for that is because not the reason for that is because not the reason for that is because not necessarily all your developers are necessarily all your developers are necessarily all your developers are taking care of all this framework, taking care of all this framework, taking care of all this framework, right? They just sometimes you know right? They just sometimes you know right? They just sometimes you know create the framework what generally like create the framework what generally like create the framework what generally like AI assistant tools are generally the AI assistant tools are generally the AI assistant tools are generally the right? Or they stick to one or two right? Or they stick to one or two right? Or they stick to one or two framework, right? So, framework, right? So, framework, right? So, we just want to make sure that we just want to make sure that we just want to make sure that when you roll out such framework, make when you roll out such framework, make when you roll out such framework, make sure they are consistent across the sure they are consistent across the sure they are consistent across the company company company um and they can take care of um and they can take care of um and they can take care of uh and they can be centralized in one uh and they can be centralized in one uh and they can be centralized in one place so you can visualize their results place so you can visualize their results place so you can visualize their results as well, right? as well, right? as well, right? So, once you have that repository, so So, once you have that repository, so So, once you have that repository, so developers basically, you know, clone developers basically, you know, clone developers basically, you know, clone that repository, they can build their that repository, they can build their that repository, they can build their agent as a Docker image and containers agent as a Docker image and containers agent as a Docker image and containers and test their agent input and output and test their agent input and output and test their agent input and output through the query. And they can also through the query. And they can also through the query. And they can also traces on their local memory, maybe traces on their local memory, maybe traces on their local memory, maybe cloud span evaluator here. And you can cloud span evaluator here. And you can cloud span evaluator here. And you can also store this tracking and eval in also store this tracking and eval in also store this tracking and eval in some of your DB, maybe SQLite or maybe some of your DB, maybe SQLite or maybe some of your DB, maybe SQLite or maybe BigQuery BigQuery BigQuery um to see to make sure that agents are um to see to make sure that agents are um to see to make sure that agents are fully working on your local, right? So, fully working on your local, right? So, fully working on your local, right? So, once your agents are running in local, once your agents are running in local, once your agents are running in local, um then you're ready to go ahead and um then you're ready to go ahead and um then you're ready to go ahead and generate a PR and then you can once you generate a PR and then you can once you generate a PR and then you can once you get your two PR approval, then it's go get your two PR approval, then it's go get your two PR approval, then it's go to the CICD pipeline, right? So, the PR to the CICD pipeline, right? So, the PR to the CICD pipeline, right? So, the PR is basically going to firstly is going is basically going to firstly is going is basically going to firstly is going to uh register your agent into some to uh register your agent into some to uh register your agent into some centralized agent registry system, which centralized agent registry system, which centralized agent registry system, which provide a length with URL to it to make provide a length with URL to it to make provide a length with URL to it to make sure that you see all the traces.

sure that you see all the traces. sure that you see all the traces. And then the CICD pipeline is basically, And then the CICD pipeline is basically, And then the CICD pipeline is basically, you know, whatever the local Docker you know, whatever the local Docker you know, whatever the local Docker container you built here is going to container you built here is going to container you built here is going to build that container into production build that container into production build that container into production system. Uh it will scan that image, make system. Uh it will scan that image, make system. Uh it will scan that image, make sure that there's no vulnerability. Um sure that there's no vulnerability. Um sure that there's no vulnerability. Um if everything looks great, security if everything looks great, security if everything looks great, security approved, you can move that image back approved, you can move that image back approved, you can move that image back into the registry, right? Further, you into the registry, right? Further, you into the registry, right? Further, you have your own deployment platform or have your own deployment platform or have your own deployment platform or orchestrator platform which going to orchestrator platform which going to orchestrator platform which going to deploy that agent into the Kubernetes deploy that agent into the Kubernetes deploy that agent into the Kubernetes cluster in a given namespace, right? cluster in a given namespace, right? cluster in a given namespace, right? So, the good part here is that this dev So, the good part here is that this dev So, the good part here is that this dev portal is not just creating the portal is not just creating the portal is not just creating the repository, repository, repository, um right? It also do create um bunch of um right? It also do create um bunch of um right? It also do create um bunch of other production resources, for example, other production resources, for example, other production resources, for example, the workload identity, um right? And the workload identity, um right? And the workload identity, um right? And then memory DB or tracking and eval then memory DB or tracking and eval then memory DB or tracking and eval tables and then agent secret management, tables and then agent secret management, tables and then agent secret management, right? Um so, it based on whatever the right? Um so, it based on whatever the right? Um so, it based on whatever the name you provide, it name you provide, it name you provide, it not just again it like not just not just again it like not just not just again it like not just repository, it all these different repository, it all these different repository, it all these different resources as well. resources as well. resources as well. So, what happened is when this internal So, what happened is when this internal So, what happened is when this internal orchestrator when it deploy the agents, orchestrator when it deploy the agents, orchestrator when it deploy the agents, the agent already know that hey, what the agent already know that hey, what the agent already know that hey, what workload identity I supposed to work workload identity I supposed to work workload identity I supposed to work with, right? That binding should already with, right? That binding should already with, right? That binding should already happen. So, once this K to GSA binding happen. So, once this K to GSA binding happen. So, once this K to GSA binding happens, the agent will start happens, the agent will start happens, the agent will start communicating to all these resources.

communicating to all these resources. communicating to all these resources. Once everything in place, um agent is Once everything in place, um agent is Once everything in place, um agent is further going to communicate to the rest further going to communicate to the rest further going to communicate to the rest of your endpoint services like for of your endpoint services like for of your endpoint services like for example, AI gateway or MCP servers example, AI gateway or MCP servers example, AI gateway or MCP servers or length fuse for tracing or some other or length fuse for tracing or some other or length fuse for tracing or some other API endpoints. API endpoints. API endpoints. Right? Right? Right? So, now firstly, we So, now firstly, we So, now firstly, we talk about the MCP server as well. So, talk about the MCP server as well. So, talk about the MCP server as well. So, we talk about architecture. Now, we're we talk about architecture. Now, we're we talk about architecture. Now, we're going to the different component. So, going to the different component. So, going to the different component. So, MCP server is something that you can MCP server is something that you can MCP server is something that you can definitely use something which is definitely use something which is definitely use something which is outside available, but if you're working outside available, but if you're working outside available, but if you're working on air gap environment, it's highly on air gap environment, it's highly on air gap environment, it's highly recommended that you build your own MCP recommended that you build your own MCP recommended that you build your own MCP servers. So, um yeah, uh and you can use servers. So, um yeah, uh and you can use servers. So, um yeah, uh and you can use the the the uh I will not go with the definition. uh I will not go with the definition. uh I will not go with the definition. Everyone knows about it. So, Everyone knows about it. So, Everyone knows about it. So, the way the framework has been built the way the framework has been built the way the framework has been built here is that uh you you can leverage the here is that uh you you can leverage the here is that uh you you can leverage the fast MCP server package um or wrapper to fast MCP server package um or wrapper to fast MCP server package um or wrapper to um you know, build that MCP servers and um you know, build that MCP servers and um you know, build that MCP servers and you can create a different set of tools you can create a different set of tools you can create a different set of tools inside that a fast MCP wrapper to inside that a fast MCP wrapper to inside that a fast MCP wrapper to construct your um you know, the overall construct your um you know, the overall construct your um you know, the overall servers. servers. servers. And then you make sure that, you know, And then you make sure that, you know, And then you make sure that, you know, like every tool when you define in MCP like every tool when you define in MCP like every tool when you define in MCP server, they had to be server, they had to be server, they had to be fully reviewed by your own internal fully reviewed by your own internal fully reviewed by your own internal security team to make sure that only security team to make sure that only security team to make sure that only authorized tool can be go in production.

authorized tool can be go in production. authorized tool can be go in production. And that's how you can you know, enforce And that's how you can you know, enforce And that's how you can you know, enforce some filters, limits, and block the some filters, limits, and block the some filters, limits, and block the fields which is really not required to fields which is really not required to fields which is really not required to go to LLM and get you the information go to LLM and get you the information go to LLM and get you the information which is you're not supposed to get, which is you're not supposed to get, which is you're not supposed to get, right? Um and third part is like a of right? Um and third part is like a of right? Um and third part is like a of course, you need to make sure that you course, you need to make sure that you course, you need to make sure that you build your MCP server in a way that build your MCP server in a way that build your MCP server in a way that whenever the traffic is coming from your whenever the traffic is coming from your whenever the traffic is coming from your MCP client to MCP gateway to MCP MCP client to MCP gateway to MCP MCP client to MCP gateway to MCP servers, either it's coming from user servers, either it's coming from user servers, either it's coming from user service account, they has to be service account, they has to be service account, they has to be authenticated to the MCP gateway first authenticated to the MCP gateway first authenticated to the MCP gateway first and so that they can be access your MCP and so that they can be access your MCP and so that they can be access your MCP tool tool tool to get the right information. to get the right information. to get the right information. All right. Now, the gateway part. So, All right. Now, the gateway part. So, All right. Now, the gateway part. So, the how the gateway works. So, this is the how the gateway works. So, this is the how the gateway works. So, this is again, you can build it your own again, you can build it your own again, you can build it your own internal MCP gateway if you are again internal MCP gateway if you are again internal MCP gateway if you are again working on your air gap environment. So, working on your air gap environment. So, working on your air gap environment. So, the way it work is you are working on the way it work is you are working on the way it work is you are working on cloud assist like local, you know, code cloud assist like local, you know, code cloud assist like local, you know, code assistance tool or maybe your agent, assistance tool or maybe your agent, assistance tool or maybe your agent, right? So, the moment you submit your right? So, the moment you submit your right? So, the moment you submit your first query to your agent, it will send first query to your agent, it will send first query to your agent, it will send that one-time authentication request to that one-time authentication request to that one-time authentication request to the MCP gateway, right? MCP gateway the MCP gateway, right? MCP gateway the MCP gateway, right? MCP gateway recognized and take your request. First, recognized and take your request. First, recognized and take your request. First, it will save your session state into the it will save your session state into the it will save your session state into the Redis, right? And then it will, you Redis, right? And then it will, you Redis, right? And then it will, you know, return back the JWT access token know, return back the JWT access token know, return back the JWT access token back to the your client, right? Once back to the your client, right? Once back to the your client, right? Once client will receive that JWT token, client will receive that JWT token, client will receive that JWT token, it will know that what MCP server needs it will know that what MCP server needs it will know that what MCP server needs to call, right? So, it will submit that to call, right? So, it will submit that to call, right? So, it will submit that tool {slash} call request on a given tool {slash} call request on a given tool {slash} call request on a given name space {slash} MCP, meaning let's name space {slash} MCP, meaning let's name space {slash} MCP, meaning let's say Artifactory or GitHub {slash} MCP say Artifactory or GitHub {slash} MCP say Artifactory or GitHub {slash} MCP with the auth bearer JWT token

with the auth bearer JWT token with the auth bearer JWT token into that. So, once MCP gateway receive into that. So, once MCP gateway receive into that. So, once MCP gateway receive that, it will understand that what name that, it will understand that what name that, it will understand that what name space, meaning MCP server needs to look space, meaning MCP server needs to look space, meaning MCP server needs to look up for, and then it will unwrap that JWT up for, and then it will unwrap that JWT up for, and then it will unwrap that JWT token, and then it will attach that user token, and then it will attach that user token, and then it will attach that user token in that request and send it to the token in that request and send it to the token in that request and send it to the respective MCP servers. respective MCP servers. respective MCP servers. Now, once MCP server receive it, it will Now, once MCP server receive it, it will Now, once MCP server receive it, it will make that respective API calls to that make that respective API calls to that make that respective API calls to that endpoint of that, you know, to the endpoint of that, you know, to the endpoint of that, you know, to the service and then it will exchange that service and then it will exchange that service and then it will exchange that ID token ID token ID token and return the response based on and return the response based on and return the response based on whatever scope that user is authorized whatever scope that user is authorized whatever scope that user is authorized to see, right? So, once the user receive to see, right? So, once the user receive to see, right? So, once the user receive the token, the token, the token, you know, MCP server basically, you you know, MCP server basically, you you know, MCP server basically, you know, unwrap that all the data, make the know, unwrap that all the data, make the know, unwrap that all the data, make the payload which is compatible for MCP payload which is compatible for MCP payload which is compatible for MCP gateway to receive, and then finally the gateway to receive, and then finally the gateway to receive, and then finally the final payload will send back to the final payload will send back to the final payload will send back to the client to see the data points. client to see the data points. client to see the data points. So, what it shows is that, you know, So, what it shows is that, you know, So, what it shows is that, you know, like we build this MCP gateway to make like we build this MCP gateway to make like we build this MCP gateway to make sure that we have a centralized sure that we have a centralized sure that we have a centralized authentication and routing hub that authentication and routing hub that authentication and routing hub that securely connect any coding assistant securely connect any coding assistant securely connect any coding assistant tool or agents to the endpoint via this tool or agents to the endpoint via this tool or agents to the endpoint via this MCP gateway and MCP server. This There MCP gateway and MCP server. This There MCP gateway and MCP server. This There should not be direct access provided um should not be direct access provided um should not be direct access provided um on any endpoints. They has to go through on any endpoints. They has to go through on any endpoints. They has to go through the MCP gateway through proper the MCP gateway through proper the MCP gateway through proper authentications authentications authentications uh before they're reaching to the MCP uh before they're reaching to the MCP uh before they're reaching to the MCP server and the endpoints.

server and the endpoints. server and the endpoints. All right? All right? All right? So, now the part is the gateway So, now the part is the gateway So, now the part is the gateway monitoring, right? We talk about again, monitoring, right? We talk about again, monitoring, right? We talk about again, we talk about architecture, we talk we talk about architecture, we talk we talk about architecture, we talk about MCP server, MCP gateway. Now, it's about MCP server, MCP gateway. Now, it's about MCP server, MCP gateway. Now, it's a monitoring. Again, the AI agent cannot a monitoring. Again, the AI agent cannot a monitoring. Again, the AI agent cannot be trusted without continuous be trusted without continuous be trusted without continuous monitoring, evaluation, and feedback, monitoring, evaluation, and feedback, monitoring, evaluation, and feedback, right? This is why we are building the right? This is why we are building the right? This is why we are building the MCP gateway MCP gateway MCP gateway uh monitoring. So, the way the uh monitoring. So, the way the uh monitoring. So, the way the monitoring work is usually like the monitoring work is usually like the monitoring work is usually like the moment you submit any calls to your moment you submit any calls to your moment you submit any calls to your agent, uh it will send all the agent, uh it will send all the agent, uh it will send all the operational log into the log store. You operational log into the log store. You operational log into the log store. You can use ELK, Loki, or CloudWatch also, can use ELK, Loki, or CloudWatch also, can use ELK, Loki, or CloudWatch also, right? right? right? And generally, this contains the And generally, this contains the And generally, this contains the information like show me like all last information like show me like all last information like show me like all last 401s in my last 10 minutes call, right? 401s in my last 10 minutes call, right? 401s in my last 10 minutes call, right? And what did OAuth do when the token And what did OAuth do when the token And what did OAuth do when the token refresh failed, right? So, those kind of refresh failed, right? So, those kind of refresh failed, right? So, those kind of requests can goes to the log store. requests can goes to the log store. requests can goes to the log store. Now, there are some, you know, traces Now, there are some, you know, traces Now, there are some, you know, traces and metrics data um like real-time data and metrics data um like real-time data and metrics data um like real-time data that goes to the OpenTelemetry as well. that goes to the OpenTelemetry as well. that goes to the OpenTelemetry as well. Um OpenTelemetry again is open source Um OpenTelemetry again is open source Um OpenTelemetry again is open source solutions, which going to collect all solutions, which going to collect all solutions, which going to collect all your traces um and metrics and it's your traces um and metrics and it's your traces um and metrics and it's going to process the data and redirect going to process the data and redirect going to process the data and redirect to the back-end system, right? And in to the back-end system, right? And in to the back-end system, right? And in this case, the traces will go to the this case, the traces will go to the this case, the traces will go to the Tempo. This is again an open source Tempo. This is again an open source Tempo. This is again an open source solutions, which going to store all your solutions, which going to store all your solutions, which going to store all your traces in terms of like traces in the traces in terms of like traces in the traces in terms of like traces in the sense like, you know, like information sense like, you know, like information sense like, you know, like information around like which tool was slow on this around like which tool was slow on this around like which tool was slow on this request, whether this call is coming request, whether this call is coming request, whether this call is coming from agent or IDE, from agent or IDE, from agent or IDE, uh what was the exact error um on this uh what was the exact error um on this uh what was the exact error um on this failed call or so, right? So, these kind failed call or so, right? So, these kind failed call or so, right? So, these kind of data are going to store into the of data are going to store into the of data are going to store into the Tempo.

Tempo. Tempo. My bad. Okay. And then the metrics is My bad. Okay. And then the metrics is My bad. Okay. And then the metrics is about the Prometheus is all about the about the Prometheus is all about the about the Prometheus is all about the information about like what are P95 information about like what are P95 information about like what are P95 latencies for a given tool, right? latencies for a given tool, right? latencies for a given tool, right? Whether my auth Whether my auth Whether my auth rejection is spiking on a particular rejection is spiking on a particular rejection is spiking on a particular time or not, right? So, those kind of time or not, right? So, those kind of time or not, right? So, those kind of metrics is going to store in Prometheus. metrics is going to store in Prometheus. metrics is going to store in Prometheus. So, once you get all these data point So, once you get all these data point So, once you get all these data point like traces and metrics, this entire like traces and metrics, this entire like traces and metrics, this entire aggregate data aggregate data aggregate data actually save into the graph now for actually save into the graph now for actually save into the graph now for your visualizations. All right. So, the MCP server gateway All right. So, the MCP server gateway and monitoring is covered. Now, we're and monitoring is covered. Now, we're and monitoring is covered. Now, we're going to talk about the framework. So, going to talk about the framework. So, going to talk about the framework. So, the agent framework, right? So, we use the agent framework, right? So, we use the agent framework, right? So, we use LangGraph for most of the cases LangGraph for most of the cases LangGraph for most of the cases and is basically LangGraph again it's a and is basically LangGraph again it's a and is basically LangGraph again it's a provided by LangChain community. It's provided by LangChain community. It's provided by LangChain community. It's provide a graph based architecture that provide a graph based architecture that provide a graph based architecture that support support support loops, branching, state management for loops, branching, state management for loops, branching, state management for complex multi-agent AI workflows, right?

complex multi-agent AI workflows, right? complex multi-agent AI workflows, right? And the way it work is like the moment And the way it work is like the moment And the way it work is like the moment you submit your first query, you submit your first query, you submit your first query, we do interpret we do interpret we do interpret you know, like your question first, like you know, like your question first, like you know, like your question first, like what type of questions it is and what what type of questions it is and what what type of questions it is and what response is looking for response is looking for response is looking for and based on that it's create a plan, and based on that it's create a plan, and based on that it's create a plan, right? Then it will send for classify right? Then it will send for classify right? Then it will send for classify classify area where it will determine classify area where it will determine classify area where it will determine that hey, whether it is just like a read that hey, whether it is just like a read that hey, whether it is just like a read operations or whether it is just like a operations or whether it is just like a operations or whether it is just like a you know, modification level changes you know, modification level changes you know, modification level changes coming up or whether you really want to coming up or whether you really want to coming up or whether you really want to delete something, right? Or some delete something, right? Or some delete something, right? Or some sensitive operations. Based on that sensitive operations. Based on that sensitive operations. Based on that it adjust the overall structure and then it adjust the overall structure and then it adjust the overall structure and then plan accordingly. The next step is plan plan accordingly. The next step is plan plan accordingly. The next step is plan is actually itself is a sub graph where is actually itself is a sub graph where is actually itself is a sub graph where you do the major of your operations. you do the major of your operations. you do the major of your operations. Meaning that in the plan like once you Meaning that in the plan like once you Meaning that in the plan like once you know what operation you're performing, know what operation you're performing, know what operation you're performing, it goes to the like let's say in a fetch it goes to the like let's say in a fetch it goes to the like let's say in a fetch evidence section where it actually call evidence section where it actually call evidence section where it actually call that whatever the question you ask here, that whatever the question you ask here, that whatever the question you ask here, after all interpret and classify, after all interpret and classify, after all interpret and classify, um it actually go ahead and make the um it actually go ahead and make the um it actually go ahead and make the call to the MCP gateway, call to the MCP gateway, call to the MCP gateway, uh make sure they get authenticated, um uh make sure they get authenticated, um uh make sure they get authenticated, um and then it make the call to the MCP and then it make the call to the MCP and then it make the call to the MCP server using the right tool, server using the right tool, server using the right tool, um and get the data, and then generate um and get the data, and then generate um and get the data, and then generate that evidence, right? Once you validate that evidence, right? Once you validate that evidence, right? Once you validate all your finding, then it will check the all your finding, then it will check the all your finding, then it will check the policies.

policies. policies. Uh it will check whether it is a Uh it will check whether it is a Uh it will check whether it is a read-only, then go ahead and execute, read-only, then go ahead and execute, read-only, then go ahead and execute, and I will save all the record points and I will save all the record points and I will save all the record points into our audit logs. into our audit logs. into our audit logs. But if it is a right and sensitive But if it is a right and sensitive But if it is a right and sensitive operations, definitely it will go operations, definitely it will go operations, definitely it will go through the HITL process where you wait through the HITL process where you wait through the HITL process where you wait for an approval. If it is approved, you for an approval. If it is approved, you for an approval. If it is approved, you go ahead and execute. If it is get go ahead and execute. If it is get go ahead and execute. If it is get rejected, in either case it's going to rejected, in either case it's going to rejected, in either case it's going to save into the audit log. save into the audit log. save into the audit log. Right? Right? Right? So, So, So, if we combine all this together, like if we combine all this together, like if we combine all this together, like MCP gateway, MCP server, MCP gateway, MCP server, MCP gateway, MCP server, um and the agent, right? This is how um and the agent, right? This is how um and the agent, right? This is how things looks like. So, ultimately, things looks like. So, ultimately, things looks like. So, ultimately, whatever the agent you're building, whatever the agent you're building, whatever the agent you're building, whatever the call uh you're submitting, whatever the call uh you're submitting, whatever the call uh you're submitting, it will go through the MCP gateway for it will go through the MCP gateway for it will go through the MCP gateway for authentications, and then it will authentications, and then it will authentications, and then it will redirect to the right MCP servers to redirect to the right MCP servers to redirect to the right MCP servers to execute uh you know, your query on a execute uh you know, your query on a execute uh you know, your query on a given tool, given tool, given tool, uh so that it make the API call to that uh so that it make the API call to that uh so that it make the API call to that endpoint and get the response. Right? endpoint and get the response. Right? endpoint and get the response. Right? All right. If it If we think from the All right. If it If we think from the All right. If it If we think from the infrastructure perspective, this is how infrastructure perspective, this is how infrastructure perspective, this is how it looks like. So, agent is it looks like. So, agent is it looks like. So, agent is communicating to the MCP gateway, goes communicating to the MCP gateway, goes communicating to the MCP gateway, goes to the MCP server, and to the endpoint.

to the MCP server, and to the endpoint. to the MCP server, and to the endpoint. All right? All right? All right? So, now the uh we talked a lot about MCP So, now the uh we talked a lot about MCP So, now the uh we talked a lot about MCP gateway, MCP servers, um the monitoring, gateway, MCP servers, um the monitoring, gateway, MCP servers, um the monitoring, um and the agent framework. Now we talk um and the agent framework. Now we talk um and the agent framework. Now we talk about uh three and four additional about uh three and four additional about uh three and four additional framework that we should have to take framework that we should have to take framework that we should have to take care of when building the end-to-end care of when building the end-to-end care of when building the end-to-end ecosystem, right? One is them is agent ecosystem, right? One is them is agent ecosystem, right? One is them is agent memory management. Again, it's a uh you memory management. Again, it's a uh you memory management. Again, it's a uh you know, it's a it's refer It's basically a know, it's a it's refer It's basically a know, it's a it's refer It's basically a process where you can save and restore process where you can save and restore process where you can save and restore the state of your workflow over the the state of your workflow over the the state of your workflow over the time, right? Ensuring that time, right? Ensuring that time, right? Ensuring that it's provide you a durability and it's provide you a durability and it's provide you a durability and seamless recovery across across your seamless recovery across across your seamless recovery across across your every query run. every query run. every query run. So, the way it work is like whenever you So, the way it work is like whenever you So, the way it work is like whenever you submit any query to your agent, it submit any query to your agent, it submit any query to your agent, it actually generate a thread, right? On actually generate a thread, right? On actually generate a thread, right? On that given thread, when you ask your that given thread, when you ask your that given thread, when you ask your first question, it will going to create first question, it will going to create first question, it will going to create a run ID for that. So, the moment you a run ID for that. So, the moment you a run ID for that. So, the moment you ask some deeper questions, it will go on ask some deeper questions, it will go on ask some deeper questions, it will go on a particular node in that langraph and a particular node in that langraph and a particular node in that langraph and execute that query. And on that node, it execute that query. And on that node, it execute that query. And on that node, it going to save that respective checkpoint going to save that respective checkpoint going to save that respective checkpoint here, right? Now, if you ask a follow-up here, right? Now, if you ask a follow-up here, right? Now, if you ask a follow-up questions on the same thread, it going questions on the same thread, it going questions on the same thread, it going to create another run ID. And again, to create another run ID. And again, to create another run ID. And again, depend on your follow-up question, it go depend on your follow-up question, it go depend on your follow-up question, it go to a specific node and save the to a specific node and save the to a specific node and save the checkpoint over there, right? And all checkpoint over there, right? And all checkpoint over there, right? And all this data for your local development, this data for your local development, this data for your local development, you can use the SQLite SQLite DB, you can use the SQLite SQLite DB, you can use the SQLite SQLite DB, but in production, unless in GCP, you but in production, unless in GCP, you but in production, unless in GCP, you can use Cloud Spanner. And can use Cloud Spanner. And can use Cloud Spanner. And in respective databases, you can use in in respective databases, you can use in in respective databases, you can use in Azure Azure and AWS as well, right? So,

Azure Azure and AWS as well, right? So, Azure Azure and AWS as well, right? So, just walk you through the quick demo just walk you through the quick demo just walk you through the quick demo here on the memory. So, for example, here on the memory. So, for example, here on the memory. So, for example, I have, you know, like just build this I have, you know, like just build this I have, you know, like just build this agent locally and then, agent locally and then, agent locally and then, you know, trying to run this query on you know, trying to run this query on you know, trying to run this query on SQLite DB to see if there is any data or SQLite DB to see if there is any data or SQLite DB to see if there is any data or not. So, I'm just, you know, initially I not. So, I'm just, you know, initially I not. So, I'm just, you know, initially I don't have any data. Now, I submit my don't have any data. Now, I submit my don't have any data. Now, I submit my query, you will see that I received the query, you will see that I received the query, you will see that I received the one record, which is saying that, "Hey, one record, which is saying that, "Hey, one record, which is saying that, "Hey, this is my thread ID. This is the this is my thread ID. This is the this is my thread ID. This is the question I have been asked." And exact question I have been asked." And exact question I have been asked." And exact time, you know, when this query was time, you know, when this query was time, you know, when this query was executed. Now, if I run the next query executed. Now, if I run the next query executed. Now, if I run the next query to see that, "Hey, how many messages to see that, "Hey, how many messages to see that, "Hey, how many messages I've been produced on a given thread?" I've been produced on a given thread?" I've been produced on a given thread?" You can get this information by running You can get this information by running You can get this information by running the further queries, right? the further queries, right? the further queries, right? Now, if I would like to run to see that, Now, if I would like to run to see that, Now, if I would like to run to see that, you know, like all the metadata level you know, like all the metadata level you know, like all the metadata level information on my query, I can run my information on my query, I can run my information on my query, I can run my query again and And can see the detail query again and And can see the detail query again and And can see the detail description around, you know, like how description around, you know, like how description around, you know, like how many tool I get invoked, what is my many tool I get invoked, what is my many tool I get invoked, what is my thread ID, what is my run ID, thread ID, what is my run ID, thread ID, what is my run ID, um you know, like and all those um you know, like and all those um you know, like and all those informations. Now, if I want to also see informations. Now, if I want to also see informations. Now, if I want to also see that, hey, what tool have been executed that, hey, what tool have been executed that, hey, what tool have been executed when I run this query, um you can go when I run this query, um you can go when I run this query, um you can go ahead and see the exact tool name as ahead and see the exact tool name as ahead and see the exact tool name as well on that given MCP servers.

well on that given MCP servers. well on that given MCP servers. Now, further I will also want to see Now, further I will also want to see Now, further I will also want to see that how the progression was happened on that how the progression was happened on that how the progression was happened on my lang graph framework, that also you my lang graph framework, that also you my lang graph framework, that also you can see by um running this type of query can see by um running this type of query can see by um running this type of query on your memory DB. You can see that here on your memory DB. You can see that here on your memory DB. You can see that here like different kind of, you know, nodes like different kind of, you know, nodes like different kind of, you know, nodes been called when actually, um you know, been called when actually, um you know, been called when actually, um you know, that query has been run. that query has been run. that query has been run. All right? All right? All right? Now, the tracking and eval. So, just to Now, the tracking and eval. So, just to Now, the tracking and eval. So, just to make sure that make sure that make sure that um there is no confusion here like the um there is no confusion here like the um there is no confusion here like the we talked about earlier the MCP gateway we talked about earlier the MCP gateway we talked about earlier the MCP gateway monitoring, this is the agent level monitoring, this is the agent level monitoring, this is the agent level tracking. So, we do this because we tracking. So, we do this because we tracking. So, we do this because we really want to track that what query you really want to track that what query you really want to track that what query you have been submitting to the agent have been submitting to the agent have been submitting to the agent um and what the response is coming, what um and what the response is coming, what um and what the response is coming, what alarm model you have been used, what's alarm model you have been used, what's alarm model you have been used, what's the cost of your token, um how many the cost of your token, um how many the cost of your token, um how many token you have been burned, and what's token you have been burned, and what's token you have been burned, and what's your, you know, your, you know, your, you know, evaluation criteria looks like. What is evaluation criteria looks like. What is evaluation criteria looks like. What is the quality of your response or not, the quality of your response or not, the quality of your response or not, right? So, you need to store that data right? So, you need to store that data right? So, you need to store that data from the day one. And imagine that if from the day one. And imagine that if from the day one. And imagine that if you have a thousand plus agent, every you have a thousand plus agent, every you have a thousand plus agent, every agents will have their own agents will have their own agents will have their own um you know, um agent tracking and eval um you know, um agent tracking and eval um you know, um agent tracking and eval um project and the tables, right? So, if um project and the tables, right? So, if um project and the tables, right? So, if you combine, let's say if you have 100 you combine, let's say if you have 100 you combine, let's say if you have 100 agents, you will have 100 different agents, you will have 100 different agents, you will have 100 different projects and each project has their own projects and each project has their own projects and each project has their own respective tracking and eval tables.

respective tracking and eval tables. respective tracking and eval tables. Now, tomorrow, 6 months down the line, Now, tomorrow, 6 months down the line, Now, tomorrow, 6 months down the line, if somebody ask you that how your agent if somebody ask you that how your agent if somebody ask you that how your agent has been performed, what was the has been performed, what was the has been performed, what was the response, what's was the quality, how response, what's was the quality, how response, what's was the quality, how much token you have been burned, you can much token you have been burned, you can much token you have been burned, you can aggregate all this data and you can aggregate all this data and you can aggregate all this data and you can create executive dashboards around it to create executive dashboards around it to create executive dashboards around it to see um with to see the what is the ROI see um with to see the what is the ROI see um with to see the what is the ROI in building so many agents, right? in building so many agents, right? in building so many agents, right? So, let me walk you through the the So, let me walk you through the the So, let me walk you through the the tracking and eval um demo here. tracking and eval um demo here. tracking and eval um demo here. So, So, So, um here like if I look at uh this one, um here like if I look at uh this one, um here like if I look at uh this one, so first like, you know, like if you see so first like, you know, like if you see so first like, you know, like if you see that there are six tables are listed that there are six tables are listed that there are six tables are listed here. One is a job tracking table, um here. One is a job tracking table, um here. One is a job tracking table, um and which is contains most of this like and which is contains most of this like and which is contains most of this like information around what query like what information around what query like what information around what query like what is your query, what's the response and is your query, what's the response and is your query, what's the response and everything. And there are a bunch of everything. And there are a bunch of everything. And there are a bunch of other tables as well like your quality other tables as well like your quality other tables as well like your quality related and your tool invocations and related and your tool invocations and related and your tool invocations and then node level executions and all that. then node level executions and all that. then node level executions and all that. So, in this case I'm going to submit my So, in this case I'm going to submit my So, in this case I'm going to submit my first query first query first query on my agent um and you will see that um on my agent um and you will see that um on my agent um and you will see that um there will not be there will not be there will not be uh any data in the first place uh uh any data in the first place uh uh any data in the first place uh because it takes some time. So, I will because it takes some time. So, I will because it takes some time. So, I will refresh this again.

refresh this again. refresh this again. And this time I will see that when I And this time I will see that when I And this time I will see that when I refresh, I do see that for a given query refresh, I do see that for a given query refresh, I do see that for a given query I get my request ID, my session ID, my I get my request ID, my session ID, my I get my request ID, my session ID, my agent name, um you know, what question agent name, um you know, what question agent name, um you know, what question I've been asked, I've been asked, I've been asked, um and what was the output um of that uh um and what was the output um of that uh um and what was the output um of that uh question, uh whether it has completed or question, uh whether it has completed or question, uh whether it has completed or not, and then what is the time it took, not, and then what is the time it took, not, and then what is the time it took, what is the model I've been used, um and what is the model I've been used, um and what is the model I've been used, um and then some of the middle level uh middle then some of the middle level uh middle then some of the middle level uh middle level informations, right? level informations, right? level informations, right? Further, if I look for the quality, Further, if I look for the quality, Further, if I look for the quality, again there is a eight quality checks again there is a eight quality checks again there is a eight quality checks per run, per run, per run, right? And you can see that this one row right? And you can see that this one row right? And you can see that this one row each for, you know, like success, like each for, you know, like success, like each for, you know, like success, like latency, cost, tool accuracy, safety, latency, cost, tool accuracy, safety, latency, cost, tool accuracy, safety, etc. Like all this has been stored here, etc. Like all this has been stored here, etc. Like all this has been stored here, right? right? right? Now, the eval scores. This is again this Now, the eval scores. This is again this Now, the eval scores. This is again this going to be a one row. This is going to going to be a one row. This is going to going to be a one row. This is going to be a scoreboard for your given run, be a scoreboard for your given run, be a scoreboard for your given run, which is basically varies between zero which is basically varies between zero which is basically varies between zero to one, which is scores to one, which is scores to one, which is scores uh you know, like all the quality uh you know, like all the quality uh you know, like all the quality dimension in a single row, dimension in a single row, dimension in a single row, right? Now, the quality metrics. So, right? Now, the quality metrics. So, right? Now, the quality metrics. So, again it's a deep dive on that again it's a deep dive on that again it's a deep dive on that particular one run. Basically a nested particular one run. Basically a nested particular one run. Basically a nested field for the response field for the response field for the response quality, safety, latency, cost, uh and quality, safety, latency, cost, uh and quality, safety, latency, cost, uh and the tool stat in one white row, right?

the tool stat in one white row, right? the tool stat in one white row, right? Now, the tool invocations, so every tool Now, the tool invocations, so every tool Now, the tool invocations, so every tool call, which MCP tool ran, how long it call, which MCP tool ran, how long it call, which MCP tool ran, how long it took, and whether it succeeded or not. took, and whether it succeeded or not. took, and whether it succeeded or not. Right? And now the tool agent Right? And now the tool agent Right? And now the tool agent trajectory, which is talk about trajectory, which is talk about trajectory, which is talk about step-by-step story of the run, each LLM step-by-step story of the run, each LLM step-by-step story of the run, each LLM call, and the graph state in order, like call, and the graph state in order, like call, and the graph state in order, like plan, summarize, etc. plan, summarize, etc. plan, summarize, etc. All right. All right. All right. So, now So, now So, now uh the next is the agent PIR treatment. uh the next is the agent PIR treatment. uh the next is the agent PIR treatment. This is very, very important because This is very, very important because This is very, very important because many times it happens that when you're many times it happens that when you're many times it happens that when you're working in agents or uh this thing, um working in agents or uh this thing, um working in agents or uh this thing, um sometimes there's a possibility that sometimes there's a possibility that sometimes there's a possibility that there are some secrets being ingested as there are some secrets being ingested as there are some secrets being ingested as a part of your query, right? Let's say a part of your query, right? Let's say a part of your query, right? Let's say you're uploading some doc which is has you're uploading some doc which is has you're uploading some doc which is has some sensitive informations um and some sensitive informations um and some sensitive informations um and sometimes while typing you may have some sometimes while typing you may have some sometimes while typing you may have some API keys or so. So, we have a framework API keys or so. So, we have a framework API keys or so. So, we have a framework which define that different sort of data which define that different sort of data which define that different sort of data types and their severity level. If your types and their severity level. If your types and their severity level. If your severity is above 90% or so, uh we will severity is above 90% or so, uh we will severity is above 90% or so, uh we will block and redact that uh block and redact that uh block and redact that uh secrets uh immediately. So, that uh make secrets uh immediately. So, that uh make secrets uh immediately. So, that uh make sure that that secret is never goes to sure that that secret is never goes to sure that that secret is never goes to LLM models, right? Neither it will goes LLM models, right? Neither it will goes LLM models, right? Neither it will goes to your memory DB, or nor it will go to to your memory DB, or nor it will go to to your memory DB, or nor it will go to tracking and eval tables, or tracing uh tracking and eval tables, or tracing uh tracking and eval tables, or tracing uh systems as well. So, the goal here is to systems as well. So, the goal here is to systems as well. So, the goal here is to just block this sensitive data as it is just block this sensitive data as it is just block this sensitive data as it is the moment it get ingested, right?

the moment it get ingested, right? the moment it get ingested, right? Moment moment you type. Moment moment you type. Moment moment you type. All right. Um the last framework is all All right. Um the last framework is all All right. Um the last framework is all about now is the about now is the about now is the context optimizer. This is one of my context optimizer. This is one of my context optimizer. This is one of my favorite area where like um you know, uh favorite area where like um you know, uh favorite area where like um you know, uh whenever you submit, it's very whenever you submit, it's very whenever you submit, it's very important, right? Whenever you submit important, right? Whenever you submit important, right? Whenever you submit too many queries, sometimes you upload a too many queries, sometimes you upload a too many queries, sometimes you upload a lot of the lot of the lot of the um data points and you upload bunch of um data points and you upload bunch of um data points and you upload bunch of file systems or so, right? It's very file systems or so, right? It's very file systems or so, right? It's very important that time is important that time is important that time is um you know, like you make sure that you um you know, like you make sure that you um you know, like you make sure that you save your token in every query. So, we save your token in every query. So, we save your token in every query. So, we build in context optimizer framework build in context optimizer framework build in context optimizer framework where you actually shape your query and where you actually shape your query and where you actually shape your query and then you prune it before it go to LLM. then you prune it before it go to LLM. then you prune it before it go to LLM. In the pruning In the pruning In the pruning system, uh we do select the payload system, uh we do select the payload system, uh we do select the payload based on the token limit. Then we based on the token limit. Then we based on the token limit. Then we re-rank that, "Hey, which is the most re-rank that, "Hey, which is the most re-rank that, "Hey, which is the most relevant relevant information that we relevant relevant information that we relevant relevant information that we should target for when you submit this should target for when you submit this should target for when you submit this huge giant, you know, context?" And then huge giant, you know, context?" And then huge giant, you know, context?" And then we dedupe that, uh remove all the we dedupe that, uh remove all the we dedupe that, uh remove all the duplicates, and create a very unique set duplicates, and create a very unique set duplicates, and create a very unique set of of of uh very curated data like uh I mean the uh very curated data like uh I mean the uh very curated data like uh I mean the main data for the LLM. And then we main data for the LLM. And then we main data for the LLM. And then we finally compress this large payload finally compress this large payload finally compress this large payload before it goes to the LLM. So, let's see before it goes to the LLM. So, let's see before it goes to the LLM. So, let's see this uh demo here.

this uh demo here. this uh demo here. Uh this will be quick. So, for example, Uh this will be quick. So, for example, Uh this will be quick. So, for example, let's say I will submit my first query, let's say I will submit my first query, let's say I will submit my first query, "Hey, find me all the artifacts in my "Hey, find me all the artifacts in my "Hey, find me all the artifacts in my JFrog Artifactory." JFrog Artifactory." JFrog Artifactory." Um you see that first when before even I Um you see that first when before even I Um you see that first when before even I not submitted any query yet, there is no not submitted any query yet, there is no not submitted any query yet, there is no data here. So, the moment I submit this data here. So, the moment I submit this data here. So, the moment I submit this first query, and if I refresh my query here, and if I refresh my query here, um and then you can see that the record um and then you can see that the record um and then you can see that the record has been added. So, you can see there is has been added. So, you can see there is has been added. So, you can see there is a 24,000 token has been used. I will go a 24,000 token has been used. I will go a 24,000 token has been used. I will go back a little bit um just to show the back a little bit um just to show the back a little bit um just to show the thing. So, you can see this 24,000 token thing. So, you can see this 24,000 token thing. So, you can see this 24,000 token has been used for given questions. Now, has been used for given questions. Now, has been used for given questions. Now, if I ask uh now I enable the context if I ask uh now I enable the context if I ask uh now I enable the context optimizer behind the scene and I'm rerun optimizer behind the scene and I'm rerun optimizer behind the scene and I'm rerun the exact same query again. the exact same query again. the exact same query again. So, this time you will see that uh you So, this time you will see that uh you So, this time you will see that uh you know, for the same query, if I refresh know, for the same query, if I refresh know, for the same query, if I refresh this run, and you will see that uh this this run, and you will see that uh this this run, and you will see that uh this time it used just 2,400 token. time it used just 2,400 token. time it used just 2,400 token. Right? For exact same query. Right? For exact same query. Right? For exact same query. So, which is almost like 90% saving. I So, which is almost like 90% saving. I So, which is almost like 90% saving. I can rerun the same demo on the Slack bot can rerun the same demo on the Slack bot can rerun the same demo on the Slack bot as well where I'm asking Slack bot, as well where I'm asking Slack bot, as well where I'm asking Slack bot, "Hey, tell me all my Docker images "Hey, tell me all my Docker images "Hey, tell me all my Docker images across all the repositories in across all the repositories in across all the repositories in Artifactory." When I run this query, Artifactory." When I run this query, Artifactory." When I run this query, there is no data. Now, I'm going to there is no data. Now, I'm going to there is no data. Now, I'm going to submit my query.

submit my query. submit my query. So, So, So, once this query will run, um and if I once this query will run, um and if I once this query will run, um and if I run my query again here, and you will run my query again here, and you will run my query again here, and you will see the record get created. It's again see the record get created. It's again see the record get created. It's again used like 24,000 for this this used like 24,000 for this this used like 24,000 for this this particular query. particular query. particular query. Now, what I will do is I will Now, what I will do is I will Now, what I will do is I will turn on my optimizer behind the scene turn on my optimizer behind the scene turn on my optimizer behind the scene and I rerun the same query in the Slack and I rerun the same query in the Slack and I rerun the same query in the Slack bot. Um and then if I run this query bot. Um and then if I run this query bot. Um and then if I run this query again, um it will show like it exactly again, um it will show like it exactly again, um it will show like it exactly used 1,500 almost token, which is again used 1,500 almost token, which is again used 1,500 almost token, which is again like 90% saving. In most of the cases, like 90% saving. In most of the cases, like 90% saving. In most of the cases, we get like 70 to 78% token saves. So, we get like 70 to 78% token saves. So, we get like 70 to 78% token saves. So, imagine that you have thousands of imagine that you have thousands of imagine that you have thousands of um you know, like um um you know, like um um you know, like um um if you have thousands of like agents, um if you have thousands of like agents, um if you have thousands of like agents, which is running like which is running like which is running like you know, many queries per day, you you know, many queries per day, you you know, many queries per day, you imagine how much token you can save with imagine how much token you can save with imagine how much token you can save with this framework. All right. So, now the this framework. All right. So, now the this framework. All right. So, now the key takeaway here is uh you know, like key takeaway here is uh you know, like key takeaway here is uh you know, like um um um for us is like make sure that we for us is like make sure that we for us is like make sure that we build an enterprise framework, um not build an enterprise framework, um not build an enterprise framework, um not just an agent, right? Which make sure just an agent, right? Which make sure just an agent, right? Which make sure that you have all the framework that you have all the framework that you have all the framework underneath. Don't try to be in rush to underneath. Don't try to be in rush to underneath. Don't try to be in rush to building some agent quickly and deploy building some agent quickly and deploy building some agent quickly and deploy ship it. Make sure that ship it. Make sure that ship it. Make sure that it has to go through all the framework it has to go through all the framework it has to go through all the framework all together, so that you should not all together, so that you should not all together, so that you should not realize a letter state that, you know, I realize a letter state that, you know, I realize a letter state that, you know, I should have this framework in place.

should have this framework in place. should have this framework in place. Right? In in the very first place. Right? In in the very first place. Right? In in the very first place. Second is like your Make sure that you Second is like your Make sure that you Second is like your Make sure that you have a proper gateway system. Make sure have a proper gateway system. Make sure have a proper gateway system. Make sure that every tool call request either you that every tool call request either you that every tool call request either you are doing this from your local or are doing this from your local or are doing this from your local or through agent, it has to go through some through agent, it has to go through some through agent, it has to go through some sort of centralized MCP gateway. Uh make sort of centralized MCP gateway. Uh make sort of centralized MCP gateway. Uh make sure it authenticated before it reaching sure it authenticated before it reaching sure it authenticated before it reaching to the MCP servers. And then make sure to the MCP servers. And then make sure to the MCP servers. And then make sure that your MCP servers take a control on that your MCP servers take a control on that your MCP servers take a control on what tool um you know, uh what tool um you know, uh what tool um you know, uh you should use uh to execute a you should use uh to execute a you should use uh to execute a particular query, right? And make sure particular query, right? And make sure particular query, right? And make sure that you have monitoring in place and that you have monitoring in place and that you have monitoring in place and complete auditing in place for that tool complete auditing in place for that tool complete auditing in place for that tool executions. And of course, you should um executions. And of course, you should um executions. And of course, you should um you know, keep the always human in the you know, keep the always human in the you know, keep the always human in the loop process for every um every action loop process for every um every action loop process for every um every action items, every high-risk items. items, every high-risk items. items, every high-risk items. Um make sure that there is a Um make sure that there is a Um make sure that there is a observability observability observability um and you know, um auditing has to be um and you know, um auditing has to be um and you know, um auditing has to be in place. And last but not least, make in place. And last but not least, make in place. And last but not least, make sure that context optimizer should sure that context optimizer should sure that context optimizer should always be there to justify your cost always be there to justify your cost always be there to justify your cost on your agents, right? on your agents, right? on your agents, right? So, just want to share is that I So, just want to share is that I So, just want to share is that I released my MCP book recently on Amazon.

released my MCP book recently on Amazon. released my MCP book recently on Amazon. So, feel free to download. It's free in So, feel free to download. It's free in So, feel free to download. It's free in Kindle. Kindle. Kindle. And it's hard copy is also available. And it's hard copy is also available. And it's hard copy is also available. All right. All right. All right. Thank you so much for watching this Thank you so much for watching this Thank you so much for watching this video. video. video. And if you do have any questions, feel And if you do have any questions, feel And if you do have any questions, feel free to, you know, like reach out to me free to, you know, like reach out to me free to, you know, like reach out to me over LinkedIn or Twitter. I'm happy to over LinkedIn or Twitter. I'm happy to over LinkedIn or Twitter. I'm happy to answer. Thank you again.