How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner

Greetings. How are you all, how are you Greetings. How are you all, how are you doing? It's difficult to doing? It's difficult to doing? It's difficult to speak after speak after speak after such a report. Why did such a report. Why did such a report. Why did everyone leave? No, this is a everyone leave? No, this is a everyone leave? No, this is a repeat of the speech repeat of the speech repeat of the speech I already gave yesterday I already gave yesterday I already gave yesterday upstairs. So this one will be upstairs. So this one will be upstairs. So this one will be better than yesterday's. better than yesterday's. better than yesterday's. So thank you for So thank you for So thank you for coming to this one. I coming to this one. I coming to this one. I work on VS Code, and work on VS Code, and work on VS Code, and thanks to AI, we've thanks to AI, we've thanks to AI, we've moved from a monthly moved from a monthly moved from a monthly to a weekly to a weekly to a weekly release cycle. This is a story release cycle. This is a story release cycle. This is a story about our experience and the about our experience and the about our experience and the systems that systems that systems that have allowed us to not only have allowed us to not only have allowed us to not only release updates release updates release updates faster, but also work more faster, but also work more faster, but also work more efficiently. This is not efficiently. This is not efficiently. This is not about how we about how we about how we implement agents implement agents implement agents in VS Code, you can in VS Code, you can in VS Code, you can try that anytime. try that anytime. try that anytime. You can visit You can visit You can visit our booth and our booth and our booth and see agents in see agents in see agents in action, but right now we're action, but right now we're action, but right now we're talking about how we talking about how we talking about how we use use use agents to develop the agents to develop the agents to develop the product itself. This product itself. This product itself. This applies to VS Code, Copilot applies to VS Code, Copilot applies to VS Code, Copilot AI, and other areas. AI, and other areas. AI, and other areas. But also about how, But also about how, But also about how, due to the use of due to the use of due to the use of agents and AI, everything agents and AI, everything agents and AI, everything starts to break down, and starts to break down, and starts to break down, and you have to you have to you have to rethink your rethink your rethink your approach to work. This is a approach to work. This is a approach to work. This is a very small very small very small team building a team building a team building a product for over 50 product for over 50 product for over 50 million million million users. users. Last year, we Last year, we started tracking started tracking code survival rates in VS Code at an code survival rates in VS Code at an early stage. That is, early stage. That is, early stage. That is, this is the percentage of code this is the percentage of code this is the percentage of code written by agents written by agents written by agents that was actually committed that was actually committed . How much . How much garbage has the agent created garbage has the agent created garbage has the agent created that you threw away, that you threw away, that you threw away, didn't trust, didn't trust, didn't trust, read, and deleted.

read, and deleted. GPT 4.1 started at 55%, GPT 4.1 started at 55%, but over time, thanks to but over time, thanks to but over time, thanks to improved improved improved environments and new environments and new environments and new models, Cloud Opus 4.6 models, Cloud Opus 4.6 models, Cloud Opus 4.6 now reaches 86%. This now reaches 86%. This now reaches 86%. This growth clearly growth clearly growth clearly demonstrates demonstrates demonstrates developers' confidence in developers' confidence in developers' confidence in faster release of faster release of faster release of code created code created code created with AI. But this with AI. But this with AI. But this success created new success created new success created new problems. Regarding VS Code, problems. Regarding VS Code, problems. Regarding VS Code, we are one of the we are one of the we are one of the largest, if not the largest, if not the largest, if not the largest open source largest open source largest open source project on GitHub. We have project on GitHub. We have project on GitHub. We have seen a seen a seen a huge huge huge increase in the number of increase in the number of increase in the number of tickets as AI tickets as AI tickets as AI helps helps helps create them. We are seeing create them. We are seeing create them. We are seeing more high-quality more high-quality more high-quality calls, but calls, but calls, but at the same time more at the same time more at the same time more automated, automated, automated, low-quality low-quality error messages. We are also error messages. We are also seeing more seeing more seeing more open open open merge requests (PRs) from our merge requests (PRs) from our merge requests (PRs) from our team, due to our own team, due to our own team, due to our own work rate and work rate and work rate and from the community. Interestingly from the community. Interestingly , despite , despite , despite many expectations of a large many expectations of a large many expectations of a large number of "garbage number of "garbage " PRs, the number of " PRs, the number of " PRs, the number of accepted requests accepted requests accepted requests from the community is also from the community is also from the community is also increasing, allowing increasing, allowing increasing, allowing for significantly for significantly for significantly more contributions. As more contributions. As more contributions. As part of this new part of this new part of this new workflow, workflow, workflow, a few months ago a few months ago a few months ago we abandoned the we abandoned the we abandoned the monthly releases we had monthly releases we had monthly releases we had practiced for over 10 practiced for over 10 practiced for over 10 years. Since the years. Since the years. Since the launch of VS Code 1.0, we launch of VS Code 1.0, we launch of VS Code 1.0, we have released a new have released a new have released a new version every month.

version every month. Including Including interesting facts like the interesting facts like the interesting facts like the January release January release January release coming out in February. coming out in February. coming out in February. So when you read the So when you read the So when you read the release notes in release notes in release notes in VS Code, you wonder VS Code, you wonder VS Code, you wonder why it's February and you're why it's February and you're why it's February and you're reading the reading the reading the January notes like you're a January notes like you're a January notes like you're a month late. month late. month late. No, it's just the way No, it's just the way No, it's just the way we we we tracked our tracked our tracked our iterations. This rhythm iterations. This rhythm iterations. This rhythm was needed to keep up with was needed to keep up with the speed, but it the speed, but it also showed us that it's also showed us that it's also showed us that it's not about not about not about using using using more AI in more AI in more AI in everyday work, everyday work, everyday work, as as as many companies try to impose many companies try to impose many companies try to impose —you don't —you don't —you don't maximize tokens, maximize tokens, maximize tokens, you don't use you don't use you don't use AI every day. It's about AI every day. It's about AI every day. It's about evolving the entire evolving the entire evolving the entire system, how you system, how you system, how you deliver deliver deliver software, to software, to software, to better better better leverage AI leverage AI leverage AI throughout the throughout the throughout the process. And that's what process. And that's what process. And that's what makes this makes this makes this speed work, speed work, speed work, not just not just not just generate more generate more generate more code. I'll break this down into the code. I'll break this down into the code. I'll break this down into the first part, which first part, which first part, which many have already many have already many have already encountered—faster encountered—faster encountered—faster code delivery. And that's the code delivery. And that's the code delivery. And that's the easiest part. easiest part. easiest part. You have these 100X You have these 100X You have these 100X engineers who engineers who engineers who built this great built this great built this great scalable scalable scalable MCP plugin system, and they MCP plugin system, and they deliver code insanely fast, and deliver code insanely fast, and then more and more then more and more then more and more people on the team people on the team people on the team adopt those same adopt those same adopt those same patterns, and you see patterns, and you see patterns, and you see a lot more code a lot more code a lot more code being delivered faster.

being delivered faster. But the problem is that But the problem is that when you deliver when you deliver when you deliver faster, you are faced faster, you are faced faster, you are faced with code reviews and the with code reviews and the with code reviews and the question of how to question of how to question of how to deliver quality deliver quality deliver quality code that doesn't break code that doesn't break code that doesn't break all the time. And this is all the time. And this is all the time. And this is important for us at VS Code important for us at VS Code , as we , as we , as we deliver deliver deliver binaries to binaries to binaries to users' machines. So users' machines. So users' machines. So if something breaks, it if something breaks, it if something breaks, it costs costs costs much more to repair. But much more to repair. But much more to repair. But when you get up when you get up when you get up to speed by to speed by to speed by delivering quality delivering quality delivering quality code faster and more confidently code faster and more confidently , how do you know if , how do you know if , how do you know if you're delivering the you're delivering the you're delivering the right thing, or if right thing, or if right thing, or if it's even worth it's even worth it's even worth delivering? And this is where delivering? And this is where delivering? And this is where AI comes into play to AI comes into play to AI comes into play to ultimately learn ultimately learn ultimately learn faster and produce faster and produce faster and produce better products, something that better products, something that better products, something that many people lack when many people lack when many people lack when they write a lot of they write a lot of they write a lot of code—applying code—applying code—applying product vision product vision product vision and acquired knowledge. and acquired knowledge. So, let's start with So, let's start with faster delivery. faster delivery. faster delivery. Now that everyone Now that everyone Now that everyone can work in a can work in a can work in a stream, we stream, we stream, we can all really can all really can all really deliver faster, and a deliver faster, and a deliver faster, and a big part of that is big part of that is preparing the preparing the preparing the codebase for working with codebase for working with codebase for working with agents. This is an obvious agents. This is an obvious agents. This is an obvious thing that everyone should be thing that everyone should be thing that everyone should be doing already: thinking about doing already: thinking about doing already: thinking about how to make the codebase how to make the codebase agent-ready—creating agent-ready—creating agents.md files that are agents.md files that are agents.md files that are lightweight enough and lightweight enough and lightweight enough and give agents a nice give agents a nice give agents a nice map of the codebase to map of the codebase to map of the codebase to look at.

look at. look at. Mapping this Mapping this Mapping this means that each means that each means that each agent is now at your agent is now at your agent is now at your fingertips, has a slash fingertips, has a slash command that gives you a command that gives you a command that gives you a nice rough nice rough nice rough draft, and then you draft, and then you draft, and then you keep keep keep iterating. These are living iterating. These are living iterating. These are living documents that documents that documents that should evolve as the should evolve as the should evolve as the agent agent agent makes makes makes mistakes and works. And this is where mistakes and works. And this is where mistakes and works. And this is where I see I see I see many teams that many teams that many teams that have invested in developer experience have invested in developer experience benefit significantly, as benefit significantly, as quality documentation quality documentation quality documentation and onboarding for the and onboarding for the and onboarding for the codebase codebase codebase is read by the agent, is read by the agent, is read by the agent, which helps him a lot which helps him a lot which helps him a lot . Next are . Next are skills. VS Code has always skills. VS Code has always skills. VS Code has always been heavily invested in been heavily invested in been heavily invested in accessibility. So, accessibility. So, accessibility. So, one of the skills one of the skills one of the skills we added early on we added early on we added early on was was was implementing all the implementing all the accessibility best practices into a accessibility best practices into a tool that tool that everyone would use. Previously, one person had to be involved to one person had to be involved to get feedback, but get feedback, but get feedback, but now this skill now this skill now this skill is checked and is checked and is checked and supported by the person supported by the person supported by the person responsible for the responsible for the responsible for the accessibility direction accessibility direction . This is done to . This is done to make make make accessibility, which should accessibility, which should accessibility, which should work everywhere, work everywhere, work everywhere, available to available to available to everyone. Think about everyone. Think about everyone. Think about how you can turn how you can turn how you can turn your your your expertise into a skill expertise into a skill expertise into a skill to help those to help those to help those few users few users who keep who keep who keep asking about it. And asking about it. And asking about it. And finally: when you finally: when you finally: when you do everything right, the do everything right, the do everything right, the litmus test for us at VS Code litmus test for us at VS Code litmus test for us at VS Code is: is: is: can I, as a PM, can I, as a PM, can I, as a PM, effectively write code effectively write code effectively write code in the VS Code repository? Which is what in the VS Code repository? Which is what in the VS Code repository? Which is what I do. And how is this I do. And how is this I do. And how is this perceived by the perceived by the perceived by the engineering team?

engineering team? engineering team? How much extra How much extra How much extra work does this create? Does work does this create? Does work does this create? Does it work right out of it work right out of it work right out of the box? Does the box? Does the box? Does this work locally for me, or does it this work locally for me, or does it this work locally for me, or does it break at break at break at release? So, release? So, release? So, most of these most of these most of these quality improvements will quality improvements will quality improvements will ultimately prove that ultimately prove that ultimately prove that everyone can be more everyone can be more everyone can be more efficient in efficient in efficient in working with the working with the working with the repository. Another repository. Another repository. Another improvement, and a improvement, and a improvement, and a pretty significant one, is the pretty significant one, is the pretty significant one, is the switch to TypeScript Go. switch to TypeScript Go. If you have worked with If you have worked with any any any codebase where building, codebase where building, codebase where building, linting, or CICD linting, or CICD linting, or CICD takes a lot of time, takes a lot of time, takes a lot of time, you know how you know how you know how annoying it is. But annoying it is. But annoying it is. But developers are finding a developers are finding a developers are finding a way out. They can way out. They can way out. They can review the PR, review the PR, review the PR, switch to another switch to another switch to another task while the process is in progress task while the process is in progress task while the process is in progress , and do , and do , and do something else. But when you something else. But when you something else. But when you add agents—10 or add agents—10 or add agents—10 or 20 working 20 working 20 working simultaneously and simultaneously and simultaneously and hitting the same hitting the same hitting the same “bottleneck”—any “bottleneck”—any delay in your delay in your delay in your CICD becomes critical. CICD becomes critical. For us, TypeScript Go has been a For us, TypeScript Go has been a tenfold tenfold tenfold speedup in builds speedup in builds , which is a huge improvement when , which is a huge improvement when , which is a huge improvement when scaling scaling scaling automated PRs from automated PRs from automated PRs from agents who agents who agents who need a CICD loop for need a CICD loop for need a CICD loop for feedback.

feedback. And this And this allows me allows me allows me to create great PR. to create great PR. to create great PR. At the beginning of the year, I At the beginning of the year, I At the beginning of the year, I proposed an proposed an proposed an “ask questions to Lynn” feature that “ask questions to Lynn” feature that “ask questions to Lynn” feature that I thought was much I thought was much I thought was much needed, and needed, and needed, and presented it to presented it to presented it to the team so they the team so they the team so they could try it out in the could try it out in the could try it out in the product. Within a few product. Within a few product. Within a few weeks, we had a weeks, we had a weeks, we had a perfectly perfectly perfectly polished polished polished interface interface interface based on my based on my based on my initial, rather initial, rather initial, rather crude PR. But it crude PR. But it crude PR. But it unlocked those unlocked those unlocked those discussions and discussions and discussions and everyone everyone everyone else's involvement because we had else's involvement because we had else's involvement because we had laid the foundation and the laid the foundation and the laid the foundation and the idea and experience were idea and experience were idea and experience were clear. Then, when clear. Then, when clear. Then, when agents actually agents actually agents actually work on the work on the work on the interface, we've all interface, we've all interface, we've all encountered the situation where encountered the situation where encountered the situation where the agent delivers the agent delivers the agent delivers the interface, says the interface, says the interface, says everything looks everything looks everything looks perfect, and you perfect, and you perfect, and you open it—and open it—and open it—and everything is shifted. Even with everything is shifted. Even with everything is shifted. Even with SVG, even with the SVG, even with the SVG, even with the newest models newest models , you still , you still , you still run into this. And this is the run into this. And this is the run into this. And this is the number one thing I number one thing I number one thing I keep keep keep telling everyone: today, telling everyone: today, telling everyone: today, if your agent if your agent if your agent can't use can't use can't use your app or your app or your app or product product product directly to directly to directly to get get get feedback, is it feedback, is it feedback, is it working? And this is a very working? And this is a very working? And this is a very big investment big investment big investment that pays off that pays off that pays off every time you every time you every time you work on the work on the work on the interface. We interface. We interface. We invested in two invested in two invested in two areas. One of them is the areas. One of them is the component browser, an component browser, an component browser, an automated automated automated build that build that build that runs whenever runs whenever changes are made in VS Code.

Let me Let me open this in this open this in this open this in this ticket as an example. In ticket as an example. In ticket as an example. In this case, we this case, we this case, we added a back button added a back button added a back button to the to the to the customization screen, and customization screen, and customization screen, and because the changes are because the changes are because the changes are extensive, we have an extensive, we have an automated automated process running that takes process running that takes process running that takes screenshots of screenshots of screenshots of each component in each component in each component in VS Code and points out the VS Code and points out the VS Code and points out the differences. This differences. This differences. This allows us allows us allows us to detect unexpected to detect unexpected to detect unexpected changes, such as changes, such as changes, such as when we change one when we change one when we change one component and it component and it component and it causes a causes a causes a domino effect, causing something else domino effect, causing something else domino effect, causing something else to shift or to shift or to shift or another icon to disappear. another icon to disappear. another icon to disappear. This also allows you to This also allows you to This also allows you to quickly review PRs quickly review PRs quickly review PRs because you can immediately because you can immediately because you can immediately see what exactly that PR see what exactly that PR see what exactly that PR does. And this is something does. And this is something does. And this is something we've asked every we've asked every we've asked every developer for before. We developer for before. We developer for before. We asked them to at least asked them to at least asked them to at least add a video or add a video or add a video or screenshot of what screenshot of what screenshot of what they had implemented they had implemented they had implemented so we didn't have to so we didn't have to so we didn't have to start everything from scratch start everything from scratch start everything from scratch and we could quickly and we could quickly and we could quickly evaluate and provide evaluate and provide evaluate and provide feedback. This is now feedback. This is now feedback. This is now automated using the automated using the component browser. Next, component browser. Next, we have a we have a we have a self-correction loop, because self-correction loop, because self-correction loop, because in VS Code we launch in VS Code we launch in VS Code we launch a web browser and a web browser and a web browser and actually open a actually open a actually open a website. That's what VS website. That's what VS website. That's what VS Code is. It's all HTML. We Code is. It's all HTML. We Code is. It's all HTML. We can can can use use use Playwright to Playwright to Playwright to automate automate automate many processes in the many processes in the many processes in the browser. So, we browser. So, we browser. So, we have a "slash launch" skill have a "slash launch" skill have a "slash launch" skill that simply opens that simply opens that simply opens VS Code and gives it a VS Code and gives it a VS Code and gives it a specific script specific script specific script to run in VS to run in VS to run in VS Code. This is a very powerful Code. This is a very powerful Code. This is a very powerful way to diagnose way to diagnose way to diagnose problems because you problems because you problems because you can get can get can get logs as you go, and logs as you go, and logs as you go, and also check the also check the also check the "before" and " "before" and " after" fixes without having to click after" fixes without having to click after" fixes without having to click everything manually. So, you everything manually. So, you everything manually. So, you can just can just can just click "run" click "run" , move on to the , move on to the , move on to the next task, next task, next task, and then come back

and then come back and then come back when the agent when the agent when the agent checks its checks its checks its work, since it work, since it work, since it has access to your has access to your has access to your application. It's worth application. It's worth application. It's worth noting that this is noting that this is noting that this is great since we great since we great since we have Playwright and are have Playwright and are have Playwright and are a web app on a web app on a web app on Electron, but there are Electron, but there are Electron, but there are many other solutions. many other solutions. many other solutions. One of my One of my One of my favorite favorite favorite tools is Xcode MCP, tools is Xcode MCP, tools is Xcode MCP, as I work on as I work on as I work on iOS apps. When iOS apps. When iOS apps. When you see how Xcode MCP you see how Xcode MCP you see how Xcode MCP creates an iOS app in VS creates an iOS app in VS creates an iOS app in VS Code without the involvement of Code without the involvement of Code without the involvement of Xcode itself, opens it, Xcode itself, opens it, Xcode itself, opens it, clicks on elements, and clicks on elements, and clicks on elements, and takes screenshots—this is takes screenshots—this is takes screenshots—this is real magic that real magic that real magic that provides very provides very provides very fast fast fast feedback. So, for feedback. So, for feedback. So, for Android there are also Android there are also Android there are also other tools that other tools that other tools that allow you allow you allow you to implement this. Let's to implement this. Let's to implement this. Let's talk about talk about talk about code review, which code review, which code review, which I believe is the I believe is the I believe is the next big next big next big investment. We investment. We investment. We know that in the past, know that in the past, know that in the past, when we enabled when we enabled when we enabled automatic automatic automatic code reviews through code reviews through code reviews through GitHub Copilot, we weren't GitHub Copilot, we weren't GitHub Copilot, we weren't sure about sure about sure about the results, but they have improved significantly in the results, but they have improved significantly in the results, but they have improved significantly in recent months and recent months and recent months and weeks weeks weeks . And now . And now . And now it's a mandatory it's a mandatory it's a mandatory check every time check every time check every time someone opens a someone opens a someone opens a PR, and new improvements are in the works PR, and new improvements are in the works PR, and new improvements are in the works to to to adjust the level of adjust the level of adjust the level of granularity of granularity of granularity of the check depending the check depending the check depending on the risks in the on the risks in the on the risks in the repository. You repository. You repository. You can choose can choose can choose low, medium, or low, medium, or low, medium, or high high high verification levels to find a verification levels to find a verification levels to find a balance between cost balance between cost balance between cost and performance.

and performance. and performance. The main thing for us is that The main thing for us is that after after after the review is complete, people the review is complete, people the review is complete, people don't even look don't even look don't even look at the PR until all at the PR until all at the PR until all comments are comments are comments are taken into account and taken into account and taken into account and resolved. Okay, we've resolved. Okay, we've resolved. Okay, we've sped up development sped up development , now we need to , now we need to , now we need to maintain quality. For maintain quality. For maintain quality. For us, as us, as us, as GitHub users, an GitHub users, an GitHub users, an important indicator important indicator important indicator is, of course, tickets ( is, of course, tickets ( issues). Have any of you ever issues). Have any of you ever created a created a created a ticket on GitHub? Has anyone ticket on GitHub? Has anyone ticket on GitHub? Has anyone had had had problems with VS Code before? Yes, problems with VS Code before? Yes, problems with VS Code before? Yes, cool. It seems so. Perfectly. The cool. It seems so. Perfectly. The cool. It seems so. Perfectly. The previous audience previous audience previous audience raised fewer hands, raised fewer hands, raised fewer hands, so that's good. Yes, this is a so that's good. Yes, this is a so that's good. Yes, this is a really powerful really powerful really powerful indicator. As I indicator. As I indicator. As I said, we now sometimes said, we now sometimes said, we now sometimes get even get even get even better descriptions of problems better descriptions of problems better descriptions of problems thanks to AI and the fact that thanks to AI and the fact that thanks to AI and the fact that more people with more people with more people with different language different language different language backgrounds can backgrounds can backgrounds can create quality create quality create quality reports. But we also reports. But we also reports. But we also receive a lot of receive a lot of receive a lot of requests. requests. requests. Previously, engineers Previously, engineers Previously, engineers disassembled them manually. disassembled them manually. And now AI And now AI filters out spam, filters out spam, filters out spam, supplements data, supplements data, supplements data, translates, and translates, and translates, and assigns people assigns people assigns people responsible for responsible for responsible for specific areas to specific areas to specific areas to improve the quality of improve the quality of improve the quality of ticket work. We ticket work. We ticket work. We also added also added also added human control to this.

human control to this. Since an agent can Since an agent can make mistakes, we make mistakes, we make mistakes, we need need need feedback, and to feedback, and to feedback, and to fix fix fix duplicate tickets, duplicate tickets, duplicate tickets, we created a we created a we created a Chrome extension. Chrome extension. So, we have an agent So, we have an agent that does the work, that does the work, that does the work, and a feedback mechanism and a feedback mechanism where people can where people can make corrections, make corrections, make corrections, which are then which are then which are then taken into account in the taken into account in the agent's further work. It's important agent's further work. It's important to remember this: when to remember this: when to remember this: when you delegate work to you delegate work to you delegate work to agents, you must agents, you must agents, you must provide provide provide the opportunity for human the opportunity for human the opportunity for human feedback. feedback. Our other source of Our other source of data is a large data is a large data is a large number of number of number of error stacks. Whenever error stacks. Whenever error stacks. Whenever an exception occurs, which an exception occurs, which an exception occurs, which happens in happens in happens in most programs, you most programs, you most programs, you get an get an get an error stack trace report, and we error stack trace report, and we error stack trace report, and we don't actually don't actually don't actually use any use any use any third-party third-party third-party tools for that. All of this is tools for that. All of this is tools for that. All of this is based on our based on our based on our own data, and we own data, and we own data, and we developed all the developed all the developed all the relevant relevant relevant tools ourselves. tools ourselves. tools ourselves. For the most part, we For the most part, we For the most part, we already had them. We already already had them. We already already had them. We already had a had a had a machine learning system machine learning system machine learning system to sort them. And to sort them. And to sort them. And now this is done now this is done now this is done using a using a using a combination of different combination of different combination of different criteria. So, we criteria. So, we criteria. So, we collect raw collect raw collect raw telemetry, telemetry, telemetry, filter it— filter it— filter it— about 51 billion about 51 billion about 51 billion records per day. We records per day. We records per day. We filter only filter only filter only those error stacks that those error stacks that those error stacks that contain complete contain complete contain complete information. Then we information. Then we information. Then we group and group and group and distribute them, which is distribute them, which is distribute them, which is more like the more like the more like the process of creating process of creating process of creating prints. Finally, prints. Finally, prints. Finally, after all these steps after all these steps , we create 10 , we create 10 , we create 10 tickets, assign tickets, assign tickets, assign them to the owners of the them to the owners of the them to the owners of the corresponding areas, corresponding areas, corresponding areas, and automatically and automatically and automatically create a PR to create a PR to create a PR to try to fix try to fix try to fix the problem. This is our the problem. This is our the problem. This is our error page where error page where error page where we see all the stacks, we see all the stacks, we see all the stacks, the number of hits, and the number of hits, and the number of hits, and the users the users the users affected, to affected, to affected, to quickly respond to quickly respond to quickly respond to any threats.

any threats. As soon as all this works , we get a , we get a , we get a created ticket and an created ticket and an created ticket and an automatically automatically automatically opened PR. This is a ticket opened PR. This is a ticket opened PR. This is a ticket containing an error with containing an error with containing an error with preliminary preliminary preliminary diagnostics, after diagnostics, after diagnostics, after which a PR is opened. which a PR is opened. The agent The agent logs into the system on their own and logs into the system on their own and logs into the system on their own and finds out why it finds out why it finds out why it happened and who was happened and who was happened and who was responsible. In responsible. In responsible. In this case, he this case, he this case, he found who added found who added found who added the changes that caused the changes that caused the changes that caused this and found that the this and found that the this and found that the cancellation request cancellation request cancellation request was missing from the was missing from the was missing from the RPC protocol. So the RPC protocol. So the RPC protocol. So the agent already opened the PR, agent already opened the PR, agent already opened the PR, we could just merge we could just merge we could just merge it, and it's a really it, and it's a really it, and it's a really great great great automated automated automated way. You don't way. You don't way. You don't care about bugs, you care about bugs, you care about bugs, you just want a more just want a more just want a more stable stable stable codebase. All of this is codebase. All of this is codebase. All of this is automated: automated: automated: first, sorting first, sorting first, sorting using agents using agents using agents and deterministic and deterministic and deterministic systems, then systems, then systems, then creating a ticket, creating a ticket, creating a ticket, passing it to an agent, a passing it to an agent, a multi-agent multi-agent system working on the PR, and system working on the PR, and system working on the PR, and finally, the finally, the automatic automatic automatic implementation of implementation of implementation of fixes, which are fixes, which are fixes, which are still overseen by a still overseen by a still overseen by a human for human for human for PR approval. That's it. Previously, VS Code PR approval. That's it. Previously, VS Code PR approval. That's it. Previously, VS Code released updates released updates released updates according to the “YOLO” principle: on according to the “YOLO” principle: on according to the “YOLO” principle: on the day of release, everything was the day of release, everything was the day of release, everything was stable, we stable, we stable, we tested tested tested and sent and sent and sent updates to 100% of updates to 100% of users. We were users. We were users. We were simply opening the simply opening the simply opening the floodgates. But through floodgates. But through floodgates. But through agents, we want to agents, we want to agents, we want to minimize the risks of minimize the risks of minimize the risks of this process, and this process, and this process, and perhaps we should have perhaps we should have perhaps we should have done this done this done this earlier. So now we earlier. So now we earlier. So now we are doing a phased are doing a phased are doing a phased rollout. During rollout. During rollout. During deployment, we deployment, we deployment, we monitor monitor monitor error logs, tickets, and error logs, tickets, and error logs, tickets, and everything else. As conscientious everything else. As conscientious web application developers, we web application developers, we take the take the take the same approach because

same approach because same approach because rolling back installed rolling back installed rolling back installed programs is too programs is too programs is too expensive a process. expensive a process. Okay, now we Okay, now we have faster have faster have faster delivery, delivery with delivery, delivery with delivery, delivery with quality. How do we quality. How do we quality. How do we actually do this actually do this actually do this at a speed that at a speed that at a speed that allows us to learn allows us to learn allows us to learn faster? Which is a much faster? Which is a much faster? Which is a much bigger challenge for bigger challenge for bigger challenge for artificial intelligence. artificial intelligence. artificial intelligence. You really want You really want You really want to create a better to create a better to create a better product faster. So, product faster. So, product faster. So, one example is one example is one example is our VSCBench. We are our VSCBench. We are our VSCBench. We are releasing an agent releasing an agent releasing an agent product, so it is important product, so it is important that we always have that we always have that we always have our own our own our own product evaluations on hand that are product evaluations on hand that are product evaluations on hand that are easy to expand. This is easy to expand. This is easy to expand. This is all in the GitHub issue. It's all all in the GitHub issue. It's all all in the GitHub issue. It's all on GitHub. When we on GitHub. When we on GitHub. When we create a ticket to create a ticket to create a ticket to add a new add a new add a new script, the agent takes script, the agent takes script, the agent takes the template and independently the template and independently the template and independently deploys this deploys this deploys this script for you. script for you. script for you. So, we're trying to make it So, we're trying to make it So, we're trying to make it as as as easy as possible to add easy as possible to add easy as possible to add all all all developer scenarios that developer scenarios that developer scenarios that come up in come up in come up in requests, requests, requests, customer conversations, and through customer conversations, and through customer conversations, and through other channels to other channels to other channels to integrate them into our integrate them into our integrate them into our assessments for assessments for assessments for further further further improvement. And then improvement. And then improvement. And then also confirm that also confirm that also confirm that all of this improves in all of this improves in all of this improves in offline mode and offline mode and offline mode and then in online then in online experiments. So, experiments. So, experiments. So, this is a classic this is a classic product life cycle for us. And we product life cycle for us. And we see these interesting things see these interesting things see these interesting things in our stack.

in our stack. in our stack. Understanding your Understanding your Understanding your grades is critically grades is critically grades is critically important. One important. One important. One experiment, which experiment, which experiment, which you can read about on you can read about on you can read about on our blog, was to our blog, was to our blog, was to provide a simple provide a simple provide a simple evaluation script: evaluation script: evaluation script: just create a file just create a file just create a file with “hello world”. We were simply with “hello world”. We were simply with “hello world”. We were simply testing our testing our testing our scoring system scoring system scoring system from start to finish from start to finish from start to finish and the reaction of different and the reaction of different and the reaction of different models. For the models. For the models. For the same five- same five- same five- character file, the most expensive character file, the most expensive character file, the most expensive model used model used model used 70 times more 70 times more 70 times more tokens. And it wasn't the tokens. And it wasn't the tokens. And it wasn't the most most most intelligent intelligent intelligent model. You can model. You can model. You can read more about it read more about it read more about it on the blog. We're on the blog. We're on the blog. We're not really not really not really talking about talking about talking about specific models, but specific models, but specific models, but just understanding just understanding just understanding how your models how your models how your models interact with your interact with your interact with your system is a very system is a very system is a very important exercise. important exercise. Next is prototyping , which I do a lot , which I do a lot , which I do a lot because I noticed because I noticed because I noticed that I didn't want to put that I didn't want to put that I didn't want to put most of my most of my most of my work directly work directly work directly into VS Code PRs. This is a great into VS Code PRs. This is a great into VS Code PRs. This is a great way to start way to start way to start a conversation, but a conversation, but a conversation, but most of the time I most of the time I most of the time I want a quick want a quick want a quick discussion discussion discussion since we since we since we meet every day meet every day meet every day and want to have quick and want to have quick and want to have quick feedback loops feedback loops feedback loops : here's an idea, : here's an idea, : here's an idea, here's what it could here's what it could here's what it could look like, what if look like, what if look like, what if we do it like this?

we do it like this? we do it like this? The next day you The next day you The next day you come back with come back with come back with updated updated updated prototypes and prototypes and prototypes and continue this continue this continue this conversation. So, conversation. So, conversation. So, much faster much faster much faster feedback— feedback— from monthly cycles from monthly cycles from monthly cycles to weekly or to weekly or to weekly or daily sprints where daily sprints where daily sprints where you’re constantly working you’re constantly working you’re constantly working on issues, and on issues, and on issues, and prototypes prototypes prototypes open up deeper open up deeper open up deeper discussions about what the discussions about what the user experience should look like. It also user experience should look like. It also means we have means we have means we have consistent consistent consistent accountability for the accountability for the accountability for the work: as I work: as I work: as I mentioned, daily mentioned, daily mentioned, daily sprints, smaller sprints, smaller sprints, smaller teams, smaller work teams, smaller work teams, smaller work streams that really streams that really streams that really prioritize and prioritize and prioritize and push narrower push narrower push narrower directions to directions to directions to move faster. So move faster. So , try this to , try this to , try this to find your “ find your “ find your “ bottlenecks” to really bottlenecks” to really bottlenecks” to really produce a better produce a better produce a better product faster, product faster, product faster, learn faster, and learn faster, and learn faster, and find and eliminate the find and eliminate the find and eliminate the next obstacle. And next obstacle. And next obstacle. And don't just don't just don't just customize how you work customize how you work customize how you work with agents, but with agents, but with agents, but customize how customize how customize how they get they get they get feedback— feedback— how do you build these how do you build these how do you build these cycles? And then, after cycles? And then, after cycles? And then, after those moments of success, those moments of success, those moments of success, figure out how you figure out how you figure out how you can improve that can improve that can improve that feedback loop feedback loop feedback loop for the for the for the next stage to next stage to next stage to eliminate what’s eliminate what’s eliminate what’s stopping you from moving stopping you from moving stopping you from moving even faster? And this is a even faster? And this is a even faster? And this is a constant constant constant intercyclical process intercyclical process intercyclical process between all of them. That's all between all of them. That's all . Come visit the . Come visit the . Come visit the VS Code booth, I'd be happy VS Code booth, I'd be happy VS Code booth, I'd be happy to chat.

to chat. to chat. Happy coding.