Scaling Rust in the Enterprise: What Changes When Rust Gets Big

All right. Hey everyone, and welcome to scaling Rust in the enterprise. Welcome to our webinar with Incredibuild. My name is Orhun Parmaksiz. I'm a Rust developer advocate at JetBrains, and I'll be your host today. Today we have very special guests. I'm joined by Nir Stern, VP of product at Incredibuild, and Dory Exton, chief evangelist at Incredibuild. Welcome, guys. How's it going? >> Awesome. Thanks for having us. >> Great. >> Yeah, great to hear that. So today we will talk about Rust and Rust in enterprise. So as we all know, as Rust moves into larger teams and mixed language code bases, there are some bottlenecks. For example, build times, testing, CI can sometimes be serious problems.

So today we are looking at Incredibuild, a product that will tries to solve those problems. And along with that, we will also talk about RustRover. So we will basically give you some tips on how to keep development Rust development moving at the enterprise scale. So with that, I will give the stage to Dory to begin with a presentation about uh Incredibuild, and then we'll be back. I will be talking about RustRover, and lastly, Nir will take the stage. And we will ask uh we will answer any of your questions, so feel free to use the chat in the meantime. So Dory, you have the stage. >> Thank you, Orhun. So just to set the tone, this is less of a presentation of about Incredibuild, although we will have a little bit of it. And more, you know, how Rust looks like when it meets scale in enterprises. And when I'm when I'm saying enterprises, I don't mean the the large banks. I'm I'm meaning, you know, large teams, complex software development. This is This is where we're taking it. So, let me share the presentation first. Let's see.

Yeah. I think that Yeah, it's it's coming. So, welcome. Again, my name is Dori Exterman. And I'm chief evangelist at Incredibuild. And today, my purpose is to share with you the challenges and the opportunities of of Rust meeting dev organization at scale, as I said. So, from the vast experience of Incredibuild with the largest and most complex leading dev teams in the world, we have tons of information and tons of experience to know what happens when when these these trends are happening and meeting real life. Uh over the last 3 years, Rust has crossed a major operational pressure. From very promising experimental language into core back-end infrastructure. Uh multi-team, mono repos, mission-critical production engine, all of that. So, if your bet was on Rust, your bets were placed in the right place.

And when Rust scales at enterprises, we see the scale across three dimensions. The first is is the growing usage across engineering teams. The sheer amount of people working on the same projects. And and the project themselves. As the code base complexity it grows in complexity, it grows in size, and which with that it brings a lot of challenges. And the last is how it plays with the other languages and tools that were there before Rust came in. This is when we're speaking about large projects. Of course, there are projects that are only Rust. But this is, you know, more in the periphery when you're speaking about large enterprises. And I'll show you a few a few examples soon. So, these scaling factors, they introduce new operational bottlenecks that only increase with the rapid adoption of a gentle coding and autonomous software factories. I don't know if you have heard about this term, but this is this is the trend. This is where software organizations are going, autonomous software factories.

And in this session, I'm going to explore together with you the challenges that are introduced when scaling Rust at large teams and complex codebases. And how these challenges are addressed by leading dev organizations today, because they're already there. So, to understand where we are today, let's first look at the macro adoption data. Rust has officially transitioned from a niche programming language into an enterprise infrastructure standard, let's say. So, let's take a look at the market momentum. According to a recent JetBrains survey, there are now over 2 million active Rust developers worldwide with nearly 50% of of surveyed dev organization that reported they're running non-trivial Rust workloads in production. These are huge adoption numbers. And since the survey was published, I believe the numbers are still growing. The second is the production security.

This, for example, is taken from Google's Android data. Uh after Google systematically migrated memory-critical subsystems to Rust, vulnerability density dropped by a factor of 1,000. This is a huge number, down to just 0.2 CVs per million line of code. This brought X4 reduction in production rollback rates and shaved 25% off code review times. This is This is really a great validation to the Rust promise that that was there all along. And the third is the hyperscale efficiency. AWS built Firecracker in Rust and clear Rust to power Lambda and Fargate. And it delivers micro VM boot times to almost 100 milliseconds with less than 5 megabytes of of memory overhead per instance. And this unlocks massive infrastructure density. And this is These are the promises of Rust, uh parallelization, critical safety, and we will soon see some others. So, to better understand how it plays in the enterprises' cases and the trends of Rust, let's examine how the value proposition of Rust are adopted across verticals and use cases, which is different approach than looking at specific companies.

So, one of the value and promises of Rust is security and safe code execution. And that was always the main reason of Rust of existence. And in automotive and embedded systems, but also in gaming, these capabilities are are critical. The challenge here isn't just writing isolated Rust. It's the integration of decades of native C and C++ SDKs. And by wrapping these unsafe C codes behind strict Rust API boundaries, teams are able to modernize core systems without memory safety regression. And this is This is all the promise that that Rust gave, and this actually works now in real large systems. Another critical aspect of Rust is predicted, but also safe real-time performance. And this means that it's a it's a major goal where across fintech, edtech, and telecom, but especially across fintech and and trading. And the goal here is to eliminate the unpredictable garbage collection collection latency, the the spikes that are happening under peak load.

And using the lock-free concurrency and zero data copying handling, teams achieve strict sub-millisecond P99 latency goals with Rust. And this is This is a huge achievement, especially when we are talking about real time. Another area is is in the AI infrastructure, developer tooling, SaaS. And this is where higher-level languages like Python, they hit multi-threading limits. And the winning pattern that emerges here is that you kind of have a Rust async core and it's wrapped in a developer-friendly bindings that can be consumed by these other languages. And this is this is a big win as well. So across every vertical, Rust adoption succeeds through all these benefits. This safe execution, performance determinism, scale and parallelization, but also as as as part of the you know, its ability to integrate and and the extensibility capabilities that Rust has as part of the language.

Let's now see some of these benefits and the way that they are gained in real-world enterprise case studies. So we are driving from the from the vertical and use cases to specific problems, let's say. So at Microsoft, teams are incrementally replacing legacy C and C++ code inside the core Windows kernel and Azure subsystems. This directly attack the root cause of memory safety bugs. This historically made you know, something like 70% of Windows vulnerabilities. So this is what happens in Microsoft at scale. At Cloudflare, for example, they built Pingora. This is a different kind of use case. This is more of the parallelization. This This Pingora is a is a custom async rock Rust proxy framework that replaced the very famous popular engine X stack. And today, Pingora processes over 40 million HTTP requests per second at the edge with zero memory crashes and zero downtime reloads. That's a great validation of Rust scalability and safety.

And last example, a bit different from the usual benefit that we see Rust associated with, is at 1Password, that instead of rewriting business and security logic for every operating system, they built a single headless Rust core, compiled natively to Windows, macOS, Linux, and also iOS, Android, and WebAssembly. So, they get 100% shared cryptographic logic across six different client platforms. This is This is a great achievement for any R&D team that needs to support, you know, six clients. Uh so, across global infrastructure, Rust Rust succeeds when when when when there is a need for safety, for performance, for unified core logic, uh parallelization scalability extensibility, all these This is what's being proven today by enterprises, and that that these are good news for Rust developers and for the Rust ecosystem. But now, as these codebases grow from very small services into multi-team monorepos, we encounter a severe friction point.

And this is the developer feedback latency. So, what I'm showing today now uh in this slide is according to a JetBrains Rust ecosystem survey that was uh uh was was executed uh recently. And in that survey, 46% of developers reported that in every small change, uh they rebuild and make Rebuild is a major problem. So, this is this is a big number. It's not It doesn't mean, you know, it's it's the way these developers perceive it. And 46 developers perceive it the build as a major problem. On small changes. And actually, this problem extrapolate when we are going to look more deeply about a genetic coding, which code writing is is even in higher velocity. But But we'll speak about it a bit later. Another Another metric that caught my eyes that one out of five developers waits over a minute every time they make a trivial edit.

And this is probably, you know, this one out of five. This is probably the the scaling Rust in their enterprises kind of use cases. So, probably the four out of five are the small projects, you know, the hobbyist or small web services, etc. And the one out of five is what we're going and expecting Rust to become. And these people, the one out of five, already experience on every trivial change a one more than one minute build. Another metric is that 25% of engineering organizations see CI build performance as a primary blocker for shipping software. And these bottlenecks are only increasing as Rust codebases are getting bigger and as a genetic coding highly increases code iteration frequency. And it's going to dramatically increase in the in the upcoming years. So, when a developer or AI agent waits more than two minutes or so for cargo check or for cargo test after changing a single line of code, context switching sets in. Context switching is for developer for AI it's just sitting idle.

So, at enterprise scale feedback latency stops being a minor inconvenience and it becomes really a tax on the entire engineering system, especially with the transition to a genetic software factories. Let's wait with that. So, I think just before I show the next slide, I think one of the questions is why. Why does cargo build struggle at enterprise scale? And it comes down to the mechanism and the way the Rust compiler is built. Sorry for that. And uh Yeah. I don't want to go too deeply into uh how Rust compiler works, but the main point is that the three main stages that are part of the Rust build workload are highly dependent on each other and somewhat sequential. The nature of these bottlenecks limits parallelization.

I mean parallelization on the local machine, but also when we look at distributions, we will see that there as well. And require dedicated strategies for addressing this challenge when code bases are large and especially when they are heterogeneous and include not only Rust. We will look at that in a minute. Before you can fix these bottlenecks, these dependencies and sequential tasks, etc., you have to identify them. You have to measure their times, their frequency, the acceleration potential, you know, of addressing them, how where which code changes change more frequently and and and and execute which builds. And enterprise build optimization requires moving away from just these attack CLI logs towards real-time build visualization. So, in this slide, Nir is going to show that uh uh in more details, but I just wanted to give you a perspective. In this slide, you see an incredible build visualization of Rust compilation workload.

Every green bar represents a process that is being executed as part of the build or part of the Rust build. And as you can see, this visualization give you more information, you know, automatically when you look at it than just going through logs. And you can clearly see here that the amount of parallelization Here, we can see two machines, you know, one is uh the local machine and the next machine is the machine to which tasks are being distributed, but as you can see, not many tasks can be distributed and run in parallel the entire duration of the 8 minutes that this Rust project is running. And this means that even if you throw 128 core machine on this project, it will still only finish in eight minutes. So, you cannot accelerate it with more compute power. You need other measures to to to to accelerate this kind of problems. And for example, caching is is the best way to to to address sequential tasks.

What you can also see is that there are some tasks that are highly sequential. And you know, if we will be able to break this sequentiality, it will be better, especially if it will be parallel. And there are also dependencies. Where you see, you know, these white areas, these are dependencies that are waiting for other things to complete before they can start running. So, this is a way that you analyze a build and you try to understand, you know, where should I improve it, where are the bottlenecks, and and where where, you know, where I'm I'm missing the performance. But this is a very simple and and kind of naive uh build build monitor. There are much more complex than this one. One of these very, very complex build monitor, I'm not going to show it to you because it will take several screens, you know, and there are too many information, is the Android operating system build. Uh it's called AOSP. Uh and Android, the Android build is a great example of how Rust looks like in a very large and complex project. I'm not sure how many of you all know that Rust is already part of Android and it's actually been there for a few Android versions already.

And it's getting more and more uh bigger part in in AOSP. So, uh and it's a you very good example, you know, it's the largest one of the largest open sources that exist today. It requires strict requirements for safe execution, for performance, extensibility. Everyone has it on their phone, on their automo- on their on their cars, you know, everywhere, TVs. And it's used by the world's top automotive and embedded organizations. And as it exists for several years, it showcases the evolution of a large critical infrastructure code bases. And Rust is playing a growing part in it, as I said. So, let's check out the latest Android version. That's ASP 17, that was recently released by Google. So, Rust almost almost never compiles in a vacuum in the enterprises. In a production repository like Android, Rust C sits inside a very complex execution graph. This execution graph contain 80,000, more than 80,000 clang invocation, which is C++ for those of you who are not familiar with clang, Java, Kotlin compilers, bytecode optimizations like R8, and over 20,000 custom AIDL code generators.

And this is this is a huge heterogeneous build graph, which consists of more or less 20 different tools and 120,000 task invocations. Uh there is a recent article I wrote about what we can learn, you know, from about them on on the Android development and the trends by looking at these 120,000 invocations. So, I learned this very very carefully. And in this huge build graph, Rust is only the sec- is only second to C++. But when speaking of build times, not all compilers are born equal. Uh C++ compilations, for example, as opposed to what I showed you this before in the slide before in the build monitor of Rust, C++ are are called ridiculously parallel. This is a ridiculously parallel workload. So, they are highly parallel. They can consume 128 cores and more than that, and Rust, as opposed to that, as we've seen, is much less so. So, sequential tasks and nested dependencies, they play a critical role in role in build performance and velocity. So, it's not only the amount of invocation, it's the way these invocations are ordered, dependent on other, and and sequential.

And another trend that can be seen in the growth of Rust C invocations part of the AOSP build is that the Rust compilation invocations surged by 61% between AOSP 15 and AOSP 17. So, that means that Rust is swallowing legacy C and C++ subsystems rapidly. If your build acceleration tools when you look at all that, you know, all these languages, 20 tools, you know, if your build acceleration tools and and your uh you know, your your Rust strategies, uh when you're going to the enterprise, they only understand Cargo, then you're ignoring 80% in AOSP of the build graph. And if it only understands C++, you're ignoring the other critical bottleneck that are the remaining 18 tools that are uh that are introduced in AOSP. So, accelerating Rust at scale requires understanding the entire polyglot system and not only our Rust or C++ build.

And it's important, very important to note, especially in large organizations, that it's not only when we're looking at solutions for these polyglot systems and Rust inside this polyglot, it's super important to understand that it's not about just build times. It's also about visualization, the ongoing bottleneck and dependency analysis, the troubleshooting of failures, testing, various kind of testing, and more. So, when we're looking at the future, we when we're looking at Rust in large organizations, we need to understand, you know, how we are working with the tools that can work with Rust, but also with the other players, uh kids in the playground, if you want to say it like that. So, look at what happens when we benchmark this AOSP 17 polyglot graph on a standard 32-core workstation. Without acceleration, AOSP 17 build takes more than 2 hours, which is a huge number.

And if If accelerate C++, it will bring you down to 70 minutes. But, this is still huge for every code change. And after accelerating C C++, there are, I think, something like 17 additional tools, you know, Java, Kotlin, IDL, etc., that are part of the AOSP 17 build. And that's only just just the AOSP vanilla, you know, when when you're looking at car manufacturers, they have their own version, they have their own Android apps, you have, you know, additional build system, etc. It It bloats, but I'm just speaking about the open source vanilla. And then, if you accelerate everything, but you exclude Rust, so you accelerate something like 17 tools, it goes down to 27 minutes. And when you add Rust acceleration as well, it brings the build down to 19 minutes. From 131 minutes down to 19 minutes. But, it's important to see that Rust the 8 minute that Rust adds, these are dramatic. It's It's 30% better performance. Uh, you know, on top of the major acceleration of 27 minutes, major performance achievement to accelerate this additional 30% and get down to 19 minutes. There are ways to getting down this down further, but they require other means.

This what I'm showing you here is, you know, the way to accelerate it out of the box with with incredible shared caching and and uh distributed execution. So, incredible shared caching, this is why it works here so well, because it's not specific for one tool, it can work with all the tools that are part of the AOSP uh 17 and also with with customer custom tools and and, you know, processes and scripts and and you name it. So, this is This is the way to achieve, you know, the the playground and all the players that Rust is part of. I don't I didn't show you here the build monitor, but if you look at the build visualization over of an AOSP 17, you need to have a build visualization can show you all tests, not only Rust, not only C C++, but everything that's part of the build, all the dependencies, and you need to query that and and a lot of other stuff that are not part of this of this session. Uh so, this is just, you know, one example of our Rust-specific solutions are not enough when looking at how Rust is used in enterprise-scale software development. And, you know, Android builds is something that our customers are are daily working with, you know, the large automotive, the large embedded customers, etc.

And it's not only about the build itself. Developers and agentic coding, they don't experience Rust C in isolation, even if we are speaking about compilation. They experience the entire feedback graph. It's not just a compilation of Rust, even if it's a Rust project without all these other languages. So, when a developer saves a file, their iteration loops span across multiple phases. Do you have the dependency and the static checks as part of Rust, the if it's multi-language linking, but you have other stuff steps as well, you know, like tests and validation, and you have several kinds of tests, unit testing, integration, API testing. Sometimes you have other steps as well as part of the pipeline like code signing and packaging and and and you name it. And now that agentic AI is writing tests on a massive scale, this problem only increases, the problem of of we're going to have a lot of more testing and we'll need to choose which testing to run. So, when we're looking at, you know, the the feedback loop for software factories and for developer velocity, we need to envision, you know, the entire pipeline, the entire feedback loop. So, accelerating the compiler pass is not enough. And to unlock this this real velocity, you need a three-part strategy. First, you need to have visibility, you need to map the critical path, the dependencies, the sequential tasks, the frequency of

execution, what executes when I change what, and you need visualization in order to do that, you need to be able to query, you know, your builds and to query your pipeline very intuitively in order to have that. You cannot just analyze log files in order to achieve real real visibility. Second is distribution. And distribution means can mean it's it's distribution and parallelization in a sense because it means that I want to be able to harvest and in use all my cores. If I have a 128 core CI server, I want to use all these 128 cores. But with Incredibuild, for example, you can distribute tasks across machines in the network or in the cloud. And you can offload these compilation tasks. You can reach hundreds, thousands of tasks running in parallel. So, you want to have that as well. Sometimes you want to parallel and distribute tests, entire test suites. So, this is another way to do it. To do that. And the last is is very important and that's the way that the world is heading today. It's the smart and hopefully generic shared cache because it's always much faster and much more economic to reuse validated build artifact across your teams instead of rerunning them again. So, it's always faster and cheaper not to run anything not to run something than to run it fast. Especially if you're working in the cloud, fetching things from the cache

really reduce the amount of costs [snorts] of your cloud consumption. And where we when we're speaking about development at scale, this becomes a huge cost efficiency, but you know, I think the most is this the time save. And shared cache mean that, you know, if a CI change something, developers can use it, other CI nodes can use it. But if and and generic means that it's not only for one purpose. It can It can actually use a caching system for any deterministic task, which mean that any input, as long as it didn't change, will lead to the same output. This is what generic cash mean. It can be part of this caching mechanism. That's how Incredibuild was able to achieve this dozens of tools that are part of the shared cash preconfigured capability of Incredibuild. Okay, so now let's look at the newest multiplier multiplier driving enterprises code volume, which is the generative AI.

According to JetBrains survey, 78% of Rust developers actively use AI coding assistance. I was actually surprised by that number. I thought and I think today it's it's higher. I I think that it should be and very soon will be 100%. And at the same time programs like the Opera tractor, which is very famous, you can look at it, are using LLMs paired with formal program analysis to automatically translate massive legacy C code bases into safe Rust code. And they do that with high precision, high velocity. Really, really works nice. I really recommend you read about it. So, AI is eliminating the human friction that was historically slowing down all this Rust adoption, you know, with with training people with Rust, with the time it takes people to write code, the cost, the efficiency of migration. So, that's a huge AI agentic AI is a huge opportunity for migrating all these legacy C C++ code and others into Rust.

Uh and and in order to tap that, uh we need to kind of tackle one of the operational challenges, uh which is the downstream operational challenge. Because AI ability is to create code very quickly, much quicker than that we've seen before. But the paradox is that, you know, AI generates thousands of lines of code, very complex Rust code, very fast, generic macros, all of that in seconds. And it expands also rust the crate graphs dramatically. And the effect is that rust seeds still has to do all the work of board checking, type checking, do a bunch of other operations on every line of AI-generated code. So, AI multiplies code volume, which multiplies compiler and testing workload. Not to mention that as AI also writes the test, each project accumulates a large amount of various test execution. So, this is the effect of AI. And the result is that local CPUs become maxed out, and the background IDE, you probably maybe you felt it already, tasks they slow down, CI build queues become congested, and AI speeds up typing, but if the pipelines that deliver the velocity are not addressed to meet the AI area, we will not be able to fully tap into

the velocity potential of that the projective AI brings. And this is especially meaningful as part of the transition to fully autonomous software factories. And I can tell you that CTOs and enterprise teams across the world are now investing a lot of efforts to open the operational bottlenecks that the generative coding brings to the surface. Just recently, I had a conversation with one of the world's leading automotive companies, uh and and they shared that uh they expect their group's builds to increase by 500% during 2026 alone. So, they don't even have full visibility into 2027. And that required them to completely rethink their pipeline velocity and architecture in order to meet this amazing demand. So, that's something, you know, when when we're looking in the future, that's something we need to take into consideration. So, to complete this picture, Orhun and Neil will show you live demos of how this looks like in practice inside our RustRover IDE.

And thank you very much, and Orhun continue. Owen, over to you. I'll stop sharing in a second. >> Thank you so much, Dory. It was very insightful. I definitely have some questions about some parts, but I will leave that to the end. So, amazing. So, now I will show you guys RustRover and how Incredibuild works in practice. So, let me share my screen real quickly here. So, first thing, I have the Incredibuild website, the dashboard, and I have already built some projects here, and you can see some stats. Before I jump to RustRover, I just wanted to show all the things that we have talked about already and how it looks on this side. And if you go to our editor, so this is this is RustRover. There is a Rust project open, and it has, as you can see, many dependencies, and each dependency has some features.

The biggest one here is Bevy. If you have ever built any game with Rust or did any like graphical operation, you know that Bevy takes some time to build and I specifically picked this project to demonstrate how Incredibuild actually actually accelerates the build process. In RustRover, you can basically do many operations. There some integrated AI chat database, and like it understands your code, it can generate code insight from it, and it's basically the IDE for Rust developers. And when you combine that with something like Incredibuild, you get quite nice speed. And how do you do that if you're asking? So, there are some plugin There is a plugin system in RustRover. If you go to plugins from settings, >> [gasps] >> you will see that I have Incredibuild installed. This will be available in the marketplace. I'm right now using the alpha version here, and it has some operations such as build, rebuild, stop build, and so on. And there's also one configuration option that you can use to tweak the build job. So, if you go to Incredibuild [snorts] um

yeah, it's it's on right now. And let's go to settings, Incredibuild, and tools, you can see that the parallel jobs is 300. So, you can configure some options from settings. So, let's build this project. So, when I build this normally, I would go ahead go ahead and just, you know, click here. It will build as usual. This is just the RustRover running cargo build. But instead, what we're going to do is we are going to run Incredibuild build. So, here in this menu, you can just click here, and then it will connect to your Incredibuild um back end. It will just build it for you. So, right now, it already used the cached dependency, so it didn't really take much time. But let's try rebuild. Um let's see if it's going to do anything.

Cargo clean, and now it's building as usual. So, basically, this plugin is a layer on top of RustRover that integrates quite well with the rest of the IDE. I can just run or build things as usual. You just need to uh build using the plugin. And there you can see we have some logs. And if you go to the dashboard again, uh here. You can see that you already have some builds executed. This is just the previous one, I think. And also, it's possible to tweak some things over here as well. So yeah. That's the Incredibuild plugin. Let's see if it's actually going to build in this uh next minute or so. I don't think it will. It's There's so many things in here.

But um yeah. I think I will just uh pass this pass the mic to Ner to explain what's going on in the background, and then we can maybe come back here and see uh some other extra stuff. But, yeah. For now, I think uh this is what I wanted to show. Incredibuild plus plus Rover. >> The dashboard. >> That's great, Ahon. Thanks Thanks a lot for sharing this. And let me now share a few examples on how this looks like. So, what I have here is um the uh Incredibuild admin console, the same one that Ahon shared uh previously. Uh it has many capabilities around observability and gaining full access and visibility to all the builds that you have uh in in your uh environment. So, it could be from uh CI uh machines, from the development, and so forth.

I won't do a full demo of this because this is not really the purpose of uh this webinar. So, what I did here, I ran a few builds of Chevy, uh same open source for for gaming and and visuals that I won't mention earlier and I started with relatively weak machine with only four cores, a Windows machine and this basically the first build we see here and it took on that machine almost 9 minutes. What's really nice is that even when there is no acceleration by Incredibuild, you get a lot of additional information on the build that can really help you to improve how the system is built and how the build is structured. So I have here different information about the machine that was running. I see that there was only one machine that one with the four cores, how many tasks ran, what's the overall time. There was no build cache that was used. I'll show you in a second what that means.

And also everything ran locally. But what's really really powerful is the full build view and here I can see the full build graph of all the tasks that ran. And as Dory mentioned earlier, what I can immediately see here is that even though this is local machine with only four cores, the potential for acceleration using distribution is is quite limited because there are several very very long tasks here that are dependent on each other. But nevertheless, I can still see the different tasks here, how much time they take, what are the dependencies. So there is already here some very valuable information about potential improvements I can do in my code even without Incredibuild. What I did next, I added distribution. So on top of my four cores machine, I added a stronger machine that could be used for distribution. So this is the this run.

What I can see that there was two machines now that was that were part of this build. There is a the original one, the local one with the four cores, and I took advantage of another one with 12 cores. This could be This should not be limited to one additional helper. We We have customers who have dozens, even hundreds of helpers, and we have a mechanism called coordinator that makes sure that works are being distributed uh per priority and need and so forth. So, if I go back to the full build view, what I can see now that indeed the tasks were distributed, a lot of the work was distributed to the stronger machine. However, the benefit was not really, really meaningful because I still had these very long tasks that had to run uh sequentially in the end of the build. And indeed the the build ended up uh a little bit faster.

So, it was 8 minutes instead of almost nine, but not something that is very uh meaningful. So, generally speaking, usually builds not look like that, and we do see very strong acceleration using distribution, um but not always said. Again, even from here, I could take it back to my uh code and try to build it better, remove some dependencies, maybe break some of the tasks and so forth. But, we don't really need to do it because one of the strongest capabilities that we provide as part of Incredibuild is the shared caching. And here, what I can show is a build that had a warm cache, so this is after the cache was populated, and here we do see a dramatic acceleration. So, the whole build took only 30 seconds instead of 9 minutes, so over 90% acceleration. If we look at the full build view, I can see that this is because very few tasks were were actually executed, only what had to be uh ran ran, and this is where we do see the most significant improvement in terms of acceleration. And what we can also see is that the cache hits were very high, 98%. Not always that high, of course. This is where I did a full rebuild, but I didn't make

any code changes, just to illustrate how the caching work. If we want to look at an example that is a little bit more realistic, I had another build after I did make some code changes, and also uh simulated some changes that were done by my teammates. And this is probably the biggest advantage and change compared to other means like incremental build. So, usually when you have incremental build, also the build will only do the tasks that relevant to the code changes that you made. However, when you want to merge your changes, you will need to get the latest version from your repo that includes many other changes in the codes that were not done by you. And now when you need to rebuild, then you need to also build all the other code changes, and that usually, as we saw in the survey, can take a lot of time, especially if those changes have a lot of dependencies and so forth. So, what we see here that even though I did some code changes and there were other code changes, the build took still only 44 seconds.

And if I go, I can see here that the cache hits were a little bit lower, but still very fast. And I can see that there are additional tasks, but again, those are only those that actually had to uh be ran uh due to the code changes. And this is where we see the biggest advantage of the system, the fact that the cache is not only local, it's shared between everyone, and each teammate, each contributor benefits from the cache that was populated either by the CI or the other team members, and you do not need to run again uh tasks uh that are not needed. I will finish with another very, very cool feature that we have, which is when you're not in the happy path. So, everything I show is built that were successfully, they run very fast. Awesome. But, we all know this is not only always the case. And sometimes uh builds are failing.

And two things around that. First, even for failed builds, the fact that it runs faster means that you get the feedback quicker, and you can resolve it quicker. So, the acceleration actually comes with a lot of benefit even for failed builds. But, on top of that, many times there are different changes that happened that you don't always know exactly what happened and what caused the uh build to fail. And because we have a lot of very unique information about the build graph, all the tasks that ran, all the dependencies, what changed, we can give very useful um analysis on what caused the failure to build. And then, of course, we have an AI agent that you can use and analyze and give you the root cause. So, in this case, this was because I ran out of memory, and we have a full recommendation on how to fix it, where you can actually copy it into your AI coding agent and fix it. There are some evidences that explain exactly how we came with that conclusion, and even some alternative fixes on what you can do in order to fix it.

We are now also working to be able to show difference between different builds. So, if you have the build and it ran and succeeded uh early, and now we have failed build, you can mark them too as to compare it, and based on the data that we have, we'll be able to show you what changed between the builds and what may have caused uh this uh failure. So, I'll end up with that. I hope it was interesting and uh valuable. And back to you, home. >> Amazing. Thanks for sharing that. So, now we will go to the Q&A um session. If you guys have any questions from the chat, please send it to us. In the meantime, I have one question to you in there because you mentioned there is the like the the builds went from basically from 9 minutes to 30 seconds and it basically used the cache.

I'm just wondering how is it different from incremental builds because I think Rust already supports that. >> Yeah. Yeah, thanks. Thanks for that. So, uh basically, when you have an incre incremental build, uh the built-in build looks at the code that you change and builds only that. And that's very useful when you're on your uh work on changes. But, as I mentioned, usually what happened is that at some point you would like to merge your code and before doing that, you would want to build with the latest version of the repo. Usually, especially with AI and so much code that is being written, the version that you started is very different from the version that you're actually going to to work on. And that's the point where the build may take very, very long and usually you can may get some failures and so forth. The benefit of the shared cache is the fact that all the artifacts that are now based on the new code are in the shared cache because before merging that code into the repo, the other developers had to run it. Maybe it was also part of CI. And now, when you build, the mechanism that we have, first look if it's in the local cache, if not check for the shared cache, and then you would really end up need to change to to rebuild only very small

fraction of of your code. >> So, does that also >> Mhm. >> It's also important to know two other aspects. You know, many times, you know, when we're working you we are working with feature branches, you want to branch out, and when you branch out, many times you want to do a full rebuild and maintain multiple branches, you know, or merge them back, etc. This is one aspect. And there's also the aspect of of the non-roster over. And this is where you run your builds in CI. And in CI many times, you want to have, you know, uh ephemeral environments, you know, ephemeral machine running with nothing on them, and they will always rebuild from scratch. So, within Gradle, this rebuild on the CI machine will act, you know, the way Near showed it, you know, because usually the difference is not that big. And when we are speaking about caching, it's not caching just the final artifact. It's caching the intermediate artifacts that are generated by every task within the build.

So, usually, you know, all most of the things are not changed. And when you're looking at standards like Google, you can see that they're speaking about 93% uh hit rate of the cache across their projects. So, you you're saving a lot of times in CI. And CI is what serves roster over developers as well. So, it's both the developers, branch featuring branching, and and the CI environment. So, it's all of that together. >> Another question related to this, when you said shared cache, does that mean while working in a team, everyone in the team uses the same cache? Is that correct? >> So, you would configure um a clustered uh cache service that all developers has access to. It's clustered as I mentioned, so you can make sure that multiple people can access it and there are no uh, contradictions.

Also, in order to make sure that the cache uh, access is fastest, you you also have the local version of it. So, as I mentioned, you first go to the local, which is the fastest, and then if you don't have the artifact locally, it will go to the shared cache. And you can configure who writes to it and or and who can only read from it, but but yeah, it's single service that everyone benefited. >> So, some customers, for example, one of their strategy is that CI will populate the cache, the shared cache, while developers will only read from the cache. So, CI is periodically building the code, the entire code base, and developers are consuming, you know, most of the things that already changed. So, if you wake up in the morning and you have a lot of code that you need to rebuild because people changed it during the yesterday, usually the CI already ran, you know, this entire build and all the cache is fully populated. So, when you do a full, you know, checkout, uh, you probably will have uh, the build result in 100% uh, cache hit. So, this is this is one of the benefit of shared cache.

I just one more thing about what Nir said. When we're speaking about distribution, with IncrediBuild, it's important to know that when you distribute tasks as part of a Raster over build to another machine, this machine you don't need to have Raster over installed on that machine. You don't need to have your project copied to that machine. This can be a VM pure out of the box with nothing on it. IncrediBuild has a very unique virtualization technologies that kind of it's like containers and runtime that allows you allows us to run this task on the remote machine and virtualize everything it requires in order to run properly. So, nothing need to set up on that remote machine and it can be used by multiple developers. >> So, when you say that multiple developers can use it, do you do anything extra to make sure that the cash is it remains correct when you know, when different compiler versions come into play or different feature flags and so on. Do you do anything extra or just leave it to the rust toolchain to handle that cash?

>> [snorts] >> So, the way that the cash works is that we have fingerprint of everything that contributed to create the artifact. So, this include of course the source code but also the environment variables and the version of the compiler and maybe many other factors. Each Each factor that could potentially impact the final artifact. And based on that fingerprint, we go to the caches and check, do you have the artifact that was made with this fingerprint? If it's there, then you can use it. If not, we know that we need to create the artifact from scratch. >> Mhm. Yeah. >> [clears throat] >> That makes sense. Also, Dory, I remember from your presentation you mentioned there is like three runs, local, distributed and cached. When you showed distributed run, you you also mentioned even though this machine has like I don't know, like a lot of cores, this distribution won't make a difference because there is like some some jobs that take long time, so we cannot really parallelize them.

So, I'm wondering when distributions actually useful and like when when caching is actually useful. How do you decide on which optimization technique? >> So, I think that one of the one of the main thing with Incredibuild is that you don't need to choose. Uh you just run your build and Incredibuild will choose for you. So, if if it can bring things from the cache, that will always be faster and also more cost-effective. And it's it's better for sequential tasks because if you have one sequential task, you cannot break it and you cannot use even a local machine with 128 cores or 64 cores. So, caching is always better. But, if there is a cache miss, Incredibuild will take this task or or many tasks and will automatically run them in parallel across the grid on the local machine depending on the strategy. By default, it will run as much of them as possible according to the number of cores you have in your Incredibuild grid. And as Neo mentioned, we have customers that are running hundreds of tasks in parallel across many machines.

And uh the you know, if you have one strong machine, many developers can distribute tasks to this machine as well. So, it's not always only about performance. One of the thing that we see is that sometimes a developer want to keep working on the Raster over ID while he's building his code while the code is built in the background. And if everything is built on the local machine, kind of the machine is getting congested, right? And if Incredibuild will distribute all the tasks assuming everything will need to run and no cache it can be can be achieved, this will also free up resources from the local machine. So, this developer can continue, you know, maybe building something else or continue to work with Raster over and and the builds will will just run in the background in other machines without congesting his laptop. So, that's another benefit. And and this benefit multiplies when you're looking at CI servers because it allows you to increase the throughput and essentially the amount of builds that you can run per CI server both with caching and with distribution. So, if you have one of the main problem is not how long does a build take, is that a lot of builds are accumulating in the build queue of the CI. And if the CI can kind of break it and use cache and distribution, it can accumulate more and run more builds in parallel. So, the entire build time of the builds that are running accumulated will be much faster and and and cost less in the cloud of course with caching.

>> Yep. Yep. That sounds good. One last question before we wrap this up. So, today if developers wants want to try out Incredibuild, uh access the Incredibuild plugin for Rustrover, uh how can they do that? Is there a free trial option? How can they start using it? >> Yeah, absolutely. So, all you need to do is go to our website, incredibuild.com. And when you do that, you have a 30-days free trial. The free trial is not limited, so you can use all of the functions, uh both distribution and caching. Uh you have guidelines there on how to run the helpers, how to install the helper, sorry. So, you can run it on multiple machines, how to um set up the caching. And also on our documentation, you can have link to the plugin.

As you mentioned, currently that's the way. Uh shortly we'll have it on the marketplace of Rustrover. >> That sounds good. And with that, we'll wrap this webinar. Uh thank you, Dori and Nir, for joining us. Thanks everyone for joining us and uh we will see you next time. Check out Rustrover and Incredibuild. Have a good day. >> having us. >> Absolutely. >> Thank you. >> Cheers. >> Mhm.