NVIDIA Put a DGX Spark in a Laptop… Almost

This is the first RTX Spark laptop to This is the first RTX Spark laptop to show up at my door. show up at my door. show up at my door. Wow. The MSI Prestige N16 Flip AI Plus. Wow. The MSI Prestige N16 Flip AI Plus. Wow. The MSI Prestige N16 Flip AI Plus. Quick unboxing. You know the DJX Spark, Quick unboxing. You know the DJX Spark, Quick unboxing. You know the DJX Spark, right? That little box that's selling right? That little box that's selling right? That little box that's selling like hot cakes while the price keeps like hot cakes while the price keeps like hot cakes while the price keeps going up. Well, the RTX Spark is going up. Well, the RTX Spark is going up. Well, the RTX Spark is supposed to be like it almost. Same 128 supposed to be like it almost. Same 128 supposed to be like it almost. Same 128 GB of memory except it's in a laptop. GB of memory except it's in a laptop. GB of memory except it's in a laptop. And MSI says this will run 120 billion And MSI says this will run 120 billion And MSI says this will run 120 billion parameter models. I'm going to check parameter models. I'm going to check parameter models. I'm going to check that. Of course, they also say that it that. Of course, they also say that it that. Of course, they also say that it holds peak performance with no thermal holds peak performance with no thermal holds peak performance with no thermal throttling. I'm going to check that, throttling. I'm going to check that, throttling. I'm going to check that, too. Also, too. Also, too. Also, this is kind of cool. You can bend this this is kind of cool. You can bend this this is kind of cool. You can bend this thing all the way back. That's something thing all the way back. That's something thing all the way back. That's something you don't see every day. They had to you don't see every day. They had to you don't see every day. They had to have stickers, didn't they? I'll be have stickers, didn't they? I'll be have stickers, didn't they? I'll be testing RTX Sparks over the next few testing RTX Sparks over the next few testing RTX Sparks over the next few weeks. But if everything is true for weeks. But if everything is true for weeks. But if everything is true for building software and running local AI, building software and running local AI, building software and running local AI, this form factor might be the best of this form factor might be the best of this form factor might be the best of all worlds. Depends what you're doing. The whole thing feels extremely solid, The whole thing feels extremely solid, even though it is a flippity flip even though it is a flippity flip even though it is a flippity flip laptop. I care more about the keyboard.

laptop. I care more about the keyboard. laptop. I care more about the keyboard. Very clicky. We still have a co-pilot Very clicky. We still have a co-pilot Very clicky. We still have a co-pilot key for some reason. No numpad, which I key for some reason. No numpad, which I key for some reason. No numpad, which I appreciate. I know I'm going to get appreciate. I know I'm going to get appreciate. I know I'm going to get flack for this in the comments. Some flack for this in the comments. Some flack for this in the comments. Some people love their numpad. I am the people love their numpad. I am the people love their numpad. I am the opposite. Now, I'll be honest, coming opposite. Now, I'll be honest, coming opposite. Now, I'll be honest, coming from my MacBook, I expect certain things from my MacBook, I expect certain things from my MacBook, I expect certain things from the touchpad. It's a nice large from the touchpad. It's a nice large from the touchpad. It's a nice large touchpad, which responds pretty well, touchpad, which responds pretty well, touchpad, which responds pretty well, but I do wish it was a haptic touchpad but I do wish it was a haptic touchpad but I do wish it was a haptic touchpad instead of the older diving board instead of the older diving board instead of the older diving board design. There's a webcam up here with a design. There's a webcam up here with a design. There's a webcam up here with a webcam shutter. It's got a fingerprint webcam shutter. It's got a fingerprint webcam shutter. It's got a fingerprint reader and front firing speakers. Two reader and front firing speakers. Two reader and front firing speakers. Two USB 4 ports, HDMI, a headphone jack, SD USB 4 ports, HDMI, a headphone jack, SD USB 4 ports, HDMI, a headphone jack, SD card reader, and two USBA ports. I still card reader, and two USBA ports. I still card reader, and two USBA ports. I still appreciate the USBA ports included appreciate the USBA ports included appreciate the USBA ports included because now I don't have to use a dongle because now I don't have to use a dongle because now I don't have to use a dongle for my mouse. I got to say, I expected for my mouse. I got to say, I expected for my mouse. I got to say, I expected the charger to be a lot bigger. This the charger to be a lot bigger. This the charger to be a lot bigger. This thing is uh pretty tiny and it puts out thing is uh pretty tiny and it puts out thing is uh pretty tiny and it puts out 140 watts. For reference, this is the 140 watts. For reference, this is the 140 watts. For reference, this is the DJX Spark adapter. I don't mind carrying DJX Spark adapter. I don't mind carrying DJX Spark adapter. I don't mind carrying one of these around at all. Now, what one of these around at all. Now, what one of these around at all. Now, what I'm really interested in is whether I'm really interested in is whether I'm really interested in is whether having this thing plugged in is going to having this thing plugged in is going to having this thing plugged in is going to affect performance. affect performance. affect performance. 16in 4K tandem OLED, 120 Hz, and it's a 16in 4K tandem OLED, 120 Hz, and it's a 16in 4K tandem OLED, 120 Hz, and it's a touchcreen. It's almost a shame to touch touchcreen. It's almost a shame to touch touchcreen. It's almost a shame to touch it. Get my fingerprints all over it. If it. Get my fingerprints all over it. If it. Get my fingerprints all over it. If there's fingerprints on the B-roll, you there's fingerprints on the B-roll, you there's fingerprints on the B-roll, you can yell at me in the comments. What's can yell at me in the comments. What's can yell at me in the comments. What's this? Oh, this? Oh, this? Oh, okay. I gotta admit something. I don't okay. I gotta admit something. I don't okay. I gotta admit something. I don't use flip laptops, but now it makes a lot use flip laptops, but now it makes a lot use flip laptops, but now it makes a lot of sense. Am I going to use that?

of sense. Am I going to use that? of sense. Am I going to use that? Probably not. But let's dig into the Probably not. But let's dig into the Probably not. But let's dig into the performance. I got to check out the task performance. I got to check out the task performance. I got to check out the task manager because that's where the magic manager because that's where the magic manager because that's where the magic lives. CPU, that's the new N1X chip. So, lives. CPU, that's the new N1X chip. So, lives. CPU, that's the new N1X chip. So, a little bit different than the DJX a little bit different than the DJX a little bit different than the DJX Spark. And there is our 20 core CPU. In Spark. And there is our 20 core CPU. In Spark. And there is our 20 core CPU. In memory, we've got 128 GB of LPDDR5. This memory, we've got 128 GB of LPDDR5. This memory, we've got 128 GB of LPDDR5. This is an ARMbased CPU. So, this is running is an ARMbased CPU. So, this is running is an ARMbased CPU. So, this is running Windows on ARM. And why am I excited Windows on ARM. And why am I excited Windows on ARM. And why am I excited about saying that? Because Microsoft has about saying that? Because Microsoft has about saying that? Because Microsoft has been investing in Windows and ARM for been investing in Windows and ARM for been investing in Windows and ARM for the last couple years. I've been the last couple years. I've been the last couple years. I've been covering it for a couple years, ever covering it for a couple years, ever covering it for a couple years, ever since Visual Studio and ARM came out and since Visual Studio and ARM came out and since Visual Studio and ARM came out and Qualcomm's chips. And the first thing I Qualcomm's chips. And the first thing I Qualcomm's chips. And the first thing I wanted to know is whether my stuff even wanted to know is whether my stuff even wanted to know is whether my stuff even runs on it. And by the way, from past runs on it. And by the way, from past runs on it. And by the way, from past experiences running things on Windows experiences running things on Windows experiences running things on Windows and ARM, yeah, my stuff usually does. and ARM, yeah, my stuff usually does. and ARM, yeah, my stuff usually does. But I wanted to prove it to you. First, But I wanted to prove it to you. First, But I wanted to prove it to you. First, you can check out works on WOA. A you can check out works on WOA. A you can check out works on WOA. A tremendous efforts been put forward to tremendous efforts been put forward to tremendous efforts been put forward to get support on Windows on ARM. With a get support on Windows on ARM. With a get support on Windows on ARM. With a few edge cases, it's pretty much a good few edge cases, it's pretty much a good few edge cases, it's pretty much a good to go for most developer tools. Even to go for most developer tools. Even to go for most developer tools. Even Adobe products are native now. Mostly Adobe products are native now. Mostly Adobe products are native now. Mostly Premiere Pro is still emulated. Premiere Pro is still emulated. Premiere Pro is still emulated. Photoshop. I use Photoshop all the time, Photoshop. I use Photoshop all the time, Photoshop. I use Photoshop all the time, in case you couldn't tell by my in case you couldn't tell by my in case you couldn't tell by my thumbnails. Creative Cloud ARM 64. Adobe thumbnails. Creative Cloud ARM 64. Adobe thumbnails. Creative Cloud ARM 64. Adobe ARM 64. Photoshop ARM 64. Never mind ARM 64. Photoshop ARM 64. Never mind ARM 64. Photoshop ARM 64. Never mind that it's using 3 GB of memory, but you that it's using 3 GB of memory, but you that it's using 3 GB of memory, but you know, it's running natively. There's know, it's running natively. There's know, it's running natively. There's over 250 different services and workers over 250 different services and workers over 250 different services and workers running right now on this machine, and running right now on this machine, and running right now on this machine, and only four are still x64. Visual Studio, only four are still x64. Visual Studio, only four are still x64. Visual Studio, VS Code, Zed, Python, Node, all that VS Code, Zed, Python, Node, all that VS Code, Zed, Python, Node, all that stuff that I run is native. It runs, but stuff that I run is native. It runs, but stuff that I run is native. It runs, but is it fast?

is it fast? is it fast? I usually like to start out with testing I usually like to start out with testing I usually like to start out with testing a single core. Boom. Speedometer 3.1. a single core. Boom. Speedometer 3.1. a single core. Boom. Speedometer 3.1. This tests a bunch of JavaScript This tests a bunch of JavaScript This tests a bunch of JavaScript applications in the browser. So, it applications in the browser. So, it applications in the browser. So, it tells you how snappy the system is going tells you how snappy the system is going tells you how snappy the system is going to be when it comes to JavaScript to be when it comes to JavaScript to be when it comes to JavaScript performance. 35.2, not too bad. I ran it performance. 35.2, not too bad. I ran it performance. 35.2, not too bad. I ran it four times and it landed between 35 and four times and it landed between 35 and four times and it landed between 35 and 37 every time, but one core is only part 37 every time, but one core is only part 37 every time, but one core is only part of the story. What I really care about of the story. What I really care about of the story. What I really care about is using all 20 of them because that's is using all 20 of them because that's is using all 20 of them because that's what your builds and your compilations what your builds and your compilations what your builds and your compilations are going to be like. So, I use measure are going to be like. So, I use measure are going to be like. So, I use measure command and PowerShell to measure how command and PowerShell to measure how command and PowerShell to measure how fast this is going to be. And this is fast this is going to be. And this is fast this is going to be. And this is the mantle bra test. Boom. There it is the mantle bra test. Boom. There it is the mantle bra test. Boom. There it is running. And the reason I like this test running. And the reason I like this test running. And the reason I like this test is because of this. Take a look at that is because of this. Take a look at that is because of this. Take a look at that CPU usage, huh? Look at that. Every CPU usage, huh? Look at that. Every CPU usage, huh? Look at that. Every single core is being used. This lets me single core is being used. This lets me single core is being used. This lets me test a few things at the same time. test a few things at the same time. test a few things at the same time. First of all, the performance of this First of all, the performance of this First of all, the performance of this machine compared to other ones. machine compared to other ones. machine compared to other ones. How loud it gets and the temperatures, How loud it gets and the temperatures, How loud it gets and the temperatures, which I'll get into shortly. which I'll get into shortly. which I'll get into shortly. And as far as loudness, it's I can hear And as far as loudness, it's I can hear And as far as loudness, it's I can hear the fan. And I got 22 seconds for this the fan. And I got 22 seconds for this the fan. And I got 22 seconds for this particular run. I actually ran it a particular run. I actually ran it a particular run. I actually ran it a bunch of times. On average, I got 22 bunch of times. On average, I got 22 bunch of times. On average, I got 22 seconds. But I was not satisfied with seconds. But I was not satisfied with seconds. But I was not satisfied with running it on just Windows. Since a lot running it on just Windows. Since a lot running it on just Windows. Since a lot of the workflows that I do on my machine of the workflows that I do on my machine of the workflows that I do on my machine have to do with Python and Node, I like have to do with Python and Node, I like have to do with Python and Node, I like to run those inside WSL or Windows to run those inside WSL or Windows to run those inside WSL or Windows subsystem for Linux. And some things run subsystem for Linux. And some things run subsystem for Linux. And some things run faster on Windows and some things run faster on Windows and some things run faster on Windows and some things run faster on WSL. This is an example of one faster on WSL. This is an example of one faster on WSL. This is an example of one of those things. I'm looking over here of those things. I'm looking over here of those things. I'm looking over here because I have charts over here. I ran because I have charts over here. I ran because I have charts over here. I ran the same mandler algorithm which is the same mandler algorithm which is the same mandler algorithm which is developed by a famous mathematician

developed by a famous mathematician developed by a famous mathematician Benerbro. He discovered the whole Benerbro. He discovered the whole Benerbro. He discovered the whole fractals thing. A little aside for you fractals thing. A little aside for you fractals thing. A little aside for you right here. So WSL did that one much right here. So WSL did that one much right here. So WSL did that one much faster I'd say. Yeah, that's pretty faster I'd say. Yeah, that's pretty faster I'd say. Yeah, that's pretty good. WSL is basically Linux running good. WSL is basically Linux running good. WSL is basically Linux running inside Windows in a VM. I can open up inside Windows in a VM. I can open up inside Windows in a VM. I can open up terminal, start Ubuntu right there. And terminal, start Ubuntu right there. And terminal, start Ubuntu right there. And I have a whole separate file system here I have a whole separate file system here I have a whole separate file system here that has all my Linux tools. if you're that has all my Linux tools. if you're that has all my Linux tools. if you're not familiar with it. If you are, then I not familiar with it. If you are, then I not familiar with it. If you are, then I apologize for repeating it, okay? But apologize for repeating it, okay? But apologize for repeating it, okay? But some people are not. Now, I was also some people are not. Now, I was also some people are not. Now, I was also curious about how this does on the DJX curious about how this does on the DJX curious about how this does on the DJX Spark because I have a feeling that a Spark because I have a feeling that a Spark because I have a feeling that a lot of comparisons are going to be drawn lot of comparisons are going to be drawn lot of comparisons are going to be drawn uh between the Spark DJX Spark and the uh between the Spark DJX Spark and the uh between the Spark DJX Spark and the RTX Spark laptops. So, I ran the same RTX Spark laptops. So, I ran the same RTX Spark laptops. So, I ran the same thing on DJX Spark, 16 seconds. thing on DJX Spark, 16 seconds. thing on DJX Spark, 16 seconds. Basically, this machine running virtual Basically, this machine running virtual Basically, this machine running virtual machine inside Windows is about the same machine inside Windows is about the same machine inside Windows is about the same speed as running dedicated Linux on the speed as running dedicated Linux on the speed as running dedicated Linux on the Spark for this test. And my first Spark for this test. And my first Spark for this test. And my first thought was, well, was Windows not using thought was, well, was Windows not using thought was, well, was Windows not using all the cores? But it did. It used all all the cores? But it did. It used all all the cores? But it did. It used all the 20 cores in Linux and in Windows and the 20 cores in Linux and in Windows and the 20 cores in Linux and in Windows and in WSL. So I took Python out of the in WSL. So I took Python out of the in WSL. So I took Python out of the equation and ran the same math on one equation and ran the same math on one equation and ran the same math on one core in JavaScript. 1.24 seconds on core in JavaScript. 1.24 seconds on core in JavaScript. 1.24 seconds on Windows and 1.24 seconds on Linux. So Windows and 1.24 seconds on Linux. So Windows and 1.24 seconds on Linux. So the chip is every bit as fast under the chip is every bit as fast under the chip is every bit as fast under Windows. The slowdown here comes from Windows. The slowdown here comes from Windows. The slowdown here comes from Python. And this comes down to using the Python. And this comes down to using the Python. And this comes down to using the right tool for the right job, which is right tool for the right job, which is right tool for the right job, which is what I always say on the channel. If what I always say on the channel. If what I always say on the channel. If you're doing heavy Python work on this you're doing heavy Python work on this you're doing heavy Python work on this thing, run it in WSL. Now, I also wanted thing, run it in WSL. Now, I also wanted thing, run it in WSL. Now, I also wanted to know how this thing will perform when to know how this thing will perform when to know how this thing will perform when unplugged. And guess what? 22 seconds on unplugged. And guess what? 22 seconds on unplugged. And guess what? 22 seconds on Windows. This has kind of been a Windows. This has kind of been a Windows. This has kind of been a painoint before on Windowsbased PCs.

painoint before on Windowsbased PCs. painoint before on Windowsbased PCs. Let's take a look at what happened. I Let's take a look at what happened. I Let's take a look at what happened. I already know what happened, but you get already know what happened, but you get already know what happened, but you get to find out as I show you this chart. to find out as I show you this chart. to find out as I show you this chart. 23.2 seconds. Yes, it is just a little 23.2 seconds. Yes, it is just a little 23.2 seconds. Yes, it is just a little bit slower. And I had it set to best bit slower. And I had it set to best bit slower. And I had it set to best performance whether it was plugged in or performance whether it was plugged in or performance whether it was plugged in or not. Boom. MSI also provides MSI center not. Boom. MSI also provides MSI center not. Boom. MSI also provides MSI center which allows you to switch between which allows you to switch between which allows you to switch between performance boost or balanced or eco performance boost or balanced or eco performance boost or balanced or eco mode. Eco mode is silent by the way. So mode. Eco mode is silent by the way. So mode. Eco mode is silent by the way. So yeah, you give up some of that yeah, you give up some of that yeah, you give up some of that performance, but it's going to be a performance, but it's going to be a performance, but it's going to be a quiet machine. But before I load a quiet machine. But before I load a quiet machine. But before I load a single LLM on it, I want one more single LLM on it, I want one more single LLM on it, I want one more number, and that's memory bandwidth. number, and that's memory bandwidth. number, and that's memory bandwidth. See, there's two parts to generating See, there's two parts to generating See, there's two parts to generating things like LLMs do. One is reading the things like LLMs do. One is reading the things like LLMs do. One is reading the prompt, processing the prompt, and then prompt, processing the prompt, and then prompt, processing the prompt, and then there's generating tokens. Generating there's generating tokens. Generating there's generating tokens. Generating tokens mostly is about how fast you can tokens mostly is about how fast you can tokens mostly is about how fast you can move memory around. But I created a move memory around. But I created a move memory around. But I created a little script that would test the GPU little script that would test the GPU little script that would test the GPU memory where it copies and reads multi- memory where it copies and reads multi- memory where it copies and reads multi- gigabyte blocks on the GPU. And wow, gigabyte blocks on the GPU. And wow, gigabyte blocks on the GPU. And wow, copying 8 GB, the laptop does 143 GB a copying 8 GB, the laptop does 143 GB a copying 8 GB, the laptop does 143 GB a second. The DJX Spark 113 reading it's second. The DJX Spark 113 reading it's second. The DJX Spark 113 reading it's 309 versus 234. So yeah, that should 309 versus 234. So yeah, that should 309 versus 234. So yeah, that should lead to faster token generation. Now the lead to faster token generation. Now the lead to faster token generation. Now the GPU part itself, that's the part that GPU part itself, that's the part that GPU part itself, that's the part that does the computation or the heavy does the computation or the heavy does the computation or the heavy lifting, the parallel processing, the lifting, the parallel processing, the lifting, the parallel processing, the prompt processing. Okay, when am I going prompt processing. Okay, when am I going prompt processing. Okay, when am I going to stop? Uh, I think I mentioned enough.

to stop? Uh, I think I mentioned enough. to stop? Uh, I think I mentioned enough. The GPU itself is basically level. It's The GPU itself is basically level. It's The GPU itself is basically level. It's 101 on the laptop versus 97 on the 101 on the laptop versus 97 on the 101 on the laptop versus 97 on the Spark. This is measuring that number Spark. This is measuring that number Spark. This is measuring that number crunching I was talking about matrix crunching I was talking about matrix crunching I was talking about matrix multiplication. And this is not multiplication. And this is not multiplication. And this is not theoretical. This is actually what I theoretical. This is actually what I theoretical. This is actually what I measured. So, we should be about even on measured. So, we should be about even on measured. So, we should be about even on prompt processing and faster, a little prompt processing and faster, a little prompt processing and faster, a little bit faster on token generation. Let's bit faster on token generation. Let's bit faster on token generation. Let's see. see. see. Now, sure, you can hook up a local AI Now, sure, you can hook up a local AI Now, sure, you can hook up a local AI model to your code editor like I have model to your code editor like I have model to your code editor like I have here. In fact, this is probably going to here. In fact, this is probably going to here. In fact, this is probably going to be one of the number one use cases of be one of the number one use cases of be one of the number one use cases of this kind of laptop. 120 billion this kind of laptop. 120 billion this kind of laptop. 120 billion parameter model. Like they said, no parameter model. Like they said, no parameter model. Like they said, no problem. It's working right here inside problem. It's working right here inside problem. It's working right here inside Zed's agent. You can use VS Code or Zed's agent. You can use VS Code or Zed's agent. You can use VS Code or whatever you want to do. It's going to whatever you want to do. It's going to whatever you want to do. It's going to run locally and not leave the machine. run locally and not leave the machine. run locally and not leave the machine. But I wanted to push it a little bit But I wanted to push it a little bit But I wanted to push it a little bit more to see if we can go past the more to see if we can go past the more to see if we can go past the marketing speak. And I'm going to tell marketing speak. And I'm going to tell marketing speak. And I'm going to tell you, marketing was a little bit you, marketing was a little bit you, marketing was a little bit conservative here cuz we can do more conservative here cuz we can do more conservative here cuz we can do more with this machine. So the next question with this machine. So the next question with this machine. So the next question is, what kind of models can we run and is, what kind of models can we run and is, what kind of models can we run and how fast? Now, one cool thing about this how fast? Now, one cool thing about this how fast? Now, one cool thing about this setup is that you don't need to decide setup is that you don't need to decide setup is that you don't need to decide between Linux or Windows because you can between Linux or Windows because you can between Linux or Windows because you can use both right on this machine through use both right on this machine through use both right on this machine through WSL, which I mentioned earlier. Here's WSL, which I mentioned earlier. Here's WSL, which I mentioned earlier. Here's Nvidia Smi showing the full capacity Nvidia Smi showing the full capacity Nvidia Smi showing the full capacity available to the GPU through each one of available to the GPU through each one of available to the GPU through each one of these processes, 98 GB. Now, what does these processes, 98 GB. Now, what does these processes, 98 GB. Now, what does running models look like on this thing?

running models look like on this thing? running models look like on this thing? I decided to start out with a small I decided to start out with a small I decided to start out with a small dense model. Now, the 64 GB model will dense model. Now, the 64 GB model will dense model. Now, the 64 GB model will be able to run Quen 3.827B, be able to run Quen 3.827B, be able to run Quen 3.827B, which is a dense model. This model is which is a dense model. This model is which is a dense model. This model is about 22 GB in size. It'll easily fit in about 22 GB in size. It'll easily fit in about 22 GB in size. It'll easily fit in that 64 GB MSI Prestige 13.4 tokens per that 64 GB MSI Prestige 13.4 tokens per that 64 GB MSI Prestige 13.4 tokens per second and that's because it's such a second and that's because it's such a second and that's because it's such a dense model. It means every parameter dense model. It means every parameter dense model. It means every parameter gets used on every token. The next two I gets used on every token. The next two I gets used on every token. The next two I have on here are mixture of experts. Now have on here are mixture of experts. Now have on here are mixture of experts. Now MSI and a bunch of other people always MSI and a bunch of other people always MSI and a bunch of other people always refer to this GPT OSS120B because well refer to this GPT OSS120B because well refer to this GPT OSS120B because well it's an older model as we know. However, it's an older model as we know. However, it's an older model as we know. However, it's very widely supported and that's it's very widely supported and that's it's very widely supported and that's why everybody uses that number. So I why everybody uses that number. So I why everybody uses that number. So I included it here as well. It also runs included it here as well. It also runs included it here as well. It also runs everywhere and it's fast because it's everywhere and it's fast because it's everywhere and it's fast because it's mixture of experts. This model loads in mixture of experts. This model loads in mixture of experts. This model loads in 28 seconds and is using 61 of the 96 GB. 28 seconds and is using 61 of the 96 GB. 28 seconds and is using 61 of the 96 GB. 120 billion parameters on my lap 120 billion parameters on my lap 120 billion parameters on my lap potentially on an airplane or at a potentially on an airplane or at a potentially on an airplane or at a conference or anywhere. I mean, you conference or anywhere. I mean, you conference or anywhere. I mean, you could take this anywhere, but then could take this anywhere, but then could take this anywhere, but then you're going to not be able to plug it you're going to not be able to plug it you're going to not be able to plug it in. You know the deal. Laptop. Laptop.

in. You know the deal. Laptop. Laptop. in. You know the deal. Laptop. Laptop. 65 tokens per second for GPTOSs 12B. 65 tokens per second for GPTOSs 12B. 65 tokens per second for GPTOSs 12B. Remember that memory bandwidth number I Remember that memory bandwidth number I Remember that memory bandwidth number I was talking about? Well, this is the was talking about? Well, this is the was talking about? Well, this is the same model on the same version of Llama same model on the same version of Llama same model on the same version of Llama CPP and I got DGX Spark numbers, too. I CPP and I got DGX Spark numbers, too. I CPP and I got DGX Spark numbers, too. I bet you're surprised. This is a big bet you're surprised. This is a big bet you're surprised. This is a big difference actually. Spark here does difference actually. Spark here does difference actually. Spark here does 50.6 where the laptop does 65.4 and WSL 50.6 where the laptop does 65.4 and WSL 50.6 where the laptop does 65.4 and WSL about the same 65.1. Now, of course, about the same 65.1. Now, of course, about the same 65.1. Now, of course, that's an empty context. As we use it that's an empty context. As we use it that's an empty context. As we use it more and more, the context fills up. So, more and more, the context fills up. So, more and more, the context fills up. So, you can see the numbers on WSL suffer a you can see the numbers on WSL suffer a you can see the numbers on WSL suffer a little bit more, but overall all the little bit more, but overall all the little bit more, but overall all the numbers go down, right? But the Windows numbers go down, right? But the Windows numbers go down, right? But the Windows numbers stays up and it keeps winning. numbers stays up and it keeps winning. numbers stays up and it keeps winning. Oh, yeah. And I promised I would show Oh, yeah. And I promised I would show Oh, yeah. And I promised I would show you what it looks like unplugged. Well, you what it looks like unplugged. Well, you what it looks like unplugged. Well, I ran that, too. And unplugged it did I ran that, too. And unplugged it did I ran that, too. And unplugged it did 65.9. So, it doesn't slow down when it's 65.9. So, it doesn't slow down when it's 65.9. So, it doesn't slow down when it's on battery for the GPU work. I guess I on battery for the GPU work. I guess I on battery for the GPU work. I guess I should pay attention to the chart. It should pay attention to the chart. It should pay attention to the chart. It slows down a little bit on WSL, but not slows down a little bit on WSL, but not slows down a little bit on WSL, but not on Windows. Anyway, it's close enough. on Windows. Anyway, it's close enough. on Windows. Anyway, it's close enough. It's not like we take a huge nose dive. It's not like we take a huge nose dive. It's not like we take a huge nose dive. Now, since we have CUDA here, we have Now, since we have CUDA here, we have Now, since we have CUDA here, we have the ability to run VLM. VLM does not the ability to run VLM. VLM does not the ability to run VLM. VLM does not build in Windows, but it builds in build in Windows, but it builds in build in Windows, but it builds in Linux, so it builds in WSL. So, you can Linux, so it builds in WSL. So, you can Linux, so it builds in WSL. So, you can run VLM now on a laptop with all that run VLM now on a laptop with all that run VLM now on a laptop with all that memory available. Never before has this memory available. Never before has this memory available. Never before has this been a thing that we could do. Why do been a thing that we could do. Why do been a thing that we could do. Why do you care about VLM, Alex? Well, a lot of you care about VLM, Alex? Well, a lot of you care about VLM, Alex? Well, a lot of professional data centers use VLM to professional data centers use VLM to professional data centers use VLM to serve models, and it has very different serve models, and it has very different serve models, and it has very different characteristics than something like characteristics than something like characteristics than something like Llama CPP. By the way, if you're using Llama CPP. By the way, if you're using Llama CPP. By the way, if you're using tools like Olama or tools like LM

tools like Olama or tools like LM tools like Olama or tools like LM Studio, they will use Llama CPP under Studio, they will use Llama CPP under Studio, they will use Llama CPP under the hood. So, what does it look like to the hood. So, what does it look like to the hood. So, what does it look like to use VLM? Well, first, for prompt use VLM? Well, first, for prompt use VLM? Well, first, for prompt processing, Windows and WSL in Llama CPP processing, Windows and WSL in Llama CPP processing, Windows and WSL in Llama CPP do about 1,800 tokens per second. and do about 1,800 tokens per second. and do about 1,800 tokens per second. and VLM. Boom. VLM. Boom. VLM. Boom. 4,681 4,681 4,681 tokens per second. And yeah, I measured tokens per second. And yeah, I measured tokens per second. And yeah, I measured it a few times, don't worry. 2 and 1/2 it a few times, don't worry. 2 and 1/2 it a few times, don't worry. 2 and 1/2 times as fast. But it's not all rosy for times as fast. But it's not all rosy for times as fast. But it's not all rosy for VLM either. Generating for one user, VLM either. Generating for one user, VLM either. Generating for one user, like for example, if you were the only like for example, if you were the only like for example, if you were the only person using this thing right now and person using this thing right now and person using this thing right now and you were on a plane and you were you were on a plane and you were you were on a plane and you were chatting, that's probably the worst case chatting, that's probably the worst case chatting, that's probably the worst case scenario for using VLM. Check it out. scenario for using VLM. Check it out. scenario for using VLM. Check it out. VLM only does 35 tokens per second on VLM only does 35 tokens per second on VLM only does 35 tokens per second on GPTOSS120B GPTOSS120B GPTOSS120B whereas Llama CPP on both Windows and whereas Llama CPP on both Windows and whereas Llama CPP on both Windows and WSL does 65. And that's because VLM is WSL does 65. And that's because VLM is WSL does 65. And that's because VLM is built for a crowd. Think of it like a built for a crowd. Think of it like a built for a crowd. Think of it like a bus and Llama CPP is like a car. With bus and Llama CPP is like a car. With bus and Llama CPP is like a car. With one passenger, the car gets there first. one passenger, the car gets there first. one passenger, the car gets there first. VLM has to make a few stops. Maybe VLM has to make a few stops. Maybe VLM has to make a few stops. Maybe that's not a great analogy, but think that's not a great analogy, but think that's not a great analogy, but think about VLM uh with agents where you have about VLM uh with agents where you have about VLM uh with agents where you have multiple agents using it at the same multiple agents using it at the same multiple agents using it at the same time. then then we're going to do well.

time. then then we're going to do well. time. then then we're going to do well. Let me show you. I'll talk about the bus Let me show you. I'll talk about the bus Let me show you. I'll talk about the bus a little bit more. So, excuse me. Uh the a little bit more. So, excuse me. Uh the a little bit more. So, excuse me. Uh the bus barely slows down when you fill it. bus barely slows down when you fill it. bus barely slows down when you fill it. With eight users at once, we're getting With eight users at once, we're getting With eight users at once, we're getting 127.5 tokens a second. Notice that 127.5 tokens a second. Notice that 127.5 tokens a second. Notice that improvement the more users we have. And improvement the more users we have. And improvement the more users we have. And users could be people using your laptop users could be people using your laptop users could be people using your laptop or it could be agents, multiple agents or it could be agents, multiple agents or it could be agents, multiple agents at the same time. Now, MSI said 120 at the same time. Now, MSI said 120 at the same time. Now, MSI said 120 billion parameters, but I got a new one billion parameters, but I got a new one billion parameters, but I got a new one that's 125. And this one is pretty that's 125. And this one is pretty that's 125. And this one is pretty relevant because it's the next relevant because it's the next relevant because it's the next generation of architecture. Quen 3.8 generation of architecture. Quen 3.8 generation of architecture. Quen 3.8 Flash Next. It loads in 36 seconds. It Flash Next. It loads in 36 seconds. It Flash Next. It loads in 36 seconds. It takes 81 out of the 96 GB. So, we're takes 81 out of the 96 GB. So, we're takes 81 out of the 96 GB. So, we're really, really pushing it there to the really, really pushing it there to the really, really pushing it there to the limit. And it runs at 33.7 tokens per limit. And it runs at 33.7 tokens per limit. And it runs at 33.7 tokens per second on a thin laptop. How crazy is second on a thin laptop. How crazy is second on a thin laptop. How crazy is that? Now, aside from the usual Windows that? Now, aside from the usual Windows that? Now, aside from the usual Windows strangeness, strangeness, strangeness, >> file name too long. Come on. Is it too >> file name too long. Come on. Is it too >> file name too long. Come on. Is it too long here? long here? long here? >> Nope. That's hilarious. >> Nope. That's hilarious. >> Nope. That's hilarious. >> Aside from that, Windows actually >> Aside from that, Windows actually >> Aside from that, Windows actually performed really well as you've seen.

performed really well as you've seen. performed really well as you've seen. And of course, you need to have long And of course, you need to have long And of course, you need to have long file paths enabled in Windows. Why can't file paths enabled in Windows. Why can't file paths enabled in Windows. Why can't that be the default? Now, I've been that be the default? Now, I've been that be the default? Now, I've been saying that you can use 96 GB of saying that you can use 96 GB of saying that you can use 96 GB of dedicated GPU memory. That's actually dedicated GPU memory. That's actually dedicated GPU memory. That's actually adjustable inside the Nvidia app right adjustable inside the Nvidia app right adjustable inside the Nvidia app right here. You can dedicate whatever you here. You can dedicate whatever you here. You can dedicate whatever you want. I just happen to like the maximum. want. I just happen to like the maximum. want. I just happen to like the maximum. Hey, you know, AI. But you could have a Hey, you know, AI. But you could have a Hey, you know, AI. But you could have a different split here if, for example, different split here if, for example, different split here if, for example, you're not using large models. If you're you're not using large models. If you're you're not using large models. If you're using smaller models, very possible, and using smaller models, very possible, and using smaller models, very possible, and you want to use more of the memory you want to use more of the memory you want to use more of the memory towards your system tasks, like if towards your system tasks, like if towards your system tasks, like if you're running heavy agents, for you're running heavy agents, for you're running heavy agents, for example. example. example. Now, what happens when you run this Now, what happens when you run this Now, what happens when you run this thing heavily for 20 minutes? Well, thing heavily for 20 minutes? Well, thing heavily for 20 minutes? Well, sometimes the computers heat up so much sometimes the computers heat up so much sometimes the computers heat up so much that they have to slow themselves down, that they have to slow themselves down, that they have to slow themselves down, slow the processor down, and that's slow the processor down, and that's slow the processor down, and that's called throttling. So, I left this thing called throttling. So, I left this thing called throttling. So, I left this thing on for 20 minutes with the model on for 20 minutes with the model on for 20 minutes with the model generating on the GPU and all 20 CPU generating on the GPU and all 20 CPU generating on the GPU and all 20 CPU cores busy at the same time. And in the cores busy at the same time. And in the cores busy at the same time. And in the first 5 minutes, the model was doing first 5 minutes, the model was doing first 5 minutes, the model was doing about 50 tokens per second. In the last about 50 tokens per second. In the last about 50 tokens per second. In the last 5 minutes, 52 tokens per second. So, 5 minutes, 52 tokens per second. So, 5 minutes, 52 tokens per second. So, it's flat sort of. I mean, the chart is it's flat sort of. I mean, the chart is it's flat sort of. I mean, the chart is not flat, but you get the idea. 20 not flat, but you get the idea. 20 not flat, but you get the idea. 20 minutes and it held. Same story on CPU.

minutes and it held. Same story on CPU. minutes and it held. Same story on CPU. Matter broad pass took 26.3 seconds at Matter broad pass took 26.3 seconds at Matter broad pass took 26.3 seconds at the start and 26.3 seconds at the end. the start and 26.3 seconds at the end. the start and 26.3 seconds at the end. Now, of course, the 50 is lower than the Now, of course, the 50 is lower than the Now, of course, the 50 is lower than the 65 from before. And that's because both 65 from before. And that's because both 65 from before. And that's because both things are running at once. By itself, things are running at once. By itself, things are running at once. By itself, the model does 68 in the test. And with the model does 68 in the test. And with the model does 68 in the test. And with all 20 cores busy right next to it, now all 20 cores busy right next to it, now all 20 cores busy right next to it, now it's 50. But it's like that from the it's 50. But it's like that from the it's 50. But it's like that from the beginning, not because it's slowed down. beginning, not because it's slowed down. beginning, not because it's slowed down. It was not throttling. Now, it does run It was not throttling. Now, it does run It was not throttling. Now, it does run hot. The chip averaged about 93° C and hot. The chip averaged about 93° C and hot. The chip averaged about 93° C and peaked about 98.6. The Predator might peaked about 98.6. The Predator might peaked about 98.6. The Predator might even think it's a human. Some Some of even think it's a human. Some Some of even think it's a human. Some Some of you will get that reference. Basically, you will get that reference. Basically, you will get that reference. Basically, nothing reported thermal throttling. So, nothing reported thermal throttling. So, nothing reported thermal throttling. So, best of all worlds. Well, if you want best of all worlds. Well, if you want best of all worlds. Well, if you want one machine that runs your dev tools and one machine that runs your dev tools and one machine that runs your dev tools and 120 billion parameter models or 125, 120 billion parameter models or 125, 120 billion parameter models or 125, basically anywhere you want it, uh, on basically anywhere you want it, uh, on basically anywhere you want it, uh, on an airplane, in the park, then yeah, an airplane, in the park, then yeah, an airplane, in the park, then yeah, this will do it. If you want something this will do it. If you want something this will do it. If you want something running agents around the clock, if it's running agents around the clock, if it's running agents around the clock, if it's plugged in somewhere in a server room or plugged in somewhere in a server room or plugged in somewhere in a server room or on your desk, there's always a DJX Spark on your desk, there's always a DJX Spark on your desk, there's always a DJX Spark for that. MSI has a couple of different for that. MSI has a couple of different for that. MSI has a couple of different versions. This is the 128 gig, as I versions. This is the 128 gig, as I versions. This is the 128 gig, as I said. There's also 64 gig and a 32 gig.

said. There's also 64 gig and a 32 gig. said. There's also 64 gig and a 32 gig. And if you're near our micro center, And if you're near our micro center, And if you're near our micro center, they'll have both of them so you can go they'll have both of them so you can go they'll have both of them so you can go and flip one in person. Check out the and flip one in person. Check out the and flip one in person. Check out the links down below. And I've got more RTX links down below. And I've got more RTX links down below. And I've got more RTX Spark machines coming in, so subscribe Spark machines coming in, so subscribe Spark machines coming in, so subscribe and don't miss those videos. Thanks for and don't miss those videos. Thanks for and don't miss those videos. Thanks for watching and I'll see you next time.