AI Apps in a Flash: Ship to GPUs Without Docker — Dean Quiñanola, Runpod

Greetings, good day to Greetings, good day to everyone. I'm Dean everyone. I'm Dean everyone. I'm Dean Cignonola from RunPod Flash. Cignonola from RunPod Flash. And I want to tell And I want to tell you how you can you how you can you how you can deploy AI deploy AI applications without Docker. applications without Docker. So, this is how serverless serverless applications are deployed now. Well, applications are deployed now. Well, applications are deployed now. Well, especially AI especially AI applications. You write applications. You write applications. You write code. Create a Dockerfile. code. Create a Dockerfile. code. Create a Dockerfile. You run a bunch of You run a bunch of You run a bunch of commands, Docker build, and so on commands, Docker build, and so on commands, Docker build, and so on . Then . Then . Then you send the image to you send the image to you send the image to Docker Hub. Then Docker Hub. Then Docker Hub. Then execute it. execute it. execute it. Maybe you'll get what Maybe you'll get what you wanted. If not, you you wanted. If not, you you wanted. If not, you have to have to have to change the code again. change the code again. change the code again. Reassemble it. Reassemble it. Reassemble it. Forward. Forward. Forward. Perform. In my Perform. In my Perform. In my experience, sometimes this experience, sometimes this experience, sometimes this cycle takes about 15 cycle takes about 15 cycle takes about 15 minutes, depending on the minutes, depending on the minutes, depending on the changes. So, my name is changes. So, my name is changes. So, my name is Dean Quiñonela. I am a Dean Quiñonela. I am a Dean Quiñonela. I am a full-time engineer at RunPod full-time engineer at RunPod . Now I have a whole . Now I have a whole . Now I have a whole team working team working team working to make to make to make it easier. So why did it easier. So why did it easier. So why did we create Flash? The we create Flash? The we create Flash? The development iteration cycle development iteration cycle development iteration cycle needs updating.

needs updating. needs updating. This is really troublesome. This is really troublesome. Many of you are software or ML software or ML engineers who don't engineers who don't engineers who don't want to deal with want to deal with want to deal with all this DevOps stuff. So all this DevOps stuff. So all this DevOps stuff. So when you work on when you work on when you work on your ideas and your ideas and your ideas and experiments, you experiments, you experiments, you want a result. want a result. want a result. So, Docker is no longer So, Docker is no longer So, Docker is no longer needed. Run needed. Run needed. Run the code locally as if the code locally as if the code locally as if you had a GPU. Test you had a GPU. Test you had a GPU. Test changes instantly and changes instantly and changes instantly and quickly debug quickly debug quickly debug if needed. Here's how if needed. Here's how if needed. Here's how we try to keep we try to keep we try to keep it simple. it simple. it simple. You write code. You write code. You write code. You do it. You do it. You do it. You get results You get results . You rewrite the code, . You rewrite the code, . You rewrite the code, execute it, get execute it, get execute it, get results, right? results, right? results, right? Of course, there's still the Of course, there's still the Of course, there's still the deployment part. I'll deployment part. I'll deployment part. I'll talk about that talk about that talk about that later. It's so later. It's so later. It's so simple that it feels like simple that it feels like simple that it feels like everything is running everything is running everything is running locally. So what locally. So what locally. So what functions do we have? We can functions do we have? We can functions do we have? We can build and build and build and deploy faster. We deploy faster. We deploy faster. We optimize the optimize the optimize the use of use of use of resources. There's resources. There's resources. There's a lot of a lot of a lot of interesting stuff going on under the hood. And this is interesting stuff going on under the hood. And this is interesting stuff going on under the hood. And this is intuitively clear intuitively clear . This is just Python code. . This is just Python code. . This is just Python code. You can run You can run You can run anything in the cloud that runs anything in the cloud that runs anything in the cloud that runs locally on a GPU. What locally on a GPU. What locally on a GPU. What can you do with Flash can you do with Flash can you do with Flash today? We have today? We have today? We have serverless serverless serverless endpoints. This is endpoints. This is endpoints. This is only only only serverless for now, but we serverless for now, but we serverless for now, but we are working on are working on are working on supporting pods.

supporting pods. supporting pods. We have We have We have endpoint queues and endpoint queues and load balancers for them. Here's a really cool Here's a really cool feature I added: feature I added: endpoint communication. Sometimes you endpoint communication. Sometimes you create a lot of create a lot of create a lot of servers, but you still servers, but you still servers, but you still need to know how to need to know how to need to know how to contact contact contact each one, right? With this one each one, right? With this one each one, right? With this one you don't need to you don't need to you don't need to do that because the code just do that because the code just do that because the code just talks to other talks to other talks to other code. You execute code. You execute code. You execute a function and it seems a function and it seems a function and it seems like it's local, like it's local, like it's local, but once you but once you but once you deploy it to deploy it to deploy it to different different different endpoints, they all endpoints, they all endpoints, they all start start start interacting. All this interacting. All this interacting. All this happens "under happens "under happens "under the hood". We also the hood". We also the hood". We also have have have network volume connections. We network volume connections. We network volume connections. We have hot have hot have hot caching on caching on caching on network volumes, so network volumes, so network volumes, so if you need if you need if you need to scale your to scale your to scale your serverless endpoints, serverless endpoints, serverless endpoints, the cache is migrated the cache is migrated the cache is migrated across network volumes. across network volumes. And the most important And the most important part is live serverless. part is live serverless. part is live serverless. You write code and You write code and You write code and test it as test it as test it as if everything is if everything is if everything is happening on your happening on your happening on your local system. And local system. And local system. And you know, it would you know, it would you know, it would just be a toy just be a toy just be a toy project if you project if you project if you couldn't scale couldn't scale couldn't scale it into production, it into production, it into production, right? So you want to right? So you want to right? So you want to deploy it to deploy it to deploy it to production, production, production, serve serve serve inference and inference and inference and all the things you all the things you all the things you build, and we build, and we build, and we make that possible make that possible make that possible through the CLI. So, through the CLI. So, through the CLI. So, let's watch the let's watch the let's watch the demo. Hopefully I can do it with demo. Hopefully I can do it with demo. Hopefully I can do it with one one one hand. Okay. OK.

hand. Okay. OK. My demo My demo shows rapid shows rapid shows rapid iteration and development of iteration and development of iteration and development of the feature. The first the feature. The first the feature. The first function simply function simply function simply calls torch calls torch calls torch to find out what to find out what to find out what GPU you have. So I'll do it GPU you have. So I'll do it GPU you have. So I'll do it live. I live. I live. I will enter flash. dev and this will enter flash. dev and this will enter flash. dev and this will launch the local will launch the local will launch the local Fast API so you can Fast API so you can Fast API so you can actually actually actually experiment with experiment with experiment with it. Now you want to it. Now you want to it. Now you want to try it. try it. try it. Let's try. Let's try. Let's try. Perform. So, it Perform. So, it Perform. So, it prepares this serverless prepares this serverless prepares this serverless endpoint. And if endpoint. And if endpoint. And if we go to the we go to the we go to the RunPod console, refresh RunPod console, refresh RunPod console, refresh it, you'll see that. it, you'll see that. it, you'll see that. It's working now. And It's working now. And It's working now. And while it's while it's while it's loading, loading, let's show the let's show the logs so you can logs so you can logs so you can see everything for yourself. see everything for yourself. Okay, the worker is Okay, the worker is ready. Yes. Oh, ready. Yes. Oh, ready. Yes. Oh, the task is running.

the task is running. Cold start. This is a Cold start. This is a common thing for all common thing for all common thing for all serverless platforms. Oh, there serverless platforms. Oh, there serverless platforms. Oh, there it is. So, it it is. So, it it is. So, it works. We have logs, works. We have logs, works. We have logs, container logs. He container logs. He container logs. He tests all these things, tests all these things, tests all these things, making sure everything is making sure everything is making sure everything is working and the task working and the task working and the task is completed. Now, if is completed. Now, if is completed. Now, if we come back here, we come back here, we come back here, you'll see the you'll see the you'll see the result. It result. It result. It ran on an Nvidia GeForce ran on an Nvidia GeForce ran on an Nvidia GeForce RTX 4090. You can see the amount of RTX 4090. You can see the amount of RTX 4090. You can see the amount of RAM, RAM, RAM, the number of GPUs. And now the the number of GPUs. And now the the number of GPUs. And now the most interesting thing. What if most interesting thing. What if most interesting thing. What if I want to change this I want to change this I want to change this feature? And I just feature? And I just feature? And I just want to test it. want to test it. want to test it. Here is another iteration of Here is another iteration of Here is another iteration of this function. We're this function. We're this function. We're just adding just adding just adding more code. And this code more code. And this code more code. And this code actually just actually just actually just generates text. And I generates text. And I generates text. And I added some added some added some dependencies, Python dependencies, Python dependencies here. Oh, dependencies here. Oh, dependencies here. Oh, actually, let's actually, let's actually, let's do it do it do it without the Hugging Face token first. without the Hugging Face token first. without the Hugging Face token first. Let's see what Let's see what Let's see what happens. You see, happens. You see, happens. You see, everything has been updated. I everything has been updated. I everything has been updated. I will update this. I'm testing will update this. I'm testing will update this. I'm testing again. What prompt do again. What prompt do again. What prompt do we need?

we need? we need? Let's try the cheese Let's try the cheese braid. Is he braid. Is he braid. Is he real? real? Let's launch. You Let's launch. You see it here again. see it here again. Container. Good. Loading. You Loading. You can see that the logs are can see that the logs are can see that the logs are simpler here, because they don't simpler here, because they don't simpler here, because they don't have to have to have to overload you with a overload you with a overload you with a bunch of things. But it bunch of things. But it bunch of things. But it sets dependencies on the fly sets dependencies on the fly sets dependencies on the fly . And even . And even . And even issued a warning issued a warning issued a warning about connecting to the about connecting to the about connecting to the Hugging Face server without a Hugging Face server without a Hugging Face server without a token. And that should be token. And that should be token. And that should be all. OK. Now it's all. OK. Now it's all. OK. Now it's ready, let's look at the ready, let's look at the ready, let's look at the result. I asked, result. I asked, result. I asked, "Is the cheese pigtail "Is the cheese pigtail "Is the cheese pigtail real?" Sorry, real?" Sorry, real?" Sorry, but I can't provide but I can't provide but I can't provide an answer. That's right, an answer. That's right, an answer. That's right, interesting. Okay, interesting. Okay, interesting. Okay, let's change that. let's change that. I will connect my HF token . It's just a . It's just a . It's just a code change, right? Now we're code change, right? Now we're code change, right? Now we're launching again. Let's change launching again. Let's change launching again. Let's change the query. Let's see.

the query. Let's see. Who is Batman? Let's do this. Done. Well Let's do this. Done. Well , let's see. , let's see. , let's see. How does it work? Steeply. I How does it work? Steeply. I How does it work? Steeply. I do it with one hand do it with one hand . Okay, let's finish. . Okay, let's finish. . Okay, let's finish. Now we see the Now we see the Now we see the result. Oh, there it is result. Oh, there it is . Batman is a . Batman is a . Batman is a fictional character. fictional character. fictional character. Well, we already knew that. Let Well, we already knew that. Let 's try something 's try something more intensive than more intensive than more intensive than text generation. text generation. Now let's generate the image . OK. Updated. . OK. Updated. . OK. Updated. Let's update this Let's update this Let's update this localhost. Now you localhost. Now you localhost. Now you see " see " see " image generation". And image generation". And image generation". And we test this with a we test this with a we test this with a query. Let's create query. Let's create query. Let's create an image of a cheesy an image of a cheesy Batman. Good. This is Batman. Good. This is how creative how creative how creative AI can be.

AI can be. Let's see what's Let's see what's going on here. going on here. Execution of logs. Oh, Execution of logs. Oh, look. look. look. Deployment is ongoing. Deployment is ongoing. Good. Oh, I have Good. Oh, I have plenty of time. This is not it. plenty of time. This is not it. plenty of time. This is not it. Good. I had to Good. I had to Good. I had to stop doing this because the stop doing this because the stop doing this because the old version was working old version was working old version was working . Now new. I did . Now new. I did . Now new. I did n't want to wait for that n't want to wait for that n't want to wait for that deployment. I just deployment. I just deployment. I just want to see a want to see a want to see a cheesy Batman. So cheesy Batman. So , it responded in , it responded in , it responded in base64 format. Good. base64 format. Good. base64 format. Good. Let me get Let me get Let me get this straight before this straight before this straight before I show it. So, I show it. So, you can display this you can display this through a browser. Good.

through a browser. Good. through a browser. Good. Let's see what a Let's see what a Let's see what a cheesy cheesy cheesy Batman looks like. This is not a cheese Batman looks like. This is not a cheese braid. But that braid. But that braid. But that proves it works, proves it works, proves it works, right? He did it. right? He did it. right? He did it. What if you want to What if you want to What if you want to expand it? Here, expand it? Here, expand it? Here, look. When I look. When I look. When I cancel, it cancel, it cancel, it actually deletes that actually deletes that actually deletes that endpoint. So endpoint. So endpoint. So you don't need it to you don't need it to you don't need it to just just just hang there. You see, now hang there. You see, now hang there. You see, now she's gone. Now, what she's gone. Now, what she's gone. Now, what if we want to if we want to if we want to expand on this? So, expand on this? So, expand on this? So, let's expand on this. Rapid let's expand on this. Rapid let's expand on this. Rapid deployment. This wraps deployment. This wraps deployment. This wraps everything up. Collects this. everything up. Collects this. everything up. Collects this. Sends to our Sends to our Sends to our servers. Again, servers. Again, servers. Again, no Docker. You're not no Docker. You're not no Docker. You're not dealing with Docker here. This is dealing with Docker here. This is dealing with Docker here. This is instant packaging of instant packaging of instant packaging of your application, your application, your application, sending it to our sending it to our sending it to our servers, and servers, and servers, and maintaining it. And maintaining it. And maintaining it. And now the most interesting thing. now the most interesting thing. now the most interesting thing. Once he was there, I Once he was there, I Once he was there, I prepared this prepared this prepared this prompt. But I will prompt. But I will prompt. But I will need to need to need to change this ID later change this ID later . Because it will . Because it will . Because it will matter. Good. So, matter. Good. So, matter. Good. So, the archive is created, the flash the archive is created, the flash the archive is created, the flash is assembled. Good.

is assembled. Good. is assembled. Good. Sending, Sending, Sending, uploaded. uploaded. uploaded. Deployment. Now Deployment. Now Deployment. Now he works. Good. he works. Good. he works. Good. Let's look at the ID here. Let's look at the ID here. OK. Now let's begin . So, my prompt is a . So, my prompt is a cute baby cute baby cute baby sea otter with a bow tie sea otter with a bow tie sea otter with a bow tie . . Generate this Generate this image. Oh yes. image. Oh yes. image. Oh yes. I want to show you where it is I want to show you where it is I want to show you where it is . This is completely . This is completely . This is completely ready to ready to ready to use, right? use, right? use, right? That is, your customers That is, your customers That is, your customers can use can use can use your application that your application that your application that uses this uses this uses this backend. It is no longer backend. It is no longer backend. It is no longer on my on my on my localhost. It is localhost. It is localhost. It is hosted on a hosted on a hosted on a hosting service. Okay, let hosting service. Okay, let hosting service. Okay, let 's look at the workers. 's look at the workers. log Okay, when it's log Okay, when it's done, it should done, it should done, it should output the result output the result output the result here. That's all. Okay, here. That's all. Okay, here. That's all. Okay, let me wrap let me wrap let me wrap this up so I can this up so I can this up so I can open it. A. It's open it. A. It's open it. A. It's madness to do it with madness to do it with madness to do it with one hand. Okay, one hand. Okay, one hand. Okay, let's upload a photo of a let's upload a photo of a let's upload a photo of a sea otter with a bow tie sea otter with a bow tie sea otter with a bow tie .

. There is no bow tie. But that's not the There is no bow tie. But that's not the main thing. We're fine, you main thing. We're fine, you main thing. We're fine, you understand. This was a understand. This was a understand. This was a demo version. demo version. demo version. We had a hackathon yesterday and I We had a hackathon yesterday and I We had a hackathon yesterday and I just wanted to just wanted to just wanted to introduce this, I'm introduce this, I'm introduce this, I'm really proud of this really proud of this really proud of this guy, uh, Won guy, uh, Won guy, uh, Won Kau. He created a Kau. He created a Kau. He created a wrapper around wrapper around wrapper around our flash that our flash that our flash that allows you to load a allows you to load a allows you to load a spreadsheet with all spreadsheet with all spreadsheet with all your your your experiments, experiments, experiments, specifications, specifications, specifications, models and models and models and run them instantly. It run them instantly. It run them instantly. It just scales just scales , creates all these , creates all these , creates all these endpoints, runs endpoints, runs endpoints, runs training, and then training, and then training, and then reports back to you. And he reports back to you. And he reports back to you. And he did it in 3-4 hours did it in 3-4 hours . And he had no . And he had no . And he had no idea how idea how idea how to work with RunPod Flash. to work with RunPod Flash. to work with RunPod Flash. He asked me if this was He asked me if this was He asked me if this was even possible. I even possible. I even possible. I replied, "Let's replied, "Let's replied, "Let's check." And he was check." And he was check." And he was very surprised when he very surprised when he very surprised when he won yesterday, won yesterday, won yesterday, because his project because his project because his project probably wasn't even the probably wasn't even the probably wasn't even the best. He best. He best. He consistently lost consistently lost consistently lost at other hackathons, at other hackathons, at other hackathons, so I'm really proud of so I'm really proud of so I'm really proud of this guy. How this guy. How this guy. How to get started with RunPod to get started with RunPod to get started with RunPod Flash? So, you Flash? So, you Flash? So, you register an register an register an account on RunPod, account on RunPod, account on RunPod, scan the QR code, scan the QR code, scan the QR code, get $10 in get $10 in get $10 in credit, and all you credit, and all you credit, and all you have to do is have to do is have to do is install it, install it, install it, write the code, log in write the code, log in write the code, log in to add the API key, to add the API key, to add the API key, initialize; and initialize; and initialize; and if you want, it if you want, it if you want, it will create the framework of the will create the framework of the will create the framework of the application. Flash dev, application. Flash dev, application. Flash dev, local development, whatever local development, whatever local development, whatever you have, or flash you have, or flash deployment to deployment to deployment to production. And we have production. And we have production. And we have all these URLs.

all these URLs. all these URLs. The best advice is The best advice is to visit the Run to visit the Run to visit the Run Pod flash examples on GitHub, because there are Pod flash examples on GitHub, because there are Pod flash examples on GitHub, because there are all these cases and all these cases and all these cases and examples that you examples that you examples that you can reference. can reference. Or, if you Or, if you like AI coding, like AI coding, like AI coding, just run cloud just run cloud just run cloud code on this code on this code on this repository and it repository and it repository and it will learn everything. We will learn everything. We will learn everything. We also have skills if also have skills if also have skills if you are interested. you are interested. you are interested. This concludes my This concludes my This concludes my presentation presentation presentation .