Robots Will Be Trained on Video Games - Pim de Witte, General Intuition

[music] >> Hello. Um I want you to uh zoom out for a second and think about if somebody were to tell you today that we would take all the information on the internet and then we would train a model on it and we would use that in order to solve scientific problems that were not on the internet. I think a lot of people would tell you that that's crazy. Um and uh what we're doing um has a lot of parallels to that story. Um and so if you think about how intelligence works imagine for a second that you're in the movie Frozen and you're Olaf, the the snowman. Um if if Olaf was an LLM and it there there's a scene which is one of my favorites which is where he rolls down the mountain and there's a little stone that come that pops up and Olaf spreads out temporarily becomes a disassembled snowman and then becomes an assembled snowman and Olaf knows how to do that because Olaf can observe as he is acting. Olaf can, unlike LLMs which auto aggressively predict tokens, um constantly part of that behavior is that it's constantly observing as it is acting. And so um this is the big difference between you

know, when you see LLMs try to solve robotics problems or when you look at native architectures like what we're working on, um it's very very easy to think of LLMs in the demos that you see on Twitter as the thing that will solve everything. There's fundamental limitations that we see uh where purpose-built models are much better fit. Now today I'm going to talk a little bit about what the other side of that coin looks like. Um and so for us it all started with Medal. We built the largest video game recorder in the world um and we think of that as the internet. So, people use Medal to record themselves playing tens of thousands of different video games. And if you think about it, uh, as a human, we can teleoperate a robot with the exact same brain that we played that video game. And in fact, um, in many cases, we also can use the same controller. So, what this means is that the action space of video games, often because of game controllers, and the action space of teleoperations or the real world, in many cases, is the same.

Um, and so, every single robot built below, um, certain degrees of freedom ships with the game controller. A majority of the supply chain is already worked this way. And so, we thought, what if you just take all of the action data that we get through the video recorder, um, which associates the inputs and the actions with the actual frames. We label the actions, so we under- understand semantically meaning behind every action, and we use it to give it to a vision-based model. And so, what you see here on the left is just an intuitive way of how the model works. So, you give the model a set of keys to choose from, essentially a computer use model. It It It gives frames as the input, so you can give it a text prompt as well. And you send it through API, and you say, "Somebody waves at you, wave back." In the input space, model understands that you have a greet action available, it will output the action associated with that. Um, one way of intu- of intuitively understanding how this works. So, um, I've obviously simplified here because it's predicting next token, not the next word, but bear with me here.

Um, LLMs took large-scale internet data, then trained it to predict the next text token, and now use it for writing software. Um, and these are all workloads that are really, really well fit for futures that are not simultaneously unfolding as you are predicting right? Code is turn-based. You can say something, the model will do something, you can say something again, the model will do something me again. And um you can use it for lots of valuable work, but at the end of the day, because it's turn-based, um it's very, very difficult for it to generalize to environments, especially that include um space and time. And so, our models, the way that they work is they're continuously receiving frames or receiving batches of essentially um information from the camera of the robot or it could be a a screen. And it predicts everything as as if it were just a video game. So, you can think of the way that Anthropic focused on just nailing coding. So, you know, they were never frontier in just base text generation. They always focused on being the best at generating code.

And then what the debate that they made is that more people would start bringing more problems into coding space. If you think of general intuition, our bet is that you can as humans, we are generally pixel-to-action converters plus sensors, which in many cases you can convert into pixels as well. And you can output actions at the same time. And those actions are in this case either keyboard and mouse or a game controller. So, the output tokens of the model are um essentially equivalent to that of uh enhanced computer use model with more more more actions on the output space to internalize it. So, you send frames and you get actions. This works again because every single thing that you see here can control um all of these different embodiments, right? So, um as humans, we do not know how to predict motor torque. We do not We do not know what value to send to a machine. Um there's this beautiful device that has been made for many years, uh which actually does that work for us, so we can control them. Most robotics companies today are trying to solve the problem at the end-to-end level, trying to predict exact poses or exact um values uh in the motor. I instead of just looking and picking up the thing that every robotics company has built to generalize our very limited abilities to have physics understanding and predictive text values

and solve for it. And so, our approach is let's just use that. Um and so Metal at the moment sees about a billion videos uploaded per year. This year on we're on track for 3 billion. Um which lead to over a trillion and a half action tokens per month. So, just for comparison, think about the bet. The total amount of data that Frontier Labs used today to train robotics models, we collect in at the moment half a day. Um and so I the the type of bet and again, think of the fact that humans can use the exact same controllers and their eyes to play video games that they can teleoperate robots. It is the same thing. We just bet on scaling pixel space the same way that Anthropic scaled coding space and betting that people will just as we scale data compute into pixel space that people will bring more problems into pixel space. >> [snorts] >> And so, what happens? Um All of the sudden, you get an incredibly general agent that can go and solve problems on computers, in the real world, um and in addition, similarly to how Anthropic focuses on doing coding environments, we focus on world models, which most of you know as heard of Mira release world models, to then generate additional environments for these models to learn in. One way of internalizing a big

advantage of this approach against LLMs is if you're an if you're an LLM company and you have to bet on the ability to generate environments for your models to learn, you have you have two problems. One, you actually need to spend the compute to simulate environments and the more realistic you try to simulate, the more expensive that gets. Um and so, imagine you have you want to learn a new thing. You want to learn that new thing in the exact input space of the human, right? So, you if if we are learning from pixels, we're we're learning by looking around. If I'm trying to communicate something, I want my robot partner to observe that exact same thing. And so, um what what I don't want is for data to go through an LLM, for a digital twin to have to be created, for the model to have to get deployed, for there to be a bunch of verifier checks whether or not things are equivalent, and then for the robot to to learn much later, right? And so, one way of thinking about it is when you can learn in the native modality of the human, you can all of the sudden start solving a lot more problems a lot faster. You what you'll see when these models come out is they're significantly faster than any LLM um at doing these things. We because games are so action-dense. So, what you see, right, is um one thing that's interesting is uh you saw, for instance, the model predicting actions at the same time.

Um we're able to uh take that action density from gamers and convert them into really complex robotics workloads. So, for example, if you are operating a drone and you need to make lots of little adjustments, you really really really want to start with games. You don't want to start with uh teleoperations because the human will just take a lot less actions. It has a lot less information. So, you want a model that can predict really high APM. Um And so, the APM of the data set puts a lot of these problems all of the sudden in reach so that it can do a lot of these actions simultaneously. And as I mentioned, the model doesn't just understand the key it's predicting. It also understands the semantic meaning associated with that key it's predicting, which is really important for things like um interpretability. I So, I we will be releasing the models this year. Um I we are very happy. I think the correct word would be that we're we're stunned by the fact of what these models can do. Um it is I Hi.

This is a live stream, so I unfortunately couldn't share the >> [laughter] >> release videos without breaking the surprise, but if you're in New York, we are also in New York. And so, we have started doing a few demos of the models. So, if you can know somebody on the team, reach out and we can start showing you some of the some of the capabilities live. Um, but we have essentially one model, which can play video games to highest level and control over 10 types of robots using the exact same primitives. So, pixels in, actions out completely real time. Um, and we'll be rolling out early access to these models over next few months. We already have some customers that are on them, but it's still private, so you're not hearing too much about it. But, if you scan this QR code, you'll be able to get the models because we go by who signs up on the wait list at which times. So, um, this is a great this is a great moment to get in touch. As I mentioned, we are in New York. I think what you'll find is similar to how uh, Jeff and a lot of these decision-making models have kind of opened people's eyes on what you can do when you have a new architectural primitive. People are ready to kind of do away with the unpredictability of an LLM and put something in that is more reliable and faster and you just understand how it's going to react as opposed to having to work around it.

If that resonates, these models are exactly what you're looking for. And all of the sudden, um, you can start exploring the environments that you generate with LLMs. All of the sudden, one of the use cases that I'm really really excited about is well, these models, they can take actions as if a human was on the interface. And so, if you're doing UX work and you're trying to understand, is this an intuitive interface? Well, this is a very interesting way of solving the problem. So, all these problems that LLMs can't solve because they can't actually explore like a human, you can use these problems to start you can start using these models for. So, um, the way that they're going to be released is we still expect you to use an orchestrator model. So, you may use something like Gemini or Astra in order to do long-term planning and and everything, and you'll be able to plug these into your workloads. Um and we're actually the the models are so general, it it it is genuinely going to, I think, feel like the early LLM days when you have a new toy and you just want to figure out what it does and all the things you can use it for.

I my reaction was like that when when I when I first started using the checkpoints, and I think I think you'll have the same reaction. Um and I think a lot of you will, just like with LLMs, if you're a bit early and you start understanding these primitives, I think you'll understand uh that you can even build lots of businesses on top of this, right? You know, think about for instance, um a a sci- a science lab, right? If you all of the sudden, if you can reliably orchestrate robots in space and time, and they can take text instructions and do things, all of the sudden, um I this is a robotics problem, right? Robotics is interesting because one of the big problems with solving robotics is making more robots. And so, there are all these businesses that because it's so bottlenecked by intelligence, are going to emerge just from solving this one simple problem, which is pixels to actions. And I think if you're early on getting your hands on on these, it will have a similar effect on getting your your hands on early LLMs.

Um I am most excited also uh one of the things that that we're working on, which is interesting, is if you have any friends or family who uh are um in some way um uh need accessibility help for using their computer, because these models can use computers like a human, all of the sudden, for instance, if you're if you're uh if you only can use one arm, um all of the sudden, the model can allow you to play video games because you can you can use you can just control the mouse, and the model is predicting the actions for your keyboard. And because the model also understands which actions you're predicting, because that's the input, right? It's it's observing as it's acting, it will actually just not predict the things that you're doing and it will just predict the keys. And so, it has all these downstream effects of things that you can build on top that I'm an engineer. If you're interested in building in this space uh this is going to be the most interesting toy that comes out this year. I I I'm fairly certain. Um and I also think a lot of the initial usage will sit in simulation. So, if if a lot of you are into generating environments with Astra or if Gemini is perfect, now all of a sudden you can have something that can explore it and you don't have to do it yourself. Um and I it's perfect timing because of all the environment generation capabilities going going into these models. Um Okay. So, really quickly before I wrap up, um I want to talk about the team.

Um so, we are based in New York City. I the metal team and the general intuition team is here in New York City. So, if you're here, you should reach out to us and I I we can both show you the models, uh especially if you want to join the team. I think we'll we'll prioritize even higher. Uh >> [laughter] >> but, um I we really are looking for um people who A want to understand these models as much as we do. So, people if you're really really good at generating novel environments, uh typically if you have some sort of research uh background, but I don't have a research background, so that's okay. If you you know, if you um if you come from an engineering background and you're very very happy to learn, this is this is fine, too. Um and ideally what we want is somebody who um uh has experienced shipping uh end-to-end uh AI uh products, but it doesn't have to be on the research side. So, we just want to kind of hire people that understand the entire thing. Um Let's see. I We have three teams, so one is world modeling, the other is the policy, so the agent, and the other one is working on um essentially the the data generation and the environment generation. So, these are the main teams. If you're an engineer, the best one to join is probably the environment generation team. Um and then you sort of grow into uh getting access later on. Um yeah, we I I'm very very excited for November. I really hope that I you'll be on the lookout. Um as I mentioned, it will be either late November early December is the current planning. Um

one more thing, uh we are going to organize three hackathons. Um so the first is going to be in New York City, it will be mid-November. Um and uh we were we will send out um if you sign up for the partner waitlist, we'll send out some emails so that everybody can get access. Uh we hope to see everybody at the hackathons. I think it will be really the first time to get your hands on them the way that we're playing with them internally now. And uh there will be one in New York City and one in London. So if you're in either of the cities or want to travel to either of those cities, uh those would be a good uh times to meet the team. Um yeah, let's see. Um I have time for one or two questions and then uh I think we can we can wrap. Uh let's see. >> How is it different from Jeppa? >> Uh yeah, how's it different from Jeppa? So Jeppa is uh doesn't train alongside the action labels, so they bet on infrared action labels for the most part, which I think like one of the ways of thinking about it is Jeppa I think in the limit is the more correct approach, but it's just a lot less practical in the short term because you need to bet on a separate scaling law on the action inference itself.

Um and I we really like Jeppa as a concept. We think it's a uh uh you know, we we use some of the things in the models. Um and uh but at the end of day it's it's it's an approach, not a product. Um and I think we'll, you know, borrow concepts both ways. Um the one way of thinking about it is um because we have so much action data, let's say you're flying a plane and you're moving the rudder in the back. There's no way of being able to tell that from the video frames. It's it's very very difficult. So our mode on having the action data really transfers better to action sequences than it does to world models and to to generating the pixels. And that's why we just went all in on training these general foundation models for action predictions. And we use we use all sorts of techniques to do world models. You know, we our own, we we really like other companies building them as well. Um So.

Great. Then one more and then I got to run, I think. Great. Um Great. All right. Thank you everyone. Have a good one. >> [applause]