Google DeepMind details SIMA, an AI agent training to learn gaming skills to play like a human; SIMA trained on No Man's Sky, Goat Simulator 3, and other games
using the same pixels to keyboard/mouse controls as humans do. 1/8 [image] @amoufarek : So happy to be able to share what I've been working on with a fantastic team @GoogleDeepMind Introducing SIMA! 🤖🎮🥳🚀 Working on AI R&D as a gamer and former game developer, the SIMA project is my happy place 😀 I'll be at #GDC2024 next week, reach out if you'd like to chat! Andrew Lampinen / @andrewlampinen : We built our Scalable Instructable Multiworld Agent (SIMA) around the representations of internet-pretrained models, including a video model, and trained them primarily via instruction-conditioned BC, augmented with classifier-free guidance. 4/ [image] Shane Legg / @shanelegg : Our research project SIMA is creating a general, natural language instructable, multi 3D game-playing AI agent. The agent can carry out a wide range of tasks in virtual worlds, making AI more adaptable, helpful & fun! https://dpmd.ai/sima-1 [video] Edward Grefenstette / @egrefen : I've really enjoyed seeing this project produce an agent which does more and more amazing and diverse things every week. A great reminder that games remain an excellent medium to study generalization and complex behavior. Great job, SIMA team! @googledeepmind : SIMA needs only the images provided by the 3D environment and natural-language instructions given by the user. 🖱️ With mouse and keyboard outputs, it is evaluated across 600 skills, spanning areas like navigation and object interaction - such as “turn left” or “chop down tree.”... [video] @googledeepmind : We partnered with gaming studios to train SIMA (Scalable Instructable Multiworld Agent) on @NoMansSky, @Teardowngame, @ValheimGame and others. 🎮 These offer a wide range of distinct skills for it to learn, from flying a spaceship to crafting a helmet. https://deepmind.google/... LinkedIn: Emil Protalinski : Google DeepMind has unveiled an AI agent that learns how you play video games to help you, not beat you (for now). … Ali Arsanjani, PhD : 🚀 super excited to announce Google DeepMind #SIMA : Scalable Instructable Multiworld Agent. This generalist AI agent is focused on 3D virtual settings. … Forums: r/Futurology : Google DeepMind : Introducing SIMA, a Scalable Instructable Multiworld Agent r/Games : Introducing SIMA, a Scalable Instructable Multiworld Agent
Context & Ripple Effects
SIMA extends DeepMind’s earlier work on agents showing humanlike behavior in a first-person multiplayer setting by shifting from a single game environment toward instruction-following across several commercial 3D games.
The significance is the interface choice: SIMA acts from pixels through ordinary keyboard and mouse inputs, tying game-agent research to the broader challenge of agents operating through human-designed interfaces.
First-order effects
- Google DeepMind gains a cross-game research platform for testing whether instruction-conditioned agents can transfer navigation and object-interaction skills rather than mastering one fixed title.
- Partner studios supply varied playable environments for training, while SIMA is assessed on a common set of 600 skills across those worlds.
Second-order effects
- Game worlds become more useful as controlled yet visually diverse testbeds for multimodal agents, increasing the value of studio partnerships and environments that expose agents to varied tasks.
- Competing agent efforts face pressure to demonstrate transfer across interfaces and environments, not just high performance in a single benchmark or game.
Third-order effects
- If cross-world transfer improves, 3D simulations may become a more important intermediate training layer for agents intended to follow instructions and operate graphical software.
- The limiting question shifts from action generation alone to reliable generalization: broad game exposure can test robustness, but it does not by itself establish safe performance in real-world workflows.
The trend: SIMA is part of the move from narrow game-playing systems toward general-purpose multimodal agents trained to perceive, reason over instructions, and act through standard interfaces.