Google unveils Titans, an architecture combining RNN speed with transformer accuracy for real-time learning, which can scale to context windows of 2M+ tokens
Google’s AI work has repeatedly extended the transformer paradigm, from PaLM’s large decoder-only transformer to RT-2’s use of vision-language models to generate robotic actions. Titans shifts the emphasis from scaling a conventional transformer alone toward an architecture designed to retain transformer-level accuracy while operating with recurrent speed.
Google Research gains a new hybrid architecture positioned for real-time learning and contexts beyond 2 million tokens, expanding its alternatives to transformer-only designs.
Developers and researchers evaluating long-context systems now have a design that explicitly pairs recurrent processing with transformer accuracy, rather than treating those trade-offs as mutually exclusive.
Second-order effects
Competing model teams face added pressure to show not just larger context limits, but how efficiently their systems handle long inputs and update behavior in real time.
Long-context deployments could place greater value on architecture-level efficiency, affecting how teams allocate inference compute and evaluate the cost of retaining more working context.
Third-order effects
If hybrid designs deliver on their stated trade-offs, model competition may move from transformer scale alone toward specialized memory, recurrence, and attention combinations tuned for particular latency and context requirements.
The broader constraint becomes managing context as an operating resource: bigger windows are meaningful only where systems can sustain accuracy, responsiveness, and compute efficiency together.
The trend: Titans is part of a shift toward hybrid AI architectures that seek to make long-context, continually adaptive inference viable without relying exclusively on ever-larger transformers.
@JeffDean The Titans architecture is solid and well designed, integrating surprise based prioritization, momentum gating, and adaptive eviction into a fully trainable memory system in a clean and effective way. Its ability to outperform both Transformer++ and Mamba 2 on BABILong …
@GoogleResearch Lovely step. If Titans really wants “deeper than 2M tokens”, the choke point isn't context length but how you compose bounded memory. Right now both RNNs and Transformers are doing ad-hoc additions on activations that live in a finite dynamic range. In a
@JeffDean Extended reasoning over massive context windows is one of those capabilities that unlocks entirely new use cases we haven't even imagined yet. This is the kind of fundamental architectural innovation that compounds—each breakthrough in context + reasoning creates expone…
@GoogleResearch It solves Retrieval, but it breaks Alignment. Titans uses “Surprise” (gradient magnitude) to decide what to write to memory. But to an AI, a typo is “surprising.” A predictable tragedy is low-gradient noise. Without a Human Anchor (\Gamma) to modulate that surpris…
@JeffDean Fascinating work. The test-time updates based on ‘surprise’ seem powerful. What's the mechanism to prevent the memory from getting polluted by noisy or even adversarial inputs over time?
We have a long history of model architecture innovations, and we're continuing to explore new ideas. Here's some early work that shows promising results on extending the context length over which a model can reason over.
@JeffDean Great to see memory becoming a first-class citizen in model architectures. We've been exploring external cognitive memory systems using spatial semantic indexing to extend long-range reasoning, fascinating to watch internal and external memory paradigms converge.
@JeffDean This is a major step forward combining the efficiency of RNN-style recurrence with the expressive power of Transformers has been a long-standing goal in the field. Looks like Titans can truly deliver real-time learning with multi-million-token context windows, that resh…
@GoogleResearch This is genius! This is exactly the kind of architectural leap that enables AI systems to handle truly complex, long-horizon tasks. The implications for autonomous agents are massive.
Today at #NeurIPS2025, we present Titans, a new architecture that combines the speed of RNNs with the performance of Transformers. It uses deep neural memory to learn in real-time, effectively scaling to contexts larger than 2 million tokens. More at: https://research.google/... …