Microsoft researchers unveil Kosmos-1, a multimodal LLM they claim can understand image content, pass visual IQ tests, and accepts a variety of input formats
Microsoft believes a multimodal approach paves the way for human-level AI. — On Monday, researchers from Microsoft introduced Kosmos-1 …
Ars TechnicaBenj Edwards
Context & Ripple Effects
Kosmos-1 is a research milestone in a long Microsoft thread of marrying vision to language: the Semantic Machines acquisition laid conversational groundwork back in 2018, and by 2020 an Azure image-captioning service was improving accessibility across Word and Outlook as a bolt-on API. What changes with Kosmos-1 is architectural — image understanding moves inside the language model itself, which accepts images, text, and other formats as one input stream.
The same research lineage kept compounding after this announcement: VASA-1 turned a portrait photo plus audio into realistic talking-face video, and Magma fused visual and language processing into a foundation model that controls software interfaces and robotic systems. Kosmos-1 is the early proof point that native multimodality — not chained-together services — became Microsoft's chosen path.
First-order effects
Microsoft's own accessibility tooling — the Word and Outlook captioning that runs on Azure's image-captioning AI — is the most immediate internal consumer of a model that reads image content natively instead of through a separate vision endpoint.
By claiming performance on visual IQ tests, Microsoft hands researchers and rivals a concrete public yardstick for multimodal capability, moving the debate beyond curated demos.
Second-order effects
The vision-plus-language base gets reused downstream rather than shelved: VASA-1's photo-to-talking-video generation and Magma's interface-and-robot control both consume exactly the perceptual grounding Kosmos-1 demonstrated.
Standalone Azure vision features face internal pressure to consolidate — once a single model handles captioning-class tasks alongside reasoning, separately packaged image APIs lose their rationale within Microsoft's own stack.
Third-order effects
If the pattern holds, Microsoft converges on single foundation models that perceive screens and cameras, reason over them, and act on them — folding what began as discrete accessibility services into one multimodal platform layer, which is precisely where Magma's software-control ambitions point.
The trend: Microsoft's AI research is collapsing separately packaged vision, speech, and language capabilities into unified multimodal foundation models that both interpret and act on visual input.
Microsoft's new Kosmos-1 is incredible. It's a new Multimodal Large Language Model (MLLM). Their model can understand images, text, images with text, OCR, image captioning, visual QA. It can even solve IQ tests. Paper: https://arxiv.org/... Code: https://github.com/... https://tw…
@omarsar0
Elvis on x
Here we go! Microsoft introduces a multimodal large language model called Kosmos-1. Achieves great performance on language understanding, OCR-free NLP, perception-language tasks, visual QA, and more. https://twitter.com/...
@gerardsans
@gerardsans on x
Nice! Microsoft mocking Google Brain classic paper *Attention is all you need * 🤠 Kosmos1 Multimodal AIs is what's up people: text, image and sound 👽🛸 paper: https://arxiv.org/... https://twitter.com/...
@pkedrosky
Paul Kedrosky on x
Everything is fine. AI researchers are only here to help us. #xp https://twitter.com/...
@tsendeemts
@tsendeemts on x
Very cool work from MSFT on multimodal pre-training. Notice the input formatting they used. They annotated each modality with different tags and introduced HTML-like nesting to capture the layout. https://arxiv.org/...
@xixidu
Alexander Kruel on x
This new Multimodal Large Language Model (MLLM) paper is really cool: https://arxiv.org/... “We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language.” https://twitter.com/...
@cherrvak
K Sadov on x
“To the best of our knowledge, it is the first time that a model can perform such zero-shot Raven IQ tests.” https://arxiv.org/... https://twitter.com/...
@intuitmachine
Carlos E. Perez on x
Wait a second here! Is it attempting to perform visual analogies here?! https://twitter.com/...
@realhongyu_wang
Hongyu Wang on x
LLM can read, write and see now, and will quickly learn to hear, say, draw and use external systems in the future. https://twitter.com/...
@joshuasegeren
Joshua Segeren on x
How do AI large language models and chatbots get even smarter? By learning how to “see” and “hear”—not just how to read/write. MLLMs like @Microsoft's KOSMOS are the next step toward basic AGI. https://twitter.com/...
@aibreakfast
@aibreakfast on x
The next big leap in useable AI: The ability to understand images (see examples) Microsoft's Kosmos-1, a Multimodal Large Language Model (MLLM) conducts various vision tasks - and suggests MLLMs may be capable of nonverbal reasoning Link to paper: https://arxiv.org/... https://tw…
@joedevon
Joe Devon on x
This #AI research paper is significant. Including for #a11y. Web scale model that reads, hears and sees. Apply learnings across senses. It performed very well. Read text w/o OCR. Answering visual questions. Nonverbal reasoning. https://arxiv.org/... #chills