Researchers from Apple and Columbia University released Ferret, an open source multimodal LLM that can recognize and describe any shape in an image, in October
VentureBeatSharon Goldman
Context & Ripple Effects
Ferret marks an early Apple–Columbia research release focused on connecting language models to regions and shapes within images. Its open-source framing makes the work relevant beyond a single product or demonstration.
The release gives researchers and developers an inspectable open-source multimodal model for experiments involving image regions, shapes, and descriptions.
Apple and Columbia gain a public research reference point in multimodal AI, while Ferret’s capabilities become available for outside evaluation and follow-on work.
Second-order effects
The release creates a practical baseline for teams building visual-language systems, particularly where models must refer to parts of an image rather than only produce an overall caption.
Its research direction is reinforced by Apple’s subsequent UI-screen-focused Ferret-UI work, suggesting that visual grounding can be adapted to interface understanding as well as general image tasks.
Third-order effects
If this progression continues, multimodal models may increasingly be judged on whether they can ground responses in specific visual elements, not just identify broad image content.
Open research releases can accelerate common evaluation and experimentation around visual-language capabilities, even as companies differentiate through later model scale, deployment, and product integration.
The trend: Ferret is an early data point in the shift from text-first LLMs toward multimodal systems that can reason about and reference specific visual context.
Apple is going to integrate AI into their phones, and with their custom neural chips that excel at running machine learning, their new LLM will be integrated into your phone and will be local, not run on the cloud. I expect that will satisfy a lot of security and privacy conscio…
🧵2/2 They have proposed an inference cost model that coordinates with the behavior of flash memory usage, guiding optimization in two key areas: reducing the amount of data transferred from flash memory and reading data in larger, more continuous blocks. …
#news Apple has created a methodology for using Large Language Models (LLM) on iPhones. OpenAI's ChatGPT or Google's Bard are models that use a lot of memory, a real challenge for mobile devices. Apple researchers addressed the problem by leveraging flash memory. …
While Apple may be behind in the race for LLM, they are certainly catching up. Apple is sure to take a different path from other LLM companies, focusing on how to distill LLM and run it effectively on the iPhone. …
Apple's dive into open source AI with ‘Apple Ferret’ is a game-changer! 🚀 In just a year, LLMs run on phones, opening a universe of possibilities. Imagine AI decentralized right in your pocket—fewer data centers, more power to individuals. It's no surprise giants like Google a…
🧵3/5 In everyday life, possible applications of Ferret include: - Assisting visually impaired individuals: By accurately describing the location and shape of objects in the surrounding environment. - Education and research: Helping students and researchers analyze image data, pr…
“Apple is missing the boat on generative AI!” says the tech crowd. MLX. Dynamic caching in M3. ANE everywhere. Pooled memory in the SOC for CPU and GPU. SLM instead of LLM. And now this paper: https://arxiv.org/... (h/t @sung.kim.mw ). Do you see where Apple is going? …
Apple is sure to make LLM a major upgrade for Siri on the iPhone. Their recently announced method, LLM in a flash, is the best evidence. LLM in a flash: Efficient Large Language Model Inference with Limited Memory. Apple's approach is to store model parameters on flash memory …
🍎 leaks the future Ready to have a touch and talk assistant on your phone? > you point to something on your phone, ask a question, your phone responds > you ask your phone something, it pulls up an image, and tells you the answer while circling things you should note [image]
🎁🎁 Ferret is a multimodal LLM that is able to refer and ground, and is now open-sourced. Find out our code and checkpoints below: https://github.com/.... Merry Christmas and Happy new year! work led by @XyouH @HaotianZhang4AI @yinfeiy
@DrTBehrens ... With the advancement of Edge AI technology becoming a reality, Microsoft's reported annual spending of $50 billion on data centers needs to be re-examined. This spending pattern may need to be adjusted as more AI compute moves from traditional data centers to edge…
@sharongoldman ... could really change the AI landscape. If they would integrate LLMs on the operating systems level, it would start a new era of AI augmented human productivity and creativity!
I somehow missed this. @Apple joined the open source AI community in October. Ferret's introduction is a testament to Apple's commitment to impactful AI research, solidifying its place as a leader in the multimodal AI space. Way to go @Apple - ps: I'm looking forward to the day..…
In a CNBC interview published yesterday with Apple's Johny Srouji and John Ternus, the interviewer asks: “Critics have been surprised that Apple appears to be falling behind when it comes to Al, how do you respond?” They both chuckle and say “Not too worried.” [image]
FERRET: Advanced Multimodal LLM for spatial image references (Apple) - Hybrid Regions with discrete and continuous features. - GRIT Dataset: 1.1M samples for spatial accuracy. - SOTA in spatial understanding - Open-vocabulary, adaptable to varied inputs https://arxiv.org/...
LLLMs (L = “local") are the killer app for catalyzing people caring about collecting their own personal data (as they create it and getting it back from others under CCPA and GDPR type access rights) so they can start getting amazing value for their own lives from their own data.
Apple releases Ferret Refer and Ground Anything Anywhere at Any Granularity @Gradio demo: https://github.com/... An End-to-End MLLM that Accept Any-Form Referring and Ground Anything in Response [image]