Researchers from Apple and Columbia University released Ferret, an open source multimodal LLM that can recognize and describe any shape in an image, in October
With little fanfare, researchers from Apple and Columbia University released an open source multimodal LLM, called Ferret, in October 2023.
VentureBeatSharon Goldman
Context & Ripple Effects
Ferret put Apple and Columbia University into the open-source multimodal-model conversation with a system designed to recognize and describe arbitrary image shapes. The release is notable because it exposed a research capability rather than presenting a finished consumer feature.
Researchers and developers can inspect and build on Ferret’s approach to image regions and shape-grounded descriptions, rather than treating that capability as available only through a closed service.
Apple and Columbia gain a public research reference point in multimodal AI, while Ferret’s open-source release makes its methods easier for the wider research community to evaluate.
Second-order effects
Open availability raises the competitive baseline for multimodal research teams: differentiation shifts toward model quality, training data, product integration, and deployment efficiency rather than a paper-only capability.
The move creates a technical path from general image grounding toward application-specific interfaces, reflected in the later Ferret-UI work on mobile UI understanding.
Third-order effects
If major device-platform companies continue publishing and releasing multimodal research artifacts, the market may split between open research building blocks and proprietary production models, with value concentrating in data, distribution, and on-device integration.
Ferret is an early data point in a broader shift toward multimodal models that interpret interfaces and visual context, not just generate text; the pace of commercialization remains uncertain from this coverage alone.
The trend: Open multimodal research is turning visual grounding into a reusable model capability while product advantage moves toward deployment and integration.
Apple is going to integrate AI into their phones, and with their custom neural chips that excel at running machine learning, their new LLM will be integrated into your phone and will be local, not run on the cloud. I expect that will satisfy a lot of security and privacy conscio…
While Apple may be behind in the race for LLM, they are certainly catching up. Apple is sure to take a different path from other LLM companies, focusing on how to distill LLM and run it effectively on the iPhone. …
🧵2/2 They have proposed an inference cost model that coordinates with the behavior of flash memory usage, guiding optimization in two key areas: reducing the amount of data transferred from flash memory and reading data in larger, more continuous blocks. …
#news Apple has created a methodology for using Large Language Models (LLM) on iPhones. OpenAI's ChatGPT or Google's Bard are models that use a lot of memory, a real challenge for mobile devices. Apple researchers addressed the problem by leveraging flash memory. …
Apple's dive into open source AI with ‘Apple Ferret’ is a game-changer! 🚀 In just a year, LLMs run on phones, opening a universe of possibilities. Imagine AI decentralized right in your pocket—fewer data centers, more power to individuals. It's no surprise giants like Google a…
🧵3/5 In everyday life, possible applications of Ferret include: - Assisting visually impaired individuals: By accurately describing the location and shape of objects in the surrounding environment. - Education and research: Helping students and researchers analyze image data, pr…
“Apple is missing the boat on generative AI!” says the tech crowd. MLX. Dynamic caching in M3. ANE everywhere. Pooled memory in the SOC for CPU and GPU. SLM instead of LLM. And now this paper: https://arxiv.org/... (h/t @sung.kim.mw ). Do you see where Apple is going? …
Apple is sure to make LLM a major upgrade for Siri on the iPhone. Their recently announced method, LLM in a flash, is the best evidence. LLM in a flash: Efficient Large Language Model Inference with Limited Memory. Apple's approach is to store model parameters on flash memory …
I somehow missed this. @Apple joined the open source AI community in October. Ferret's introduction is a testament to Apple's commitment to impactful AI research, solidifying its place as a leader in the multimodal AI space. Way to go @Apple - ps: I'm looking forward to the day..…
FERRET: Advanced Multimodal LLM for spatial image references (Apple) - Hybrid Regions with discrete and continuous features. - GRIT Dataset: 1.1M samples for spatial accuracy. - SOTA in spatial understanding - Open-vocabulary, adaptable to varied inputs https://arxiv.org/...
In a CNBC interview published yesterday with Apple's Johny Srouji and John Ternus, the interviewer asks: “Critics have been surprised that Apple appears to be falling behind when it comes to Al, how do you respond?” They both chuckle and say “Not too worried.” [image]
Apple releases Ferret Refer and Ground Anything Anywhere at Any Granularity @Gradio demo: https://github.com/... An End-to-End MLLM that Accept Any-Form Referring and Ground Anything in Response [image]
@DrTBehrens ... With the advancement of Edge AI technology becoming a reality, Microsoft's reported annual spending of $50 billion on data centers needs to be re-examined. This spending pattern may need to be adjusted as more AI compute moves from traditional data centers to edge…
@sharongoldman ... could really change the AI landscape. If they would integrate LLMs on the operating systems level, it would start a new era of AI augmented human productivity and creativity!
🍎 leaks the future Ready to have a touch and talk assistant on your phone? > you point to something on your phone, ask a question, your phone responds > you ask your phone something, it pulls up an image, and tells you the answer while circling things you should note [image]
LLLMs (L = “local") are the killer app for catalyzing people caring about collecting their own personal data (as they create it and getting it back from others under CCPA and GDPR type access rights) so they can start getting amazing value for their own lives from their own data.
🎁🎁 Ferret is a multimodal LLM that is able to refer and ground, and is now open-sourced. Find out our code and checkpoints below: https://github.com/.... Merry Christmas and Happy new year! work led by @XyouH @HaotianZhang4AI @yinfeiy