Zuckerberg says Meta wants to build and open source AGI, and brings AI group FAIR closer to the generative AI team; Meta will own 340K+ H100 GPUs by 2024's end

And he wants Meta to open source it. Eventually. Maybe. … Fueling the generative AI craze is a belief that the tech industry …

The Verge 2024-01-19 Alex Heath

Context & Ripple Effects

Meta had already outlined plans to put generative text, image, and video tools into Messenger, WhatsApp, and Instagram through its earlier product-integration plan. This announcement ties that application push to a reorganized research effort and a much larger compute commitment.

The stated openness goal also foreshadows Meta's later strategy of using open-sourced models to broaden adoption and pressure rivals' pricing. Its subsequent 100K-plus-H100 Llama 4 training cluster suggests the infrastructure commitment became central to the model roadmap.

First-order effects

FAIR moves closer to Meta's generative-AI organization, reducing the separation between long-horizon research and teams building deployable models and features.
Meta becomes a major committed buyer of NVIDIA H100 capacity, while its AGI and eventual open-source ambition sets a public direction for the company's AI work.

Second-order effects

Rival model developers face added pressure to match Meta on both training scale and model availability, especially if Meta releases capable systems under open terms.
Large H100 commitments concentrate demand among a small group of AI developers, making access to high-end training compute a more consequential competitive input.

Third-order effects

If Meta sustains this approach, frontier AI competition may increasingly pair proprietary infrastructure spending with open model distribution: compute remains concentrated even as model access broadens.
The move is part of a durable split in AI strategy between firms seeking differentiation through closed services and those using open releases to shape the broader developer ecosystem.

The trend: Frontier AI is becoming a contest to secure enormous compute capacity while using model-release strategy to influence the ecosystem built on top of it.

Discussion

@zuck Mark Zuckerberg on threads
Our long term vision is to build general intelligence, open source it responsibly, and make it widely available so everyone can benefit.
@zuck Mark Zuckerberg on threads
Some updates on our AI efforts
@zuck Mark Zuckerberg on threads
We're building a massive amount of infrastructure. At the end of this year, we'll have ~350k Nvidia H100s — and overall ~600k H100s H100 equivalents of compute if you include other GPUs
@zuck Mark Zuckerberg on threads
There's someone for everyone.
@jonathangarelick Jonathan Garelick on threads
this guy has 600,000 H100 equivalents of compute. how are men supposed to compete in this dating market?
@aaldahle Ahmad Al-Dahle on threads
Today Mark shared that we are beginning to train Llama 3, and are moving toward a vision to achieve artificial general intelligence (AGI). We're doing this backed up by an incredible amount of AI compute to power these efforts now, and in the future. …
@jasonblackxyz Jason Black on threads
Meta casually able to do 20,400,000,000,000,000,000 operations *per second* across their 340,000 H100s. That's 2.04e19. 2.04 quintillion. And that's just with their H100 compute. Wow.
@sung.kim.mw Sung Kim on threads
I assume that Meta is probably the number one user of GPUs for its internal needs. Both Microsoft and Google have a large number of GPUs, but they are also major hyperscalers.
@mattnavarra Matt Navarra on threads
Mark Zuckerberg's new goal is creating artificial general intelligence And he wants Meta to open source it. Eventually. Maybe. Zuck also says Meta hasn't pivoted to AI after pushing the metaverse hard when it changed its name 2 years ago. …
@alexheath Alex Heath on threads
I talked with @zuck about Meta's new goal to create artificial general intelligence, or AGI. He's also moving the company's AI research group into the bigger product org today. And he's ready to flex about the immense amount of compute he's collecting to build super intelligenc…
@BrendanSinclair@mastodon.social Brendan Sinclair on mastodon
It's amazing to me that these execs can deflect questions about their last money pit trend chasing scheme that lost billions by hyping up their next money pit trend chasing scheme that will lose billions — https://www.theverge.com/...
@ylecun Yann LeCun on x
FAIR's mission is to develop the science and technology for human-level AI assistants: machines that understand the world, can perceive, remember, reason, plan, and act. In the not-to-distant future, all of our interactions with the digital world will be mediated by AI assistant…
@altryne Alex Volkov on x
Just in case you don't want to click over to the other sites, Big Zuck update - Open sourcing will continue - Currently training LLama 3 - AI + Metaverse - Will have 350,000 H100s and ~600 H100 equivalents of compute 🤯 -Ideal AI formfactor is 🕶️ [video]
@levie Aaron Levie on x
Everyone's goal in life should be to build a product where customers voluntarily share the volume of units they purchase. This is why NVIDIA wins. [image]
@norabelrose Nora Belrose on x
Open sourcing AGI will guarantee a “universal high income” for all, largely independent of government policy. There will ~always be an option to spin up a cheap AI and have it take care of you, either by trading in the market or by making food, shelter, etc. “off the grid”
@downingark Frank Downing on x
This is wild. 600k H100 equivalents at Meta is roughly 8x Tesla's roadmap by end of 2024 (assuming 1 H100 = 4 A100). The 350k H100 target by end of 2024 suggests a more than doubling of total GPU compute since H100 started volume shipping in Q1 2023 To be fair, Tesla's relative …
@jgarzik Ser Jeff Garzik on x
Meta spends billions on clunky VR headgear: did not move the needle. Meta spends billions on AGI: this will move the needle.
@alyssamvance Alyssa Vance on x
This is false! Zuck said: - Llama 3 is training now, *and* - By the end of 2024, Facebook's total compute infra will have 350K H100s (600K if you include other chips) Llama 3 itself isn't using 600K chips. Obviously, a training run now can't use the infra of a year from now. [ima…
@rihardjarc Rihard Jarc on x
$META in the AI race: - data on over 3B active users with multiple touch points - infrastructure: 600k GPU capacity by end of 2024 - AI talent - now Zuck laser focused on AI with the willingness and ability to invest Conclusion: I still don't own enough $META [image]
@rihardjarc Rihard Jarc on x
If people are still wondering why is $META open sourcing its LLM, here is the reason in as short of an answer as it goes. Again, Zuck sees where the puck is going while others are busy “controlling” the inevitable commoditized LLM models. [image]
@goyal_vivek Vivek Goyal on x
Zuck “people are also going to need new devices for AI” Smartphone sales last peaked in 2016/17 Desktop peaked in 2010/11 Laptops in 2020/21 With Meta launching Gen-AI powered glasses, Apple about to launch a new AI-powered iPhone, and innovative startups such as @humane and... […
@aravsrinivas Aravind Srinivas on x
Yann has confirmed that Llama 3 will be open-sourced!
@aravsrinivas Aravind Srinivas on x
Open Source AGI is an amazing vision. You are building a very powerful technology, and, actually aligning to what makes sense for the world: more people have a say in what makes sense and doesn't. Zuck and @ylecun are leading the revolution, and it's going to have an incredible…
@ballmatthew Matthew Ball on x
Zuckerberg on claims he or the company has “pivoted from Metaverse to AI”: “I don't know how to more unequivocally state that we're continuing to focus on Reality Labs and the metaverse”
@byrnehobart Byrne Hobart on x
@bryancsk Infinite supply of human-quality text raises the value of algorithmic moderation, at which Meta excels above all other companies. All you need is platform liability for comments and every comments section has to be run by Meta.
@dylan522p Dylan Patel on x
Zuck is so Chad - Committed to open sourcing general intelligence - Currently training LLama 3 - AI + Metaverse - Will have 350,000 H100s and ~600,000 H100 equivalents of compute - Ideal AI formfactor is glasses Full transcript below “Today I am bring Metas two AI reasearch...
@emollick Ethan Mollick on x
I keep telling every organization I speak to that the AI you are using now is the worst AI you will ever use, and the future is coming sooner than you think. If the scaling law continues for AI (bigger models = smarter models), we are going to see a rush of very powerful AI soon
@shiringhaffary Shirin Ghaffary on x
Zuck coming out hot with his plans to join OpenAI and Deepmind in building AGI (which means something different to everyone I talk to) Interesting that he -Frames this announcement as a way to attract talent -Is flexing Meta's GPU power by sharing numbers around that
@soumithchintala Soumith Chintala on x
Can finally talk some GPU numbers publicly 🙃 By the end of the year, Meta will have 600k H100-equivalent GPUs. Feel free to guess what's already deployed and being used 😉! [image]
@altcap Brad Gerstner on x
People continue to underestimate just how well positioned $META is for this AI moment. No giant monopoly being attacked ($GOOG), no low margin biz model at risk of commoditization (cloud software) and reaches 3 B people per day. And might just give us Open Source AGI! 💪💪🚀🚀 [image…
@josephjacks_ @josephjacks_ on x
Holy shit. Meta is training LLaMa 3 on 600,000 H100s.... That's $20 BILLION worth of GPUs.... L.F.G.
@jason_koebler Jason Koebler on x
guy who spent billions of dollars on a failed metaverse and changed his company's name now says AI is the thing he will do https://www.theverge.com/...
@bgurley Bill Gurley on x
Open-source is the ultimate disruptive business model. This is why so many people are running at politicians trying to create regulatory capture. They are appropriately fearful of fighting open-source on a level playing field. https://www.politico.com/...
@kylebrussell Kyle Russell on x
Meta GPU cluster scale vs academic GPU cluster scale [image]

Chronicles