Inflection AI debuts Inflection-2, an LLM the company claims outperforms Llama 2 and Google's PaLM in some tests, and plans to add the model to its Pi chatbot
ForbesAlex Konrad
Context & Ripple Effects
Inflection had introduced Pi as a conversational sounding board before securing a reported $1.3 billion financing round, giving the startup both a distinct product surface and resources to develop its own models. This release connects those two efforts by bringing a proprietary model into the Pi chatbot introduced earlier in 2023.
The immediate benchmark claims are narrow—limited to some tests—but later coverage of Inflection-2.5’s reported Pi performance against GPT-4 suggests the company continued using model iteration to improve the consumer product rather than treating the model as a standalone research release.
First-order effects
Inflection gains a new in-house model to add to Pi, potentially changing the chatbot’s underlying capabilities without requiring users to adopt a separate product.
The company positions Inflection-2 against Llama 2 and Google’s PaLM on selected tests, increasing scrutiny of its model performance claims and of Pi’s resulting user experience.
Second-order effects
Rival chatbot and model providers face another comparison point in a market where model quality is increasingly judged through the consumer products built on top of it.
For Inflection, Pi becomes the primary channel for converting model advances into user engagement, making product distribution as consequential as benchmark results.
Third-order effects
If startups can repeatedly pair competitive proprietary models with focused user-facing products, AI competition may be shaped less by headline benchmark leadership alone and more by the ability to turn inference capacity into a differentiated, retained experience.
The pattern points toward tighter vertical integration between foundation-model development and application distribution, though sustained user adoption—not a single test result—would determine whether that structure endures.
The trend: Foundation-model startups are increasingly using owned applications as the proving ground and distribution channel for their models.
In benchmarks against LLaMA 2 and PaLM 2, Inflection-2 beat them on most counts, @inflectionAI said. It trails GPT-4 on several that OpenAI has published, but notes it's a much bigger model. Inflection started training its next model today, which it says will be 100x larger. [ima…
I asked Inflection CEO @mustafasuleyman about the situation at OpenAI last night, hours before the announcement of Sam Altman's return. Here's what he said — stressing empathy for @ilyasut and urging people to “be forgiving” about the boardroom mess. https://www.forbes.com/... [i…
Inflection-2 has Google PaLM 2 Large compute (10^25) and achieves 79.6 on MMLU, making it 2nd/3rd best LLM after GPT-4 and maybe Anthropic Claude 2! Strange that they don't reveal the model size or tokens trained on. [image]
even if these new models aren't up to par w/ the leading model, this makes me happy. Even if your model isn't that much better than llama, it's still a win what you're building isn't just the model, you're building the organizational muscle to train and build models
Speculation: Nobody has publicly beat GPT-4 yet, so if OpenAI keeps shipping, and there is an AI learning curve & no diminishing returns to scale, only Google keeps up. But Claude 2 and, now, Inflection-2 beat GPT-3.5, Grok is at GPT-3.5 level. And those labs are still training.
News: Inflection, the AI startup led by Mustafa Suleyman and creator of the chatbot Pi, has released a new model it says is the world's second-most powerful, Inflection-2. And Suleyman tells me at @Forbes that it's “very, very close” to catching GPT-4. https://www.forbes.com/...
Thrilled to announce that Inflection-2 is now the 2nd best LLM in the world! 💚✨🎉 It will be powering https://pi.ai/ very soon. And available to select API partners in time. Tech report linked... Come run with us! https://inflection.ai/...