Meta quietly unveils Llama 2 Long, which has been trained with longer sequences, outperforming GPT-3.5 Turbo and Claude 2 when responding to long user prompts
The long-prompt focus also foreshadows Meta’s later push to pair successive Llama generations with broad product distribution, including Meta AI’s rollout across its consumer apps.
First-order effects
Meta gains a model variant positioned around long-input responses, with reported results ahead of GPT-3.5 Turbo and Claude 2 on that specific task.
Developers evaluating Llama 2 for document-heavy or extended-prompt use have a more targeted Meta option to test against proprietary alternatives.
Second-order effects
The comparison raises pressure on rival model providers to demonstrate long-context quality, not merely advertise larger context windows.
For Meta, stronger long-prompt performance makes its open-model strategy more useful as a foundation for later model releases and product integrations; later coverage framed Llama 3 benchmark claims in a similarly competitive way.
Third-order effects
If long-context training becomes a standard differentiator, model competition will increasingly turn on reliability across extended interactions rather than one-shot prompt performance.
Meta’s trajectory suggests that open model releases and consumer-app distribution can reinforce one another: model improvements create more viable in-product uses, while distribution supplies a route to deploy them at scale.
The trend: This is one data point in the shift from general-purpose chatbot benchmarks toward models optimized for sustained, context-rich interactions and distributed through existing platforms.
Effective Long-Context Scaling of Foundation Models LLAMA 70B variant surpasses gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks https://arxiv.org/... [image]
Introducing the foundational long-context LLMs powering all the LLM agents across Meta's family of apps🥳! In short, longer context is not only an essential feature in real-world application but also a key axis in LLM scaling [1/4]
My favourite paper for today. Meta continues pretraining of llama2 with an additional 400B Tokens and closes the gap with GPT 3.5 Good news is that they used synthetic datasets and not human annotations to get quality improvement. True Tokenbenders 🫡
@Yampeleg @Scobleizer Meta's making noise with LLaMA 2 Long! 🎵 Robert Scoble's highlighting Yam Peleg's insights on this. Turns out, it's not just about tons of long texts for top-tier performance. And with LLaMA 70B outshining gpt-3.5? The AI plot thickens! 🧠💻😂
Meta finally addresses the issue of XPOS existing. Great work on RoPE; but I'm not convinced this is higher than 7B-L2-XPOS trained from scratch would show. We may be in the regime where LLMs are treated like obsolete urban infrastructure; patched up when demolition is overdue. […
Long LLaMA 2 The strongest versions of LLaMA 2 to-date!...Summary: Amazing work from meta, as always! Takeaways: - Do not train with long context from scratch (switch at the 80% mark) - You do not need long instruct datasets. You can generalize to long context via long pretrainin…