Some developers are releasing versions of Llama 3, which has a context window of 8K+ tokens, with longer context windows, thanks to Meta's open-source approach
The InformationKalley Huang
Context & Ripple Effects
Meta had already made Llama available for research and commercial use with the Llama 2 release, creating a base that outside developers could adapt rather than merely consume. Extending Llama 3's usable input length is an early test of how quickly an open model ecosystem can alter a model capability after release.
First-order effects
Developers can offer Llama 3 variants that handle inputs beyond the base model's 8K-plus-token context window, giving users more deployment options immediately.
Meta retains the reach of the Llama ecosystem, while downstream builders—not Meta alone—determine which long-context implementations gain adoption.
Second-order effects
Model users must compare extended-context variants on more than headline capacity, including behavior and operating cost for their workloads.
Other open-model providers face pressure to make context length easier to modify or to compete on the quality and reliability of their supported long-context configurations.
Third-order effects
If such modifications become routine, context capacity may become a fast-moving ecosystem feature rather than a fixed specification set solely by the original model publisher.
The trade-off between larger context and the compute required to use it will increasingly shape which open variants become practical products, not just technically available ones.
The trend: Open model releases are shifting parts of AI capability development—from context length to specialized implementations—into a distributed developer ecosystem.
We've been in the kitchen cooking 🔥 Excited to release the first @AIatMeta LLama-3 8B with a context length of over 1M on @huggingface - coming off of the 160K context length model we released on Friday! A huge thank you to @CrusoeEnergy for sponsoring the compute...
@chrisbward
Chris B. Ward on x
Llama3, 32K context window GGUF models here
@devsharma_8
Dev Sharma on x
#Llama3 from @AIatMeta with a 🤯 1M Context Window running locally with one command. Open Source ❤️
@jessebenisrael
Jesse Ben Israel on x
Exciting news in natural language processing as Llama3 achieves impressive recall with a 65k context window, promising significant advancements in understanding long-range dependencies. #NLP #Llama3 #ContextWindow https://twitter.com/...
@namp
@namp on x
wow, this llama3-gradient with 1M context window seems to actually work [image]
@drnicka
@drnicka on x
Llama3 feels as good as GPT4 at creative reasoning, but has a (presumably intentionally) narrow context window. It has limitations, but this is a very capable model and anyone can use it. Humanity just levelled up. Viva la Open Source
@mikaelemtinger
Mikael Emtinger on x
Love open source. Suddenly there's a Llama3 with 1m tokens context window. Looking forward to needle in the haystack. https://ollama.com/...
@larmedai
@larmedai on x
Some really amazing work with #Llama3 going on right now and soon limit of context window will be a problem of the past.
@youraimarketer
@youraimarketer on x
AI21 just released the new Jamba-Instruct, instruction-tuned SSM-Transformer Model. It's competitive with Llama3 and Mixtral 8x7b but; It has a 256K context window (400 page novel) [image]
@enod_bataa
@enod_bataa on x
Clear winner in cost and speed: Llama3 + @GroqInc Only drawback is the limited context window.
@mascobot
Marco Mascorro on x
The OS LLama3 is moving fast. Llama3 8B-instruct with 160K context window, done with progressive training on augmented generations of increasing context lengths of SlimPajama
@thisritchie
Mike Ritchie on x
You can now run llama3, on your Mac, with a 1 million token context. That's roughly 2 novels worth of context (typical novel is ~100k words = ~500k tokens). IF you have 100's of GB of RAM on your mac 😂 “Note: using a 256k context window requires at least 64GB of memory. Using …
@stephan007
Stephan on x
Seriously... 1M local window context using Llama3 & Ollama?! 🫣 🧠 #LLM