A look at Databricks' new open-source model DBRX, an LLM that cost ~$10M to train over several months and, Databricks says, outshines Llama 2, Mixtral, and Grok

Startup Databricks just released DBRX, the most powerful open source large language model yet—eclipsing Meta's Llama 2.

Wired 2024-03-27 Will Knight

Context & Ripple Effects

DBRX extends Databricks’ open-model work from the earlier Dolly release and its follow-on Dolly 2.0 model and employee-generated dataset. The shift is from a quickly trained, smaller effort to a model the company says required months and roughly $10 million to train.

The release matters because Databricks is positioning an open-source model against named proprietary and open-model rivals, including Meta’s Llama 2 and Mixtral. It makes model performance, rather than access alone, a more central competitive variable.

First-order effects

Databricks gains an open-source flagship model to offer customers and developers, while taking on the burden of substantiating its claimed performance lead over Llama 2, Mixtral, and Grok.
Developers evaluating self-hosted or customizable LLMs have another high-end option, but DBRX’s reported training cost underscores that reproducing frontier-class models remains expensive.

Second-order effects

Open-model rivals face added pressure to improve model quality, tuning tools, and distribution as Databricks uses DBRX to compete for the same developer and enterprise attention.
Enterprise buyers can use the availability of another claimed high-performing open model to strengthen their negotiating position with model vendors and cloud providers, particularly where customization or deployment control matters.

Third-order effects

If multiple vendors can release competitive open models, model access may become less differentiating and competition may move toward data platforms, deployment, evaluation, and support.
The reported cost suggests a bifurcated open-model market: broader downstream access to weights and customization, but a relatively small set of organizations able to finance training at the top end.

The trend: DBRX is part of the shift toward open-weight models becoming credible enterprise alternatives while the capital required to train leading systems remains concentrated.

Discussion

@lindensli Linden Li on x
Excited to release DBRX, a 132 billion parameter mixture of experts language model with 36 billion active parameters. It's not only a super capable model, but has many nice properties at inference time because of its MoE architecture. Long context (up to 32K tokens), large batch.…
@databricks @databricks on x
Meet #DBRX: a general-purpose LLM that sets a new standard for efficient open source models. Use the DBRX model in your RAG apps or use the DBRX design to build your own custom LLMs and improve the quality of your GenAI applications. https://www.databricks.com/... [video]
@mvpatel2000 Mihir Patel on x
🚨 Announcing DBRX-Medium 🧱, a new SoTA open weights 36b active 132T total parameter MoE trained on 12T tokens (~3e24 flops). Dbrx achieves 150 tok/sec while clearing a wide variety of benchmarks. Deep dive below! 1/N [image]
@code_star Cody Blakeney on x
Not only is it's a great general purpose LLM, beating LLama2 70B and Mixtral, but it's an outstanding code model rivaling or beating the best open weight code models! [image]
@code_star Cody Blakeney on x
It's finally here 🎉🥳 In case you missed us, MosaicML/ Databricks is back at it, with a new best in class open weight LLM named DBRX. An MoE with 132B total parameters and 32B active 32k context length and trained for 12T tokens 🤯 [image]
@vitaliychiley Vitaliy Chiley on x
Introducing DBRX: A New Standard for Open LLM 🔔 https://www.databricks.com/... 💻 DBRX is a 16x 12B MoE LLM trained on 📜 12T tokens 🧠DBRX sets a new standard for open LLMs, outperforming established models on various benchmarks. Is this thread mostly written by DBRX? Yes! 🧵 [image…
@code_star Cody Blakeney on x
I have to thank my amazing team (the @DbrxMosaicAI Data team @mansiege @_BrettLarsen @ZackAnkner Sean Owen and Tessa Barton) for their outstanding work. We have try made a generational improvement in our data. Token for token our data is twice as good as MPT7B was. [image]

Chronicles