Sources: Microsoft is developing “distilled”, smaller AI models for features like Bing Chat that cost less to operate and mimic OpenAI's more advanced models
Context & Ripple Effects
Microsoft is pursuing a lower-cost layer beneath OpenAI’s advanced models for Bing Chat features, making inference economics a product-design issue rather than merely a research concern. The closely related coverage describes the same cost-cutting effort around more efficient models for Bing Chat.
The strategy sits alongside a broader push toward smaller language models: Microsoft and peers have pitched smaller models as a way to reduce training and hardware requirements. Microsoft’s later work on a large in-house model suggests it is exploring both ends of the model stack.
First-order effects
- Microsoft can route suitable Bing Chat tasks to distilled models, lowering the operating cost of those features while retaining a path to OpenAI-class capabilities for harder requests.
- OpenAI becomes less central to the specific Bing Chat functions that Microsoft can serve adequately with its own smaller models.
Second-order effects
- A lower per-query cost gives Microsoft more room to expand AI features or absorb usage growth without relying exclusively on the most expensive model tier.
- Other AI product providers face pressure to adopt model-routing and distillation approaches, competing on the cost of delivering acceptable task performance rather than model size alone.
Third-order effects
- If this architecture proves durable, AI platforms may increasingly separate frontier-model development from mass-market serving: large models generate capabilities, while smaller specialized models deliver them economically.
- That shift could strengthen the bargaining position of major model buyers that can develop substitutes, even as frontier models remain important for training, evaluation, and complex tasks.
The trend: Generative-AI companies are moving toward multi-model stacks in which inference cost per useful task determines which model reaches the user.