Microsoft releases its SPTAG algorithm, used to answer questions on Bing, as MIT-licensed open source on GitHub
Peter Bright / Ars Technica :
Context & Ripple Effects
When Microsoft put SPTAG on GitHub in 2019, it was giving away the retrieval plumbing of its answer engine: the algorithm that finds semantically close passages so Bing can respond to questions rather than just rank links. At the time it read as a developer-relations gesture around a niche component.
Four years later that component looks like the load-bearing wall. The ChatGPT-powered version of Bing Microsoft planned for early 2023, the GPT-4 model customized for search behind the new Bing, and the generative search results that cite their sources all depend on exactly what SPTAG does — turning a query into the right passages to feed a language model. The 2019 open-source release quietly documented the retrieval half of what is now called retrieval-augmented answering.
First-order effects
- Any developer can now ship semantic nearest-neighbor search under an MIT license with no negotiation with Microsoft, removing a proprietary barrier for anyone building question-answering systems.
Second-order effects
- By commoditizing the retrieval layer while keeping the model and ranking layers closed, Microsoft set the template visible again in the new Bing: open infrastructure attracts an ecosystem, differentiation moves up the stack to the customized LLM and how answers are presented.
Third-order effects
- If the pattern holds, answer-engine competition settles into a structure where vector-retrieval components are table stakes available to everyone and advantage accrues to whoever controls the generation model and the source-attribution surface — the economics publishers and regulators are now contending with.
The trend: Search is being rebuilt as a retrieval-plus-generation stack, with companies open-sourcing the infrastructure layer while competing on proprietary models and answer presentation.