SemiAnalysis launches InferenceMAX, an open-source benchmark that automatically tracks LLM inference performance across AI models and frameworks every night
vendor-neutral suite runs nightly and tracks performance changes over time Tae Kim / Barron's Online : Nvidia Touts Software Advantage in Beating Rivals Like AMD Dion Harris / NVIDIA : NVIDIA Blackwell Raises Bar in New InferenceMAX Benchmarks, Delivering Unmatched Performance and Efficiency GitHub : InferenceMAX™, Open Source Inference Frequent Benchmarking Irrational Analysis : Semianalysis InferenceMAX Launch: Surprising Data X: @nvidianewsroom : $75M in three years. NVIDIA Blackwell sets a new standard for inference ROI. Our most recent results show the NVIDIA Blackwell platform delivers the highest ROI at AI factory scale — projecting $75M in token revenue over three years from a $5M investment. @nvidia : To help companies get the most value, NVIDIA systems are built to deliver as much output as possible at AI factory scale. Recent InferenceMAX v1 results show NVIDIA sets the standard: One NVIDIA system can enable $75 million in revenue for AI companies using (tag DeepSeek AI) @nvidiahealth : NVIDIA Blackwell delivers 15x higher industry-leading performance and efficiency—validated by the new InferenceMAX benchmark from independent research firm SemiAnalysis. 🧬🏥 Redefining AI inference for healthcare and life sciences - unlocking higher ROI and new revenue @the_ai_investor : NVIDIA Blackwell leads the new SemiAnalysis InferenceMAX v1 benchmarks with top performance and efficiency. Best ROI: The GB200 NVL72 delivers a 15x return — $5M investment generates $75M in DSR1 revenue. Lowest TCO: B200 software cuts GPT-OSS costs to two cents per million [image] Stas Bekman / @stasbekman : If you're trying to choose which gpus to use for your inference workloads @SemiAnalysis_ started to publish frequently updated throughput and latency performance benchmarks for high end NVIDIA and AMD gpus https://inferencemax.semianalysis.com/ - open source code repo https://github.com/... - [image] @nvidiaaidev : Big shoutout to the @vllm_project team for an exceptional showing in the SemiAnalysis InferenceMAX benchmark on NVIDIA Blackwell GPUs 👏 Built through close collaboration with our engineers, vLLM delivered consistently strong Blackwell performance gains across the Pareto @nvidiaaidev : Our NVIDIA Blackwell set a high bar in the latest results of @SemiAnalysis_ InferenceMAX benchmarks. This new open source initiative provides a comprehensive methodology to evaluate inference hardware and software performance. Here are 5 key benchmark takeaways that [image] Dylan Patel / @dylan522p : All results and such can be accessed at https://inferencemax.ai/ And the code and everything is open sourced here https://github.com/... Methodology and explanation of results are here https://newsletter.semianalysis.com/ ... Michael Goin / @mgoin_ : Happy that InferenceMAX is here because it signals a milestone for vLLM's SOTA performance on NVIDIA Blackwell! 🥳 It has been a pleasure to deeply collaborate with @nvidia in @vllm_project, and we have much more to do Read about the work we did here: https://blog.vllm.ai/... Anton / @atroyn : you have to be InferenceMAX™ing. you need to be efficiently allocating compute. you need to be using open-source tools to understand true cluster tco. you need to be emulating real-world applications. congrats to @dylan522p and @SemiAnalysis_ team Guru Chahal / @guruchahal : This is a 👌👌 - standardized benchmarking for leading AI accelerators by @dylan522p and team! Dylan Patel / @dylan522p : To be clear this was an immense lift from the technical staff at SemiAnalysis. They are the shining stars here. They did something that I think will reshape how people make infrastructure purchasing decisions. Also huge thank you to Nvidia and AMD as well, and they engineers Mitesh / @mitesh711 : This is phenomenal work from @SemiAnalysis_ team! Wow - the best inference study all in one place. Aim for @positron_ai is to have large enough flops deployment to be listed here @dylan522p Dylan Patel / @dylan522p : Today we are launching InferenceMAX! We have support from Nvidia, AMD, OpenAI, Microsoft, Pytorch, SGLang, vLLM, Oracle, CoreWeave, TogetherAI, Nebius, Crusoe, HPE, SuperMicro, Dell It runs every day on the latest software (vLLM, SGLang, etc) across hundreds of GPUs, $10Ms of Forums: r/AMD_Stock : InferenceMAX™: Open Source Inference Benchmarking
Context & Ripple Effects
InferenceMAX extends the inference-benchmarking arc beyond periodic scorecards such as MLPerf's expanded LLM and image-model inference results by measuring throughput, latency and total cost of ownership repeatedly across changing software stacks.
The launch also puts recurring data behind a debate exposed in comparisons of AMD MI300X with Nvidia H100 and H200, where hardware specifications alone did not settle the outcome because software maturity affected realized performance.
First-order effects
- AI infrastructure buyers gain a vendor-neutral, open-source reference that can reveal nightly changes in model, framework and hardware inference results rather than relying on static vendor benchmarks.
- Nvidia Blackwell receives an immediate published performance-and-efficiency validation, while AMD and participating framework providers are evaluated in the same comparative workflow.
Second-order effects
- Procurement teams and cloud providers can place greater weight on workload-specific, continuously refreshed inference economics; vendors will have stronger incentives to tune drivers, kernels and serving frameworks as results move.
- AMD's competitive case becomes more dependent on converting theoretical hardware advantages into stable software results, a gap highlighted by the earlier MI300X versus H100/H200 assessment.
Third-order effects
- If broadly adopted, frequent open benchmarking could shift AI infrastructure competition from peak accelerator claims toward reproducible cost-per-serving and latency outcomes across the full hardware-software stack.
- The benchmark may make inference efficiency a more transparent input to platform purchasing and service margins, though its influence will depend on whether its workloads and methodology become trusted across the ecosystem.
The trend: LLM inference is becoming a continuously measured operational cost, making software-stack efficiency as consequential as accelerator specifications.