Benchmarking AMD's MI300X and Nvidia's H100 and H200; in theory, AMD's GPU has advantages in specs and Total Cost of Ownership, but software bugs hold it back
Intro — SemiAnalysis has been on a five-month long quest to settle the reality of MI300X. In theory, the MI300X …
SemiAnalysisDylan Patel
Context & Ripple Effects
AMD positioned MI300X for LLM workloads with 192GB of HBM3 memory in its initial MI300 accelerator preview, then claimed up to 1.6x H100 inference performance at launch. This benchmarking effort tests whether those hardware-led claims translate into usable deployments.
The result makes the comparison less about peak specifications than about whether the software layer can reliably expose a lower total cost of ownership against Nvidia's H100 and H200.
First-order effects
AMD's MI300X cost and specification advantages are not fully realizable for users encountering software bugs, making real-world evaluation more workload- and software-version-specific.
Nvidia's H100 and H200 retain a practical advantage wherever their software environment delivers more dependable execution than MI300X.
Second-order effects
AMD must treat software reliability as a near-term product-competitiveness issue, not merely a support task, because benchmarked deployment friction can outweigh favorable hardware economics.
Prospective accelerator buyers are pushed toward validation on their own workloads rather than selecting hardware from headline performance and TCO comparisons alone.
Third-order effects
The episode reinforces that accelerator competition is increasingly decided by the integrated hardware-and-software stack; superior component specifications alone may not create an effective second source.
If this pattern persists, AI-compute procurement will assign a larger risk discount to platforms with less mature software, concentrating value in vendors that can make performance repeatable across deployments.
The trend: AI accelerator buying is shifting from peak-spec comparisons toward integrated-stack reliability and execution risk as the decisive measure of usable compute economics.
Ok, reading semianalysis.com/2024/12/22/ m... it seems that *most* of AMDs problems could indeed be remedied by having a team of 4-10 very strong engineers focus on always keeping default PyTorch fast on AMD hardware, with nightly regression tests.
NVIDIA to churn the hardware instruction set with abandon, sometimes even committing featurecide and relying on the PTX translator to emulate instructions that were removed. The PTX translation code is in the driver as well as the offfline toolchain (ptxas) and is 3/x
supercomputers in the world. So yeah, CUDA is a deep, deep moat. I get pretty offended when folks intimate that any luck was involved. We knew exactly what we were doing and why. /fin
why I was making sure it ran on Windows as well as Linux. It was healthy for the code base. Today, NVIDIA has parlayed CUDA's Windowa support into a monopoly position in GPU workstations, because 1,200 workstation apps use CUDA. Another pillar is PTX, which enables 2/x
multithreaded, so it can exploit modern multicore CPUs for performance gains proportional to the core count. Another triumph of software engineering. All of this great software runs on a span of platforms from tiny SOCs for cars and drones and robots, to the biggest 4/x
CUDA's software stack has a few distinct pillars that are triumphs of software engineering (let alone software architecture). The driver API was built in C, portable across both operating systems and CPU architectures. Across the 6 years I worked on CUDA, no one questioned 1/x
“AMD's software experience is riddled with bugs rendering out of the box training with AMD is impossible.” “It's not just that it's immature software, they need to change how they do development.” Good luck @dylan522p!
> Tensorwave, the largest AMD GPU Cloud has given GPU time for free to a team at AMD to fix software issues, which is insane given they paid for the GPUs. This is INSANE...
Recently tried to (painstakingly) write a convolution layer kernel using AWS Trainum with their NKI programming model (their latest and greatest), only to realize CUDA moat is well and truly alive!
Our 5-month journey conducting independent analysis & benchmarking of AMD MI300X vs Nvidia H100 + H200 Detailed, open source low-level benchmarks performance vs TCO Comprehensive public recommendations It's not just immature software, they need to change how they do development
> Tensorwave, the largest AMD GPU Cloud has given GPU time for free to a team at AMD to fix software issues, which is insane given they paid for the GPUs. glad we got the tinybox green
Repeat after me the reason Nvidia GPUs remain so well positioned - massive (and growing installed base) and architectural compatibility. Good analysis below.
“AMD” is insane because it appears to be one of the most legitimately dangerous companies with the potential to gigafry the market but exclusively employs literal turbonormies who unironically want to like design x86 processors and basically get oneshotted by their own drivers.
NVIDIA's GPUs excel due to their mature CUDA software stack, which provides better performance in practical scenarios even when they have inferior hardware specs. via @GeimanThiesen [image]