AMD drops its first small language model

While everyone is burning billions to train trillion-parameter behemoths, AMD just went the exact opposite direction. The chipmaker unveiled its first small language model, a featherweight 135-million-parameter model dubbed AMD-135M.

Built on the Llama architecture, the model is aimed squarely at enterprise private deployments. AMD's recent Silo AI acquisition is still clearing regulatory hurdles, so this appears to be homegrown work designed to show enterprise buyers that AMD hardware can handle modern AI stacks from scratch.

The draft runner for giant models

AMD-135M is not built to replace giant models. It is designed to make them faster using speculative decoding.

Standard LLMs generate text token by token, which is slow and memory-bound. Speculative decoding solves this by pairing a fast, tiny draft model with a large target model. The draft model spits out a batch of candidate words in one go, and the big model simply validates or corrects them in parallel. It speeds up generation significantly, though extra data moving between models does pull more power.

AMD rolled out two versions. The base model, AMD-Llama-135M, was trained from scratch on 670 billion general tokens in just six days across four 8-way Instinct MI250 nodes. The coding variant, AMD-Llama-135M-code, received an extra 20 billion code tokens and finished fine-tuning in four days on the same setup.

In benchmarks run by AMD, both models reportedly delivered strong speedups.

Why it matters

We think AMD's move here is less about the model itself and more about flexing full-stack credibility against Nvidia.

For enterprise teams running internal LLMs, latency and serving costs are huge headaches. A 135M model takes almost no VRAM, meaning companies can easily slot it alongside an existing 70B model to slash response times without buying a whole new rack of GPUs.

Strategically, AMD is trying to break free from being viewed merely as a secondary chip vendor. By training, releasing, and tuning its own pre-trained weights specifically optimized for its Instinct hardware, AMD is showing prospective clients that its software ecosystem is finally mature enough to deliver real-world speedups.

That said, speculative decoding only works well when the draft model guesses accurately. If the big model rejects too many tokens, you end up burning power for zero gain. Plus, AMD tested this on last-generation MI250 hardware. If this setup works half as well on current MI300X or upcoming MI325X chips, AMD might actually have a compelling enterprise inference package on its hands.