Strata Runs a 125B Model on a 12GB GPU
Strata, an MIT-licensed inference engine, claims to run a 125B model on a 12GB GPU at 60 to 95 tokens a second. What it needs and what to doubt.
Strata is an open-source inference engine that claims to run Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on a single NVIDIA GPU with 12 GB of VRAM, as long as the machine also has 32 to 64 GB of system RAM. The project reports 60 to 95 tokens a second on its test machine. Those figures come from the developers, and I found no independent benchmarks yet.
What the project says
The Strata README lists measurements on an RTX 5070 with 12 GB of VRAM, a Ryzen 5 7600 and 64 GB of RAM. It gives three quantisation levels, which trade file size and speed against output quality.
Quantisation | Output tokens per second | Input tokens per second |
|---|---|---|
Q2_0 | 93 | 2,170 |
IQ2_XS | 79 | 2,090 |
IQ3_S | 53 | 1,620 |
The engine is MIT-licensed. The model itself comes from the Qwen team, with compressed versions made by ISTA-DASLab, according to the README. AI Weekly's write-up describes the approach as keeping the experts in active use on the GPU and holding the rest in system RAM, and notes that no third-party benchmarks have been posted yet.
What you need
An NVIDIA RTX 20, 30, 40 or 50 series card with at least 12 GB of VRAM.
32 to 64 GB of system RAM, depending on the model file.
Around 80 GB of free disk, with an SSD recommended.
Windows 10 or 11, or Linux.
Why a model this big can fit at all
A mixture-of-experts model only activates a few of its specialist sub-networks for each token, so the work per token is far smaller than the total size suggests. The weights still have to live somewhere, which is why the setup leans on system RAM. If that is new, what mixture of experts means explains the idea, and what quantisation is covers why the Q2 and Q3 files are so small.
Reasons for caution
The speeds are self-reported. Treat them as an upper bound for a similar machine, not a guarantee.
Two-bit quantisation is aggressive. Expect some quality loss and test on your own tasks before trusting the output.
One request at a time. The README says requests are processed sequentially, so this is a personal workbench, not a server for a team.
Slow starts. The README warns the PC can be slow or unresponsive for one to three minutes during initial startup.
Check the repository. A search also surfaced a second repository with a near-identical description under a different owner. I read the one linked above and could not establish which is canonical. Confirm the source before running any one-click installer, and read whether it is safe to run an AI model locally first.
Why it matters for builders
The README says Strata exposes OpenAI-compatible and Anthropic-compatible endpoints on localhost. For a builder, that means an app already written against a hosted API could be pointed at a local address for testing, with no usage bill. That is useful for experiments and private data, and less useful where you need speed and concurrency.
For the wider picture of when local beats hosted, see local AI models and running an AI coding model locally. The fundamentals are in how AI models work.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
