Strata’s README warns you about something before it shows you any speed: while the model starts, your PC can be slow or stop responding for 1-3 minutes, because the installer loads 35-55 GB into system RAM and locks part of it for the graphics card. That one line taught me more about MoE expert offloading than the benchmark table did. The 12GB card never holds the 125B model. System RAM holds it, and the card only gets what the current token needs.
Why MoE expert offloading works on a 12GB card
Qwen3.8-Flash-Next is a mixture-of-experts model with about 125B parameters in total, and for every token it generates, a router picks a few experts and uses only 5.9B parameters. Everything else sits idle for that token. A dense model has no such luxury, so for dense models “fits in VRAM” and “runs” mean the same thing.
Strata is built on that gap. The experts live in system RAM, 43 GB of them for the IQ3_XXS quant and 50 GB for IQ3_S, according to its DETAILS.md. Whatever VRAM is left over becomes an expert cache. When the router asks for an expert that is not in the cache, the CPU computes it straight from RAM, or part of the work goes over PCIe to the GPU, and an “adaptive tier” shuffles experts between the two as the traffic changes. There is even a flag for it, --pcie-frac. The project’s MIT-licensed repo is twelve days old as I write this, created September 24, and it shipped v0.1.35 through v0.1.39 between October 1 and October 4. Five releases in four days. Somebody is fixing things very fast, which I like and also do not trust yet.
I did not find any independent write-up of how well the paging holds up, so everything below about speed is the project’s own claim.
What Strata measured, and what it did not
On an RTX 5070 with 12GB, a Ryzen 5 7600 and 64 GB of RAM, the README table says 94 tokens per second for writing answers on the Q2_0 quant, 79 on IQ2_XS, 62 on IQ3_XXS and 53 on IQ3_S. Reading a 32K-token prompt runs from 1,620 to 2,650 tokens per second depending on the quant. An AMD RX 9070 XT with 16GB and 47 GB of RAM gets 60 tokens per second on Q2_0. The authors also say an RTX 3090 “should” do 100-140, which is a guess, and they label it as one, so I leave it out of my own arithmetic.
Those are all single-stream numbers.
One request, one answer being written. The README says nothing about what happens when a second request arrives while the first one is still going, and I would not assume it is fine, because both requests would be fighting for the same RAM-to-GPU link that the whole trick depends on.
One more number from the docs shows how much the cache matters. At a 262K context, moving the KV cache out of VRAM took Q2_0 from 50.9 to 62.6 tokens per second, because 1,589 experts in VRAM became 3,872. And the closest-to-full-quality option, Unsloth’s UD-Q4_K_XL, reads most of the model from the SSD while it answers, so on a 64 GB PC it writes only 7-8.5 tokens per second. That is a different quant than the 53 above, so it is not a clean comparison. Still, it is the same engine and the same kind of machine, and it went from “faster than you can read” to “you will watch it type”.
Why Victoria does not rescue a 12GB card
The obvious alternative is to remove the experts instead of paging them. Victoria does that to the same Qwen base: it cuts the experts from 512 to 288 per layer, a 44% reduction, retrains at 4-bit, and keeps the 5.9B active parameters untouched. On Terminal-Bench 2.1 it scores 70.0% against the base model’s published 70.8%. Those scores are the author’s, not mine. The license is Qwen’s community license, which is open weights but not OSI open source, and the model card shows 3,095 downloads today, so nobody has really beaten on it yet.
Mettons that you want this on your 12GB card. My napkin math says it will not fit. If nearly all of the roughly 126B parameters sit in the experts, and 288 of 512 survive, you are left with something near 70B parameters, and at 4 bits that is around 35GB of weights before you store a single token of context. Victoria’s own speed numbers confirm the target: 279.6 tokens per second with its speculative draft head, measured on one NVIDIA B300. A B300 is a datacenter card.
“Single GPU” in that model card does not mean the thing under your desk.
Strata knows this. Its own “Coder” size is a version with half of the experts removed, and the README says it reaches 91% of the full model’s SWE-bench Verified score, measured by its authors, and fits in 32 GB of RAM. It is weaker outside of code, including Chinese and other CJK text. So pruning and paging are not rivals: the Coder row in the README table, 55 tokens per second on that same 5070, is both tricks at once.
The $7,000 option
The third way is to buy memory until the problem goes away. Mini PCs on the AMD Ryzen AI Max+ PRO 495, such as the Minisforum MS-S1 Max, come with 192GB of unified LPDDR5X-8533 for roughly $7,000 to $7,400, vendor pricing. The vendor also claims 131 TOPS. I found nobody who is not selling the box measuring that, so I treat it as marketing until someone runs llama.cpp on it and posts the numbers.
Me, I would skip it. 192GB is wonderful, sure, but $7,000 is a lot of hosted inference for a company our size, and Strata runs on a gaming card that plenty of people already own. If I had no GPU at all and a hard rule that customer data never leaves the building, I would reconsider.
What I am going to run at Trellis
Honestly, Strata is the one I want to try first, with Victoria’s method in mind as the part I would copy. Not Victoria itself, since it does not fit the hardware I have. Strata’s Coder size is the closest thing to that idea that runs on a normal card today.
My plan is boring. Put Strata on a spare machine with 64 GB of RAM that nobody depends on, point it at the work our agents do all day, which is parsing competitor catalogs and re-ranking listings, and let it run for a few weeks. I added up my own pipeline’s API bills in Half My Agent’s Budget Goes to Checking Its Own Work and the quality gate I built last ate 51.5% of everything across four runs, so I do not trust a cost story until I have measured the whole bill myself. Same here. A README number comes from one setup with one user.
The test I care about is simple. Start one request, then start a second one halfway through, and write down the tokens per second each stream gets. If the second stream drops below something like 20 tokens per second, I will stop calling this a server and keep calling it a toy. If it holds near the single-stream figure, then Strata matters a lot more than a twelve-day-old repo should. I do not know which way it goes, and I am curious whether v0.1.40 or 0.1.41 will already change the answer.
