MoE Models
-
MoE Expert Offloading: 125B on a 12GB GPU, With Catches
MoE expert offloading lets a 12GB GPU run a 125B model, but only for one user at a time. What Strata's numbers say, and what I'd run instead.
MoE expert offloading lets a 12GB GPU run a 125B model, but only for one user at a time. What Strata's numbers say, and what I'd run instead.