What the hell does Quant actually do?
A model is trained in BF16, released in Q8, and we use Q4 because we're told 'its nearly lossless'. Fine, but I was curious about the 'nearly' part, what is actually being lost between BF16 all the way down to Q2? As always, its complicated.
Here is a 15 second long example of local Qwen 3.8 Q3 XSS (massive aura loss from just typing that out) vs DeepSeek V4.1 Flash limited preview
Tfw you are a local tokencel
Qwen-flash-next 2 dgx sparks vs qwen3.8-27b on a 128gb Mac Studio.
Same prompt, one shot: "spark ignites in a glass box → fire grows → wind blows it out." Three.js, thinking ON (xhigh), no caps. Sparks (FP8, TP2): 21.5 min · 52,046 tok · 40.3 tok/s · thought 29.7K, wrote
I built a 3D DNA Explorer 100% Local With My OWN DNA Data.
My genome, rendered in 3D 🧬 Zero cloud calls.
You have reached the end of the archive
All of qwen38