In the latest deep dive, creator Nichonauta put Qwen 3.5 models to the test, comparing the 4B and 9B parameter versions under different quantization levels—4-bit vs 8-bit. The burning question: does a larger, more compressed model outperform a smaller, higher-precision one?
Key Comparisons
The analysis covers not just raw performance but also the impact of quantization on reasoning loops, overfitting, and task-specific outputs. Smaller models (under 4B parameters) showed clear limitations in complex reasoning tasks, often falling into reasoning loops and probability traps.
Highlights from the Video
- Reasoning & Probability: Models with fewer than 4B parameters exhibited erratic reasoning loops and over-reliance on probability, leading to less reliable outputs.
- Overfitting & Pruning: The 4B model, when heavily trained, showed signs of overfitting—memorizing specific outputs like passwords rather than generalizing. Pruning techniques were discussed as a potential fix.
- Data Structures: Both models generated similar data structures, but the 9B model maintained consistency even at 4-bit quantization, while the 4B model degraded notably at 8 bits.
- Quantization Impact: The 9B model at 4 bits retained most of its capability, outperforming the 4B model at 8 bits in development tasks, suggesting that parameter count may matter more than bit depth.
Takeaway
For developers and AI enthusiasts, the results suggest that a 9B model aggressively quantized to 4 bits can deliver better results than a 4B model at 8 bits, especially in reasoning and code generation. The video recommends prioritizing model size over precision when deploying LLMs on constrained hardware.