New Hybrid Method Shrinks LLMs While Boosting Performance

Researchers have introduced Hybrid Policy Distillation (HPD), a novel technique for compressing large language models into smaller, more efficient versions without sacrificing accuracy. The method rethinks knowledge distillation by combining forward and reverse Kullback-Leibler divergence, balancing mode coverage with mode-seeking behavior. HPD leverages off-policy data and lightweight on-policy sampling to improve optimization stability and computational efficiency. Experiments show strong results in mathematical reasoning, conversation, and code generation tasks, suggesting a practical path toward deploying capable LLMs on resource-constrained devices.