LoRA (Low-Rank Adaptation) is a popular technique for fine-tuning large language models because it's memory-efficient. But a new analysis reveals a critical assumption that can break in production: LoRA treats all updates as low-rank, yet real-world fine-tuning often requires high-rank changes.
When you fine-tune for style—like tone, format, or persona—the modifications are simple and concentrated in few dimensions. LoRA handles these well. However, when teaching new factual knowledge (e.g., medical data or statistics), the information is spread across many dimensions. A low-rank setup (e.g., rank 8) cannot capture it all, leading to plausible but wrong answers.
Increasing the rank to compensate introduces instability. Standard LoRA's scaling formula (dividing by rank r) weakens the learning signal as rank grows. The fix is RS-LoRA, which changes the denominator from r to √r. This small tweak stabilizes high-rank training, enabling the model to retain complex information without collapsing.
A code walkthrough using NumPy demonstrates the failure from first principles. Two types of weight updates are simulated: low-rank "style" and high-rank "fact" updates. By measuring how much information survives at different ranks, the analysis reveals that naively increasing rank triggers a scaling collapse—and how RS-LoRA's single-character change fixes it.