LoRA is the undisputed workhorse of LLM customization. You freeze a model's weights, inject a tiny set of trainable parameters, and teach it a new skill for a fraction of the compute of a full fine-tune. It is cheap, it is modular, and it works beautifully for style transfer, tone matching, and simple single-turn classification. But if you have ever tried to fine-tune a model on a multi-step API workflow, a long SQL generation pipeline, or a complex customer support escalation path, you may have noticed something nagging: the model looks great on the first step and falls apart by step four. A new paper on arXiv, titled "Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures," formalizes exactly that suspicion and traces the failure to a structural property of the adapter itself. The core finding: LoRA's low-rank decomposition cannot represent the full interaction matrix required to chain dependent procedural steps, and accuracy drops from 96.3 percent at step 1 to 33.2 percent by step 5 on the benchmark constructed for the study, while full fine-tuning stays above 94 percent throughout.
What did the paper actually test?
The authors build a controlled benchmark of procedural tasks where each step depends on the output of the previous one, mimicking real agent workflows like nested API calls or sequential database transformations. They fine-tune a 7B parameter model using both LoRA at ranks 8, 16, and 64, and full fine-tuning, then measure per-step accuracy as the procedure length grows from one to five steps. The pattern is stark. LoRA at rank 8, the default configuration used in most production deployments, starts at 96.3 percent accuracy on step 1 but collapses to 33.2 percent by step 5. Increasing the rank to 64 helps marginally, reaching 41.7 percent at step 5, but the fundamental decay remains. Full fine-tuning, by contrast, holds at 97.8 percent on step 1 and 94.7 percent on step 5, showing that the pretrained weights have the capacity to internalize the procedure when the full interaction matrix is trainable.
The chart above shows the per-step accuracy trajectory. The gap between LoRA and full fine-tuning widens with every additional step, turning into a chasm by step four. The authors trace this to a mathematical property: procedural knowledge requires the model to condition each step's output on the accumulated state of all prior steps, which demands a full-rank interaction matrix between the hidden representations of step inputs and the output logits. LoRA's low-rank decomposition, by definition, cannot represent this matrix. The rank bottleneck is not a tuning issue or a data problem. It is a structural constraint of the adapter architecture itself.
How bad is the rank bottleneck for real agentic workflows?
The paper's benchmark is synthetic, but the finding maps cleanly onto the real agent workflows that builders are shipping right now. Consider a typical customer support agent that retrieves a user's account, checks entitlements, looks up billing history, applies a refund policy, and generates a summary. That is a five-step procedure where each step depends on the structured output of the previous one. If you fine-tuned your model with LoRA at rank 8, the paper's data suggests your agent is getting the full chain right less than 35 percent of the time. Full fine-tuning would keep you above 90 percent. The implications are serious for teams building agents on top of fine-tuned models rather than prompt-only architectures.
For builders, the stakes break down across three areas:
- Codebase: If your agent reliability relies on a LoRA fine-tuned model handling sequential tool calls, you are carrying a structural defect. You need either full fine-tuning or a prompt-only approach with a strong base model.
- Costs: Full fine-tuning a 7B model costs roughly 3 to 5 times the compute of a LoRA rank-8 adapter, which is why LoRA is so popular. If you need full fine-tuning for procedural tasks, your training budget and potentially your serving footprint grow.
- Moat: If your fine-tuned agent was your defensibility argument, and it was trained with LoRA on procedural workflows, the moat is thinner than you think. Competitors using full fine-tuning or prompt-only approaches on stronger base models will outperform you on long chains.
This connects to broader questions about agent evaluation gaps that enterprises face. Half of organizations ship agents without rigorous evaluation, and a LoRA fine-tuned model that looks fine on single-step tests will pass those weak evaluations only to fail in production when chains get long.
Why does rank even matter for procedural knowledge?
To understand why LoRA breaks, you need to look at how it works. Standard fine-tuning updates every weight in the model. LoRA freezes those weights and injects two small matrices, A and B, whose product approximates the weight update. If the original weight matrix is of size d by d, the LoRA update is a pair of matrices of size d by r and r by d, where r is the rank, typically 8, 16, or 64. The total trainable parameters are 2 times d times r, which is tiny compared to d squared. That is the source of the efficiency. But the rank r also limits the expressiveness of the update. The matrix A times B can only represent rank-r transformations. For style transfer or simple factual recall, this is enough because the required update to the model's behavior is genuinely low-rank. For procedural knowledge, the paper argues it is not.
The authors formalize this by analyzing the interaction matrix between hidden states and output logits across sequential steps. They show that the true interaction matrix for procedural tasks has a rapidly decaying singular value spectrum, but it does not become truly low-rank until rank exceeds approximately 256, far beyond what standard LoRA configurations use. At rank 64, the model still misses a substantial portion of the required interaction structure. The authors do not test rank 256 directly, but their mathematical analysis suggests that even very high rank LoRA would remain less efficient than full fine-tuning for these tasks. The practical takeaway: you cannot just crank the rank to fix this. The LoRA architecture is the wrong tool for the job.
What should I do about my LoRA fine-tuned agents?
If you are currently using LoRA for agentic workflows, the paper suggests a clear set of actions. First, audit your agent's task graph and count the sequential dependencies. If most of your workflows are single-turn or two-step, LoRA is likely fine. If you have workflows that chain four or more dependent steps, you are in the danger zone identified by this paper. Here is what to do:
- Run a per-step accuracy audit: Do not rely on end-to-end success rates. Break your evaluation into per-step accuracy and measure the decay curve. If you see a pattern similar to the paper's findings, LoRA is your bottleneck.
- Switch to full fine-tuning for procedural tasks: If the budget allows, move procedural training to full fine-tuning. The paper's data shows the pretrained weights have the capacity. You just need to unlock it.
- Consider prompt-only approaches: For some workflows, a strong general-purpose model with good prompting and tool use may outperform a LoRA fine-tuned model, especially if you do not have the budget for full fine-tuning. The paper notes that base models with chain-of-thought prompting handle procedural logic better than LoRA fine-tuned models that have memorized the steps.
- Segment your training: If you must use LoRA, use it for what it is good at. Train style, tone, and simple factual behavior with LoRA. Train procedural logic with full fine-tuning or keep it in the prompt context.
The broader industry context matters here. As models like those discussed in our OpenAI Presence enterprise AI agents coverage show, the push toward agentic workflows is accelerating. Enterprises are building agents that chain API calls, database lookups, and document processing into long pipelines. If those agents run on LoRA fine-tuned models, this paper predicts a specific, measurable failure mode that will surface in production.
What is next for cheap fine-tuning?
The paper does not declare LoRA dead. It identifies a specific limitation for a specific class of knowledge. LoRA remains highly effective for tasks where the required weight update is genuinely low-rank: style transfer, domain adaptation, tone matching, and single-turn classification. The finding is narrow but deep: procedural knowledge specifically is not low-rank. For builders, this means the answer is not to abandon LoRA but to use it selectively and understand what knowledge it can and cannot encode. The line is now clearer: use LoRA for style and domain vocabulary, use full fine-tuning or prompt-based approaches for multi-step procedural logic.
The line you should not cross with LoRA
The promise of LoRA was that you could teach a model almost anything for cheap. This paper draws a hard line around that promise. Procedural knowledge, the kind that powers your longest and most valuable agent workflows, lives outside the space LoRA can reach. If your product depends on a model executing multi-step procedures reliably, the cost of cheap fine-tuning is not measured in compute hours. It is measured in the gap between 33.2 percent and 94.7 percent at step five. That is the gap between an agent that works and one that quietly erodes trust with every dropped step.
Sources
- arXiv Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
- Data Today Enterprise AI agent evaluation gap: half ship broken agents
- Data Today OpenAI Presence bundles engineers with enterprise AI agents
