The first production-scale telemetry from GitHub Copilot's coding agent confirms what every infrastructure team has suspected: you are no longer the one driving the LLM. After a developer types a prompt, the agent takes over, firing off an average of 6.6 LLM calls on its own before handing control back. Across 13 million sessions and 761 million LLM calls sampled from one week in June 2026, 87 percent of all LLM invocations were agent-initiated, not user-initiated. The agentic coding workload that Microsoft researchers characterize in a new paper is fundamentally different from the chatbot traffic most serving infrastructure was designed for, and the implications for capacity planning, caching, and cost are immediate and concrete.
What did Microsoft actually measure?
The paper, titled "Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale," comes from Microsoft Research and draws on sampled traces from GitHub Copilot's coding agent spanning a week in June 2026. The dataset covers 13 million sessions from over 3.2 million users, encompassing 761 million LLM calls and 95 trillion tokens. This is not a benchmark or a synthetic evaluation. It is production telemetry from real developers using Copilot's agent mode in their daily work, and it is the first study of its kind at this scale.
The scale matters because until now, the AI engineering community has had to reason about agentic coding workloads from small-scale experiments and vendor claims. Microsoft's data is the first production-scale characterization of an AI coding agent in the wild, and it reveals a workload that looks structurally different from the chatbot traffic patterns most inference infrastructure was built to serve.
Each session consists of multiple turns: a user prompt followed by an autonomous agent response chain that couples LLM calls nearly one-to-one with tool execution. The agent reads files, runs commands, edits code, and calls the model again with the results. This tight coupling between tool outputs and subsequent LLM calls is what makes the workload distinct from conversational chat, where each user message produces roughly one model invocation.
How autonomous is the agent loop, really?
After a developer sends a message, the agent runs autonomously for an average of 6.6 LLM calls before returning control. The split breaks down to approximately 87 percent agent-initiated versus 13 percent user-initiated calls. The full paper on arXiv shows that this distribution is highly skewed: a small fraction of requests trigger long execution chains that account for a disproportionate share of total LLM calls and serving load.
This means that user request arrival rates are almost useless for capacity planning. A single user prompt can cascade into a dozen LLM calls in quick succession, and the load profile depends on the complexity of the task, the tools involved, and whether anything fails along the way. If you have been sizing inference capacity based on requests-per-second from active users, you are off by a factor of roughly seven.
The researchers also identified five distinct user archetypes with a 50x range in token consumption across them. Some developers fire off quick one-turn sessions that barely move the meter. Others run long, multi-turn workflows that burn through context windows and tool calls in extended autonomous chains. This heterogeneity means that aggregate metrics like average tokens-per-session hide a bimodal reality: most sessions are cheap, but the tail is where the cost lives.
Where does the infrastructure strain actually show up?
The most consequential finding for anyone running inference infrastructure is the KV cache behavior. Prompt prefix caching, which stores the key-value pairs of processed tokens to avoid recomputation, is the backbone of modern LLM serving efficiency. The paper shows that within a single agent turn, KV cache hit rates average 90 percent because the agent's calls share long common prefixes as it works through a task. That is excellent, and it is what makes the economics of autonomous agent loops viable at all.
But the picture degrades fast at the boundaries. Across turn boundaries, when the user comes back with a follow-up, cache hit rates drop to 55 percent. After a model switch, which happens when Copilot routes to a different model version or tier, only 8 percent of the cache survives. The chart above shows the collapse from 90 percent within a turn to 55 percent at turn boundaries to 8 percent after a model switch. If you are routing traffic across multiple model versions for load balancing or A/B testing, you are potentially destroying 82 percentage points of cache efficiency with every switch.
Tool failures amplify the problem. The researchers found that 9 percent of turns contain tool failures, and these failures trigger retry loops that consume roughly 4x the compute of a clean turn. An agent that hits a file permission error or a command timeout does not give up gracefully. It retries, re-reads context, and tries a different approach, each step generating fresh LLM calls against a prefix that may or may not still be cached.
The paper also highlights the gap between agentic turnaround times and user idle periods. At turn boundaries, the agent sits idle for an average of 4.1 minutes at the container level and 2.9 minutes at the KV cache level. That is time when compute resources are allocated but unused. The researchers built a lightweight idle-time predictor that captures 86 to 90 percent of total idle time, which could enable proactive resource orchestration and container scaling. But the core observation stands: the workload alternates between bursts of intense compute and minutes of dead air, and your infrastructure needs to handle both.
What does this mean for your capacity planning?
The implications for builders running agentic AI systems are concrete and immediate.
-
Stop planning at the request level. If 87 percent of LLM calls are agent-initiated, your capacity model needs to work at the turn or session level. A single user prompt can generate 6.6 calls on average, and the skewed distribution means some prompts generate far more. Model your load based on session complexity, not request arrival rates.
-
Cache invalidation is your biggest cost lever. The 90 percent within-turn cache hit rate is what makes agentic coding economically viable. The 55 percent cross-turn rate is what makes it expensive. The 8 percent post-switch rate is what makes it unsustainable at scale. Every model switch, context compaction, or cache eviction event is a direct hit to your inference budget.
-
Tool reliability is infrastructure cost. The 9 percent tool failure rate and its 4x compute amplification mean that improving tool success rates pays directly in inference cost. If you are building agents that call tools with flaky APIs, you are paying for retries in both latency and LLM tokens.
-
The idle time is not waste, it is opportunity. The minutes-long gap between turns is a window for cache eviction, container scaling, or serving other tenants. But it also means that naively keeping containers warm for every session will burn money. The 2.9-minute KV cache idle window is a scheduling decision, not a given.
This is the infrastructure reality behind the Copilot billing shock that developers have been reporting for months. The cost is not just in the model pricing. It is in the compounding effect of autonomous loops, cache misses, and retry amplification that multiplies the per-request token bill in ways that traditional cost models never accounted for.
What should builders do differently?
If you are building or running agentic coding systems, the paper points to several practical moves that can cut serving cost and improve reliability.
First, invest in session-structured caching. The 90 percent within-turn hit rate does not happen by accident. It requires prompt prefix caching that persists across the agent's autonomous calls within a single turn. If your serving stack does not support this, you are paying roughly 10x more than you need to for the bulk of your traffic.
Second, minimize model switches during a session. Each switch destroys 92 percent of your cache. If you are load-balancing across model replicas of the same version, ensure cache affinity by routing a session to the same replica. If you are A/B testing model versions, accept the cache cost as a deliberate tradeoff and budget for it explicitly.
Third, build retry budgets into your agent loop. The 9 percent tool failure rate is real, and unbounded retries are how a cheap turn becomes an expensive one. Cap the number of retries, log tool failures, and surface them to the user rather than silently looping. This connects to the broader agent plugin and portability standards that the ecosystem has been developing as agents gain more autonomy and more tools.
Fourth, the idle-time predictor the researchers describe is worth studying. It captures 86 to 90 percent of idle time, which means you could proactively free resources during the minutes-long gaps between turns without materially affecting response latency. If you are running a multi-tenant inference platform, this is free capacity waiting to be reclaimed.
Finally, watch the context compaction problem. The paper flags context compaction as costly, and it is the next frontier of agentic serving efficiency. As agents accumulate long conversation histories and tool outputs, compaction events invalidate cached prefixes and force reprocessing of large context windows. Building a compaction strategy that preserves cacheable prefixes will separate the efficient agent platforms from the expensive ones.
The workload shift you cannot ignore
The 87 percent number is the load profile of the fastest-growing category of AI compute. Agentic coding is the traffic pattern that already dominates one of the largest AI coding platforms in the world, and it is structurally incompatible with the infrastructure assumptions that most teams carry from the chatbot era. Your capacity models, your caching strategy, and your cost projections all need to account for the fact that the agent, not the user, is your primary customer now.
