Text of code.visualstudio.com › blogs › 2026 › 07 › 06 › optimizing-vscode-coding-harness-model… as Styx crawled it on Oct 10, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

How Prompt Tuning Improved GPT-5.5 in VS Code

🎬 Watch The Story of VS Code! VS Code is now on Instagram, follow @vscode.ig for updates, tips, and more. Rewatch GitHub Copilot Day. How Prompt Tuning Improved GPT-5.5 in VS Code July 6, 2026 by VS Code Team, @code In our previous post , we introduced the VS Code coding harness, the layer that connects the model to tools, context, instructions, and the agent loop, giving the model the ability to perform coding tasks.

Each model responds to tool calls and instructions differently, and the harness can adapt to improve results. This post walks through a two-week experiment we ran in partnership with OpenAI to tune the GPT-5.5 system prompt in VS Code. The question was simple: if we nudge the agent to explore less and validate sooner, can it get faster and cheaper without getting worse? With OpenAI's model expertise and our harness data, we tested two small prompt changes, measured them against a control on live traffic, and shipped the winner.

This matters more with usage-based billing in place. Token efficiency isn't only an infrastructure metric: every token the agent spends wandering is a token you pay for and wait on. An agent that reaches a grounded edit sooner is both a better experience and a smaller bill. The hypothesis: explore less, validate sooner Following the launch of GPT-5.5 , we looked at how the model spent tokens inside the VS Code agent harness, as part of the work described in Improving token efficiency in GitHub Copilot .

Two patterns stood out: where the model spent tokens, and where it over-explored before acting. Agents can spend a lot of effort searching, rereading, and comparing nearby paths before making a useful edit. That pointed to a single, testable idea: the agent should spend less effort wandering and more effort moving through a deliberate loop of evidence, action, and validation. After testing different hypotheses and running offline evaluations, we turned that idea into two variants of the GPT-5.5 system prompt, both were promising in offline evals, and we tested them against the current default on live traffic.

Inside the experiment We ran the experiment in VS Code over a two-week window, splitting GPT-5.5 agent traffic across two treatment groups and one control group with a 25/25/25 split. Both treatments test the same hypothesis but differ in how much structure they add to the prompt. Group Variant name Description Traffic allocation Control PRPT_CTRL Current default prompt 25% Treatment A PRPT_SRCH Economical search and edit: single, compact reminder to limit exploration before acting 25% Treatment B PRPT_LRG Large prompt sections: broader restructure covering the full edit-and-validate loop 25% Note: The allocations add up to 75% because the experiment scorecard compares evenly sized groups.

The remaining GPT-5.5 traffic continued to use the default prompt outside this scorecard slice, so we could compare the treatments and control across the same kind of user traffic. Treatment A: economical search and edit Treatment A makes a small, focused change: a single, compact reminder that nudges the model to reduce unnecessary exploration. The <economical_search_and_edit> section in the prompt instructs the agent to start from a concrete anchor, gather only enough local context, avoid broad exploration, act once there is a cheap discriminating check, and avoid rereading unchanged context.

You can find the complete implementation details in gpt55BasePrompt.tsx : { economicalSearchAndEditEnabled && < Tag name = 'economical_search_and_edit' > - Start from the most concrete available anchor: a file, symbol, failing behavior, failing command, or nearby implementation surface. < br /> - Gather only enough nearby context to choose one plausible local hypothesis and one cheap check that could disconfirm it. < br /> - Prefer one targeted search or nearby read over broad repo exploration. < br /> - Once the cheapest discriminating check is known, act. < br /> - Do not re-read unchanged context unless a new result makes it relevant. < br /> </ Tag > } Treatment B: large prompt sections Treatment B tested a broader version of the same idea of limiting exploration.

Instead of adding a single, compact reminder about economical search, it reorganizes the agent workflow into explicit <Before_the_first_edit> and <After_the_first_edit> sections. Unlike Treatment A, these additions make the system prompt itself larger, so a key question was whether the added structure would still improve efficiency, not just agent behavior. The goal was to solve the full loop and not only the search step: form a local hypothesis before editing , avoid broad exploration , make a grounded first edit , and validate immediately after the first substantive edit .

You can find the complete implementation details in gpt55BasePrompt.tsx : { largePromptSectionsEnabled && <> < Tag name = 'Before_the_first_edit' > - Start from the most concrete anchor available: a file, symbol, failing behavior, failing command, test, or nearby implementation surface. If the request does not name one explicitly, use the first targeted search or nearby read to identify that anchor, then continue locally from there. < br /> - Before the first edit, gather only enough nearby evidence to state one falsifiable local hypothesis about how the requested behavior should work or why it is failing, and one cheap check that could disconfirm it. < br /> [...] - Once you can state one falsifiable local hypothesis, the nearby code path it depends on, one cheap check that could disconfirm it, and one small edit that would test it, the next action must be a grounded edit. < br /> - If confidence is incomplete, the first edit may be a small reversible probe that exposes missing types, behavior mismatches, control-flow gaps, or validation failures. < br /> - If you find yourself still searching after that local-routing budget, treat that as drift.

Recover by choosing the best current hypothesis and the best available nearby check, then make the smallest plausible edit that will let that check discriminate. < br /> </ Tag > < Tag name = 'After_the_first_edit' > - Prefer this order for that first validation action: < br /> - the cheapest behavior-scoped or failing check that can falsify the current hypothesis < br /> - a narrow test for the touched slice < br /> - a narrow compile, lint, or typecheck command for the touched slice < br /> [...] - Finish with at least one post-edit executable validation step whenever the environment provides one.

Only fall back to diff-only validation when no focused command exists or commands are unavailable. < br /> </ Tag > </> } What the two-week scorecard showed We tracked the treatments across three dimensions: quality (does the code stick), latency (how fast the first edit lands), and efficiency (tokens and tool calls). Each treatment is compared with the control group in the table below. What each metric measures 10-minute survival rate (by user): Of the code the model wrote, how much is still in the file 10 minutes later (not deleted or rewritten).

It's our proxy for "did the AI's code actually stick." Measured as surviving characters ÷ total characters written, as a %. E.g. ~90% — roughly 9 of every 10 characters the model added are kept. Commit survival rate (by user): Narrower and stricter: of the AI-written code, how much survives all the way into a git commit . This is "did it make it into real, saved work." Same character-ratio calculation, but only counting code present at commit time.

E.g. ~87%. p50 Time to First Edit (by turn): For a typical request, how long from hitting enter until the first actual change lands in your code — not just the model talking, but real work appearing. Measured in seconds. E.g. ~74s for the median turn. p95 Time to First Edit (by turn): The same clock, but for the worst 5% of requests — the "why is this taking so long?" cases. A key tail-latency guardrail. E.g. ~6.4 min (383K ms), where hard tasks or lots of expl

The rest of this page is on the live site.