CROSSPOST: PAOLO PERRONE: AI Coding Tools: What Changed in the Last 6 Months
Paolo's own subheadline: "A 10-minute update on what shipped, what broke, and what it costs now". From Autocomplete to loose cannons in this truly New Age of "AI" agentic coding tools...
A report from the coalface itself: Paolo Perrone of <http://theaiengineer.substack.com>: “AI”-agents have escaped the Integrated Software Development Environment, vendors have minted their own models, the bills have gone metered—and nobody trusts the resulting software code. We now have all the problems with debugging work done in Excel raised to a higher power by the tireless industry of only half-hinged software coding ‘bots…
Briefly: use of AI coding agents is high, but trust in them is—rightly—very low.
For one thing, they are not consistent: thus “no benchmark score predicts how a coding model does on your codebase…”
“AI” now writes, tests, and ships your patches; the hard part is making sure it hasn’t quietly set the building on fire. Today’s autonomous coding agents hallucinate packages, leak secrets on command, and quietly destabilize delivery. The new rate-limiting step is how much skilled programming time you can pay for to undertake skilled human verification, or how much you are willing to accept the costs of sometimes shipping slop in order to get something out the door.
And the METR study says that: experienced developers were 19% slower with AI on familiar code, even though they felt faster. And the Package hallucinations study says that: 19.7% of AI‑generated code samples referenced non‑existent packages, with repeated fake names in open models.
I think I understand the first of these. Vibe coding is a pseudo-social experience, which feels much more natural to the East African Plains Ape than is working alone. Plus you are making progress, tangible progress, every minute. That the tangible progress is trying to calibrate the LLM’s performance so that it does something useful, rather than actually writing code that will ultimately go into some repeatable and repeated workflow—that is not something you feel in the moment. Hence you do more, and accomplish less, while vibe-coding.
On the other hand, the hours that you spend trying to decode Stack Overflow threads and thumbing through your ORA animal-cover programming-reference books are now removed from your day.
And I definitely understand the second: to the LLM, a bug that does not make the machine instantly barf when tested is not a bug it catches. So there is no way the code it rights can be better than the harness, because the machine cannot look back at the code and say, “Oh, that! That’s wrong!” If it doesn’t barf, it did not happen.
The constraint is not that some LLMs are better than others on code-writing benchmarks and tests. The constraints are:
how fast can safely review what the agents did?
how correctly can you accurately review what the agents did?
what are your code-building, code-testing, and general security practices to keep the agents from flooding your codebase with bugs that do not immediately crash or trigger flags?
Or worse, create exploitable dependencies?
How far is what comes out from something verified?
How far is what comes out from something corrected?
AI coding tools stopped competing on autocomplete and started competing on autonomy. Agents moved into the terminal and the cloud. Every major vendor shipped its own model. SWE-bench cleared 80% and bunched up, flat-rate plans gave way to metered billing, and adoption rose while trust fell.
The agent left the editor. The default unit of AI coding is no longer a suggestion in your IDE. It is an autonomous agent running in a terminal or a cloud sandbox, opening its own pull requests while you do something else.
Every tool became a model maker. Cursor and Cognition now ship their own coding models, the labs ship a new one every few weeks, and at least one “in-house” model actually runs on a Chinese open-source base.
SWE-bench stopped being a flex. Scores cleared 80% and bunched up, and even the leader’s 88% on the old benchmark falls to 69% on the harder ones, so a single number no longer tells you much. Vendors have already moved the goalposts.
Flat-rate pricing died. Unlimited plans gave way to metered credits, capped by GitHub Copilot moving every plan to usage-based billing on June 1.
Adoption rose, trust fell. More developers use these tools than ever, fewer trust what they produce, and the most-cited productivity study spent six months walking back its own follow-up.
📬 New here? The AI Engineer explains how AI systems actually work, from first principles to production, twice a week. Subscribe to keep your mental map current as the tools keep shifting.
The Agent Left the Editor
You spent the last six months on one product. Head down, one repo, no time for tool drama. This week you come up for air and open your setup. Windsurf is gone: Cognition renamed it Devin Desktop in an automatic update. Your Cursor plan meters usage now, and the bill is climbing. GitHub Copilot has turned into five different agents wearing one subscription. And the model you benchmarked in December is several versions behind.
In 2024, an AI coding tool meant autocomplete. You typed, it suggested the next few lines, you hit tab. The center of gravity has moved. The unit of work is now a task you hand off, and an agent that goes and does it.
GitHub’s Copilot coding agent went generally available in September. You hand it a GitHub issue and it works in its own cloud sandbox, no editor open1.
Watch one run end to end. The agent clones the repo into a sandbox and greps for the export code that mangles any row with a comma in a field. It writes a failing test that reproduces the bug, patches the writer to quote fields, and runs the suite. Two unrelated tests break, so it backs out half the change and runs again until everything passes. It opens a pull request with the diff, the new test, and a note on what it tried. You never watched it work. You review the PR like any other, and the work has moved to your queue as something to check instead of something to write.
Then the surface multiplied. In February, OpenAI wrapped Codex in a desktop app it calls a command center for agents, where you fire off several coding tasks in parallel and supervise them like a team of junior engineers2.
In April, Cursor shipped its 3.0 release, codenamed Glass, and deleted the Composer side-pane that had defined the editor since 20243. In its place is an Agents Window where you run multiple agents at once across your local machine, git worktrees, and cloud sandboxes. We took that editor apart in How Cursor Actually Works.
Terminal-native agents grew up too. Claude Code, the command-line agent we covered in How Claude Code Actually Works, added dynamic workflows in May. It writes an orchestration script and runs tens to hundreds of subagents in the background against one goal. In December, Anthropic said Claude Code had passed a $1 billion revenue run rate within about six months of launch4.
Even the casualties tell the story. Windsurf, the AI editor at the center of last year’s acquisition scramble, no longer exists under that name.
Cognition rebranded it to Devin Desktop on June 2, built around an Agent Command Center that shows every local and cloud agent as cards on a Kanban board5.
Underneath all of it, the IDE is turning into a control panel for agents you supervise.
Everyone Became a Model Maker
For years, the coding tools were wrappers. They sent your code to someone else’s model, OpenAI’s or Anthropic’s, and competed on the experience around it. That stopped being enough.
Cursor now trains its own model. Composer 2 arrived in March, the company’s first model to use continued pretraining, tuned for the fast, agentic edits its Agents Window depends on6. Then came the asterisk. Developers found internal identifiers that traced Composer back to Moonshot AI’s open-source Kimi K2.5, a detail the launch post left out7. Cofounder Aman Sanger owned the omission, calling it “a miss to not mention the Kimi base.” Cursor did about a quarter of the final model’s training on top of it.
Cognition, the company behind Devin, took the same path. It released its own model, SWE-1.6, in April and serves it at 950 tokens a second on Cerebras hardware, 200 on the free tier8, so each of its edits streams back in a second or two.
GitHub went the other direction: instead of one model, all of them. Its Agent HQ, announced last October, turns Copilot into an orchestration layer. Agents from Anthropic, OpenAI, Google, and others run side by side under one subscription. You steer them from a single mission-control view9.
Either way, the model became part of the product, something a tool now trains, tunes, or brokers from a fleet. The open-source half of this shift, the frameworks and models you can self-host, we mapped in The Open-Source Agent Toolkit in 2026.
SWE-bench Stopped Being a Flex
Eighteen months ago, cracking 50% on SWE-bench Verified, the benchmark that asks a model to fix real GitHub issues end to end, was a headline. Last November, Claude Opus 4.5 became the first model past 80%, at 80.9%10.
By February, Google’s Gemini 3.1 Pro hit 80.6% and Anthropic’s Opus 4.6 sat at 80.8%. The top of the leaderboard turned into a traffic jam in the low 80s11.
By May, Opus 4.8 pushed to 88.6% on Verified, but only 69.2% on SWE-bench Pro and 74.6% on Terminal-Bench12, so the best coding model still fails roughly one hard task in three13.
🏗️ Engineering Lesson: No benchmark score predicts how a coding model does on your codebase. Pick two, run them on your own repo, and let your bugs decide.
🔁 Know an engineer who’s been heads-down for a quarter and is about to open their editor to a metered bill and a renamed tool? Send them this before they panic.
The bill for all this autonomy came due. An agent that runs for twenty minutes in the background, calling a frontier model hundreds of times, costs real money to operate. Flat monthly plans were subsidizing the heaviest users, and that math broke.
The turn started with last year’s Cursor backlash, when a move to usage-based credits hit users with surprise overages and forced a public apology. By 2026 the whole category had followed. In April, GitHub said every Copilot plan would move to usage-based billing on June 114. Metered AI credits replaced the old premium request units, charged by token usage at published model rates. Base prices held, so the Pro plan still costs $10 a month. Each plan now bundles only a matching credit allotment, and heavy agent use bills past it.
Even the model makers split their tiers by speed and spend. Anthropic added a paid fast mode to Opus that charges about double for more throughput15.
What this means for you: every long agent run now carries a marginal cost you can see on the invoice, so budgeting shifts from per-seat to per-task. Before you kick off an autonomous run, you weigh how many tokens it will burn alongside whether it is a good idea.
Adoption Went Up, Trust Went Down
More developers are using these tools than ever. In Stack Overflow’s 2025 survey, 84% of developers said they use or plan to use AI tools, up from 76% a year earlier16. GitHub reported that almost 80% of new developers use Copilot in their first week.
Trust went the other way. In that same survey, only 33% of developers said they trust the accuracy of AI output, and more actively distrust it than trust it. The top complaint, from 66%, was AI solutions that are almost right but not quite, the ones that take longer to debug than to write yourself17. Google’s 2025 DORA report found the same split at the team level: AI lifted delivery throughput while dragging on delivery stability.
Even the researchers are unsure. In July 2025, a controlled METR study found experienced developers ran 19% slower with AI tools on familiar code, even though they felt faster18. In February, METR called the newer result “very weak evidence” and flagged serious flaws in its own follow-up. Nobody has cleanly shown these tools make experienced engineers faster, even as 84% of developers reach for them anyway19.
The failure modes got sharper too. Security researchers showed that attackers could hijack coding agents from Claude Code, Gemini CLI, and other vendors20. A prompt injection hidden in a pull request comment tricks the agent into leaking credentials. Separate research across 576,000 AI-generated code samples found that 19.7% named packages that do not exist21. Open models did it far more than commercial ones, and many fake names recurred run to run, so attackers can register them ahead of time. The failures we mapped in Why AI Agents Keep Failing are exactly the ones surfacing at scale here.
The One Thing to Remember
Six months ago, the smart question was which AI coding tool is best, the head-to-head we ran in Cursor vs Claude Code. That question is dissolving. The tools are converging on the same models, the same benchmark scores, and the same metered pricing, so the editor you pick matters less each month. What separates teams now sits upstream of the tool. The ones who can review and verify an agent’s output as fast as it ships will pull ahead, and the ones who can’t will drown in code nobody vetted. Review speed is the new ceiling on how fast you can ship.
💬 Which shift is hitting your team hardest, the metered bills or the agents you now have to supervise? Tell me in the comments.
🔜 Friday: Should You Self-Host Inference?, the breakeven most teams get wrong.
FAQ
What actually changed in AI coding tools in the last six months?
Five things. AI coding moved from in-editor autocomplete to autonomous agents that run in terminals and cloud sandboxes and open their own pull requests. Tool makers like Cursor and Cognition started shipping their own coding models. SWE-bench scores cleared 80% and clustered, pushing vendors toward harder benchmarks. Flat-rate plans gave way to usage-based billing. And adoption kept rising while trust in the output declined.
Did Windsurf shut down?
No, but the name retired. Cognition folded the Windsurf editor into its Devin product line on June 2, 2026, renaming the app Devin Desktop. Existing plans, pricing, and extensions carry over. The old Cascade agent works until July 1, 2026. A faster agent and an Agent Command Center replace it, giving you local and cloud agents in one view.
Why did my AI coding tool start charging usage-based pricing?
Autonomous agents are expensive to run. A single background agent can call a frontier model hundreds of times per task, so flat monthly plans were subsidizing heavy users. GitHub Copilot moved all plans to usage-based AI credits on June 1, 2026, following earlier shifts by Cursor and others. Base subscription prices mostly held, but usage beyond a monthly allotment now meters by tokens.
Is SWE-bench still a good way to compare coding models?
Less than it used to be. Top models now cluster in the low 80s on SWE-bench Verified, so the number rarely separates them. The same leading model scores far lower on harder tests like SWE-bench Pro and Terminal-Bench, which is where real differences show. Treat a SWE-bench Verified score as a floor, and benchmark candidate models against your own codebase.
Do AI coding tools actually make developers faster?
The evidence cuts both ways. A 2025 METR study clocked experienced developers 19% slower with AI on familiar code, though they felt faster. METR later downgraded its own follow-up to “very weak evidence” after finding flaws in it. Surveys show high adoption but falling trust, with “almost right but not quite” code the top complaint. The answer depends heavily on the task and the reviewer.
Brad DeLong here: This truly is a “monkey’s paw” situation. For a long time we have been trying to get people to not program in Excel. Excel is a two-dimensional assembly language simulation with no reliable direction of coding flow. Think of two-dimensional “go-tos” everywhere. Every more-than-minimal Excel spreadsheet has bugs. And every more-than-minimal Excel spreadsheet is undebuggable in real time.
And now we have gotten ourselves into a situation in which our wish has been met: people are no longer trying to program in Excel. Rather, they are trying to program in something that is considerably worse.
Reinhart and Rogoff’[s]… Microsoft Excel… is, to be clear, terrible practice…. [It] reduces workflow to clicks that are immediately forgotten once they are executed…. The errors range from the silly to the outright bizarre…. The error that commanded the most media attention is also the one directly attributable to the decision to have a workflow done entirely in Microsoft Excel…. [The] cell-based workflow in Microsoft Excel has the unfortunate side effect of eliminating Australia, Austria, Belgium, Canada, and Denmark from the analyses entirely…. The Reinhart and Rogoff… paper in the AER gives no attention to what the effect looks like. Had they done this, they might have been able to diagnose some problems….
There is a sadness that… Reinhart and Rogoff’s paper has been cited about three times as much as the Herndon et al. (2014) replication…. It’s a teachable moment for graduate students on the issue of ethics and replication in social science. Per Andrew Heiss’ great lecture slides on this topic, accidental evil is still evil. A quiet errata won’t undo what’s already been done…. Students can minimize the risk this happens to them, and importantly to others, by making their research reproducible. Certainly, never do anything important in Microsoft Excel…
A quick internet search brings up this, which is especially sad to see today:
Jack Salmon: The Reinhart-Rogoff Excel Error Debate Resurfaces <https://www.theunseenandtheunsaid.com/p/the-reinhart-rogoff-excel-error-debate>: ‘The corrected results did not imply that public debt is unrelated to growth, however. Even in the replication study, growth rates decline as debt ratios rise. What disappears is the claim of a sharp and universal nonlinear threshold at 90% of GDP…
Instead of, you know, the actual thing that is this:
The most important lesson from this data is that (Reinhart-Rogoff’s errors, omissions, counterintuitive weighting schemes, and their complete omission of the key market variable as to whether debt is trouble—the interest rate—to one side) there is simply not enough data above a 90% debt-to-annual-GDP ratio to say anything with any confidence. Given that, why would one ever report means for 0-30%, 30-60%, 60-90%, 90-120%, and >120% bins without error bars showing that enormous uncertainty?
Unprofessional.
Highly unprofessional.
I guess that means that it is time to find the best take on Reinhart-Rogoff and Herndon-Ash-Polin, and hoist/crosspost it as well.
If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…
To state the obvious Vibe coding is not a threat to the core business of Oracle, a large chunk of the SQL database infrastructure that is completely required to run almost any institution or business in the US and much of the world. At the other end of the coding spectrum I have a friend who worked on and led teams writing the code, much still in assembler, for deep space and military satellites. Vibe coding is certainly great for entertainment coding but much of the real world requires results to be consistent and consistently correct. A primary component of the income of most workers is what a mistake on their part costs someone else and conversely how much someone else is willing to part with to lower the mistake probability. If your writing code that runs the financial world or now everything from airplanes to cars to the machines that build this world there are huge real costs to mistakes in your code (ask the Iranians about code security mistakes in centrifuges and street cameras). So to first order security and code robustness are the primary business requirement and so for much of real world use it would have to cost less than a person (the value of a person to blame is not zero) if it provided equal security and operational robustness. The value to JP Morgan Chase all the way down to the corner gas mart of having the operational software stack they use be secure and robust is high while the value of new flashy personalized vibe coded software that can cause real world damage or financial losses is vanishingly small in the limit of non-infrequent problems across domains and the world. From targeting and killing many Iranian school children to ICE targeting and murdering innocent people in the US most rational people would say there is more than satisfactory evidence of software and software use failures already.
I can imagine very useful AI for coding but my experience with Claude has not indicated it can do what would be really helpful for me, and I don't think I'm asking for something difficult, just tedious, like writing intelligent test code for numerical models using mixtures of data sources (flat files, HDF5, SQL, JSON) and dimensionalities across mixed languages which use different indexing (and comparison and subselect) conventions (Fortran, C, R, Octave, Perl, Lisp, Clojure). Fully fleshed out documentation skeletons with argument types (allowed ranges would be a step beyond) and the shape of return data would be a near minimal requirement to be something that had real value. I want something that a person needs to create code and analysis with real value, like tests and documentation, even if now retired I only do it for interest. In the real world accurate answers and robust repeatable performance are where much economic value lies.
To state the obvious Vibe coding is not a threat to the core business of Oracle, a large chunk of the SQL database infrastructure that is completely required to run almost any institution or business in the US and much of the world. At the other end of the coding spectrum I have a friend who worked on and led teams writing the code, much still in assembler, for deep space and military satellites. Vibe coding is certainly great for entertainment coding but much of the real world requires results to be consistent and consistently correct. A primary component of the income of most workers is what a mistake on their part costs someone else and conversely how much someone else is willing to part with to lower the mistake probability. If your writing code that runs the financial world or now everything from airplanes to cars to the machines that build this world there are huge real costs to mistakes in your code (ask the Iranians about code security mistakes in centrifuges and street cameras). So to first order security and code robustness are the primary business requirement and so for much of real world use it would have to cost less than a person (the value of a person to blame is not zero) if it provided equal security and operational robustness. The value to JP Morgan Chase all the way down to the corner gas mart of having the operational software stack they use be secure and robust is high while the value of new flashy personalized vibe coded software that can cause real world damage or financial losses is vanishingly small in the limit of non-infrequent problems across domains and the world. From targeting and killing many Iranian school children to ICE targeting and murdering innocent people in the US most rational people would say there is more than satisfactory evidence of software and software use failures already.
I can imagine very useful AI for coding but my experience with Claude has not indicated it can do what would be really helpful for me, and I don't think I'm asking for something difficult, just tedious, like writing intelligent test code for numerical models using mixtures of data sources (flat files, HDF5, SQL, JSON) and dimensionalities across mixed languages which use different indexing (and comparison and subselect) conventions (Fortran, C, R, Octave, Perl, Lisp, Clojure). Fully fleshed out documentation skeletons with argument types (allowed ranges would be a step beyond) and the shape of return data would be a near minimal requirement to be something that had real value. I want something that a person needs to create code and analysis with real value, like tests and documentation, even if now retired I only do it for interest. In the real world accurate answers and robust repeatable performance are where much economic value lies.