Self-Improving AI, Honestly

For most of the last decade, the answer to "what is the AI product?" was a checkpoint. You train a model, you ship it, you monitor it, you retrain on some cadence. The model was the thing.

That mental model is fraying. The thing producing useful work is no longer just the model. It is the agent that wraps the model, the harness that runs the agent, the verifier that scores its output, and the feedback signal that updates either the harness, the weights, or both.

Self-improvement is the load-bearing concept under any honest argument for where this field is going. If the system can improve itself, the trajectory bends sharply. If it cannot, scaling laws and human researchers stay on the critical path.

The question worth asking is not "is self-improvement working?" The answer is yes, in narrow places. The question is: how close to the kind of self-improvement the AGI framing implies?

Self-improvement in current research has five distinct shapes. They differ in what changes and on what timescale.

Each shape answers a different question. Verbal: can the model fix its own output without changing itself? Bootstrap: can the model train on its own better outputs? Scaffold: can an outer agent rewrite the inner agent's code? Weight: can the model train at test time on feedback? Co-evolution: can both happen at once?

The 2026 SIA paper is the strongest evidence to date that co-evolution actually beats either lever alone. It posts a 36.5 percentage-point gain from harness-only to harness-plus-weights on Chinese legal classification, and a 14× CUDA-kernel speedup the harness search never reached. That is real engineering progress. It is also the moment to ask what "self-improving" actually means once the dust settles.

Read across all five categories and one thing is constant. The system improves itself with respect to a target someone else defined.

That someone is almost always a paper author. The verifier (the function that tells the system what's good) is hand-written. The dataset is hand-picked. The success metric is hand-chosen. The reward signal is engineered by a human.

The system is not figuring out what to improve. It is figuring out how to improve a thing humans pointed it at.

That distinction is the whole gap between "self-improving in a technical sense" and "self-improving in the AGI sense." The phrase is doing very different work in each. Engineers reading the research and executives reading the press release are usually working with the second interpretation while the paper is delivering the first.

Look at where these systems work. Then look at where they don't.

The pattern is clean. Self-improvement works where reward is cheap, fast, and unambiguous. It does not yet work where reward is the thing being argued about.

For an enterprise builder this is the single most important load-bearing fact. Almost every business problem worth solving lives in the second category. Customer support quality, document extraction agreement, dispute resolution, sales-call analysis, all of these have soft, contested rewards. Self-improving systems do not ship safely into that domain yet. They will sometimes look like they do, until the verifier and the underlying reality drift apart.

Across SIA, the Darwin Gödel Machine, AI Scientist, Meta-Harness, Hyperagents, the improvement mechanism itself is a frozen large language model. An LLM is choosing what to change next.

That LLM does not get better at choosing. It is the same Claude or GPT call, run again with new context. Its prior over "what to try" is whatever the foundation lab last shipped.

For genuine recursive self-improvement, the thing that decides "what to improve" has to itself be on a learning curve. None of the public systems do that. The SIA authors say so directly in their future-work section: the selection policy itself should be the object of training, not a frozen prior.

Until that exists, every "self-improving AI" system in the literature is two-level. A learnable inner loop. A frozen outer loop. The recursive open-ended improvement that the phrase culturally implies, the version that suggests an exponential trajectory, needs at least three levels with the outer one moving. Possibly more.

When two optimizers chase the same fixed reward, they don't optimize toward truth. They optimize toward whatever joint configuration maximizes the reward on the seen distribution. The SIA paper formalises this in its limitations section, and the framing applies to the whole co-evolution class.

This is not abstract. It is why a 14× speedup on a specific kernel shape can produce a 2× speedup or a regression on a slightly different shape. The system found a fixed point of a particular game. Change the game and the fixed point isn't there.

Every "self-improving" demo should be read against the question: how well does this hold up out of distribution? Most papers do not answer that question. The ones that do answer it find the fall-off is steep.

Four pieces stand between today's research and the version of self-improvement that justifies the "step toward AGI" language. None are solved.

Each of these is an active research direction. None are within striking distance of being closed. Pretending otherwise sells expectations the field cannot deliver this year, probably not next year either.

Four pieces of practical advice for anyone building agentic systems this year.

None of this is AGI strategy. It is engineering advice that maps to what the field can actually deliver in 2026, not to what the phrase "self-improving AI" tends to promise on a slide.

Self-improvement is real. Across math, code, kernels, classification, and games, the loops measurably work, sometimes spectacularly. 14× kernel speedups. Tens of percentage points on hard benchmarks. Whole categories of human iteration replaced by an outer-loop agent.

The "step toward AGI" framing is the part to push back on. The systems improving themselves today do so in narrow, well-instrumented domains, against fixed human-defined objectives, with a frozen improvement mechanism, and the resulting equilibria are fragile out of distribution.

That is engineering progress at the frontier. It is not the recursive, open-ended, self-directed improvement the AGI label calls to mind. Treat the gap between those two as load-bearing. The label travels fast. The result moves at the speed of clean verifiers and bounded outer loops.

Self-improving AI is here. Self-improving AI in the sense the phrase culturally implies is not. Both can be true at once. Pretending the gap is closing faster than it is, is the failure mode I'd watch this year.