Proved, but Not Yet Understood
On the night OpenAI put its mathematics repository on GitHub, the complexity theorist Dana Moshkovitz texted her husband about the manuscript claiming to prove the Unique Games Conjecture, a statement he says she has worked toward for as long as he’s known her. “It feels like something written by someone who’s on psychedelics,” she wrote. 😅 Her husband is Scott Aaronson, who quoted those texts in his post “The Mathocalypse”, just below a line I had to read a few times: the UGC manuscript comes with a Lean certificate, so a machine has checked the formal proof, and yet “it also appears that no human has understood just about any of these proofs yet; the race to do so has just started.”
In my last post, Let the Agents Cook. Make Them Prove It., I argued that verification is what makes it reasonable to let agents do more on their own, and I still believe that. Here, though, verification had apparently worked, and the people who cared most about the result were still standing outside it. So this post is a qualification rather than a retraction, since a check can justify trusting an output without creating the understanding you need to extend or maintain it (and in software, where someone has to change the code again next quarter, that understanding is a big part of the job).
the short version of the release
On October 6, OpenAI announced “a broad range of new mathematical results produced by an internal frontier model” (unreleased), published as a GitHub repository of 719 manuscripts organized into 372 “families” of related results. By OpenAI’s account the average result used roughly three hours of ChatGPT Pro thinking compute, and after an October 7 update about 42% of top-line results are formalized in Lean (300 of 719, all self-reported and none of it peer-reviewed). One day later the history file already recorded three withdrawals because “a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers,” which anyone who has watched one bad utility function quietly break three services downstream will recognize.
A week earlier the Advisory Group on Mathematics and Artificial Intelligence, nine independent mathematicians including Timothy Gowers, Martin Hairer and Edward Witten, had published guidelines for the responsible release of AI-generated mathematics, and on release day the group issued a carefully neutral statement calling it “the beginning, not the completion, of the process of human understanding.”
two different ledgers
I keep coming back to the process of human understanding, because it names something the word “verified” tends to swallow. A Lean certificate answers a narrow and extremely valuable question, namely whether this formal statement follows from these definitions and axioms according to this kernel, and even that answer depends on the formal statement being the right one. In a guest post on Terence Tao’s blog, Thomas Hales argues that a Lean proof “should not be accepted until a human audit is performed to ensure statement fidelity,” and a preprint by Bastounis, Circelli and Hansen argues that an autoformalized proof can follow a different argument from the paper humans are reading.
The way I think about it is that correctness and comprehension are kept on two separate ledgers, and a certificate only writes to the first one, while most of the value of a new result sits on the second (comprehension is what lets you reuse an idea, or see where it breaks if you change a hypothesis). The advisory group’s guidelines state the old norm plainly: “the authors of a paper should understand the mathematical argument of that paper, have verified its correctness themselves, and take full responsibility for the content.” Read that with a pull request in mind 🤔 and you’ll notice that understanding, verification and responsibility have always travelled together in the person of the author, so we rarely had to name them separately, while machine authorship pulls them apart.
the same bottleneck, in our repos
We’ve been running a version of this experiment in software for a while, just without the vocabulary. Harvard economists Fiona Chen and James Stratton used Jellyfish engineering analytics covering roughly 300 million work events across 718 firms, from January 2021 through March 2026, and their paper (nicely covered by Ars Technica) finds that agents raise lines of code by 30%, commits by 20% and pull requests by 23%, while resolved Jira issues and epics, the authors’ proxy for shipped software, show small, statistically insignificant changes. Meanwhile the time from opening a pull request to merging it rises 49%, the share of PRs where a reviewer formally requests changes nearly doubles, and comments per PR rise 35%.
I lead a web engineering team and still build a lot myself, so the review queue isn’t an abstraction for me (I touched on it in Agentic Engineering Is Not a Vibe). Code review is the moment a second person builds enough of a mental model of a change to share responsibility for it, which is comprehension work, and it doesn’t get faster because the code was cheaper to write. An agent’s PR also tends to arrive without the thing a human author brings for free, someone who can answer why, so the reviewer either reconstructs the reason from the diff or re-prompts and hopes to get the reason rather than a plausible story about it.
I suspect that reconstruction is where a good share of the extra review time goes, but that’s my hypothesis, not the study’s finding. The same numbers could come from larger changes, from reviewers stretched across more of them, or from teams raising their standards. The authors themselves can’t fully separate those from review getting harder, “either because the code written now is of worse quality, or because the standards of code review have changed.” It’s also a job market paper, not yet peer-reviewed, measuring firm-level adoption with data that stop before the latest generation of agents. So it’s consistent with a comprehension bottleneck without proving one, and what it does show fairly convincingly is that the cost moved downstream, to wherever humans have to understand things.
a quarrel about who sets the norms
The sharpest reaction to the math release came from the Association for Human Mathematics, whose statement on the AHM site was reposted as a guest post on Tao’s blog (which he hosted but didn’t write, a distinction plenty of people missed). It calls the release “not a demonstration of scholarship, but a demonstration of power” and urges mathematicians to stop working with OpenAI. I don’t think a boycott helps, since it doesn’t un-prove anything, but the complaint is fairly precise in economic terms: each result cost about three hours of compute, absorbing them will cost the receiving community years of skilled human attention, and the producer set the schedule. Cheap to produce and expensive to absorb is exactly the asymmetry sitting in our review queues.
The contrast Aaronson draws next is, for me, the most interesting part of his post. A day before OpenAI’s release, Josh Alman and Virginia Vassilevska Williams posted a preprint refuting two long-standing hypotheses of fine-grained complexity, and the paper says Claude discovered the core algorithm in a single session of 16 million output tokens “with no human input,” after which the authors “worked to understand, simplify, strengthen, and extend the algorithm,” wrote the paper and “take full responsibility” for it. Aaronson calls these the “OpenAI model” and the “Anthropic model” and is fair to both. The first “sets up a crazy race among humans to digest and explain a messy AI proof,” while the second “puts a private company in the position of picking and choosing which human mathematicians get to be the emissaries of the AI.” In the terms of this post, one is verification-first and the other comprehension-first, and I don’t think either is obviously right (it bothers me a little that both were chosen by companies rather than by the communities that have to live with them).
the firehose, with patches attached
Security makes the pattern clearer, because there the receiving community asked for the firehose. On October 8, Anthropic launched OSS Scanner, an opt-in service whose vulnerability reports are “fully model-generated, without human review or triage,” and maintainers kept asking for everything anyway. An independent analysis by Vulnerabilities.ai of Anthropic’s own dashboard counts 6,157 findings disclosed to maintainers as of October 2, about 78% of them sent without independent review, and 516 patched upstream (which tells me how much of the output became changed code, not how much of it anyone understood).
And here’s the part I can’t stop turning over: in security it’s OpenAI running the curated model, since its Patch the Planet initiative has Trail of Bits researchers “reproduce the evidence” and “reassess severity” before maintainers see anything, while Anthropic is the one sending raw output. Whatever is driving these choices, then, it isn’t a stable corporate philosophy about openness. 🤔
My tentative explanation is the distance between the new artifact and what the receiver already understands. A vulnerability report with a working reproducer lands in a codebase the maintainer knows intimately (OpenSSL’s Anton Arapov says in Anthropic’s post that a real exploit attached is “basically job done for an engineer as you can verify it right away”), whereas a long proof built on what Moshkovitz called “some alien craziness” lands somewhere nobody has a model at all. The Lean certificate plays the role of the reproducer for correctness, but there’s no reproducer for understanding, which is why the same volume of output can feel like a gift to a maintainer and a tax to a mathematician.
where this argument is weakest
The strongest objection is that we already depend constantly on things nobody understands in full. Nobody reads compiler output, the four colour theorem was accepted on the strength of a computer check few mathematicians follow line by line, and any node_modules folder is a monument to trust in strangers. I think that’s right, and it sharpens the claim more than it kills it, since unread code is fine when there’s an understood boundary around it and a responsible party behind it (and in a PR that touches your own business logic, there’s no one else’s boundary to hide behind).
There’s also an asymmetry with mathematics. A true theorem stays true, so a community can afford to bank an unread proof and come back to it in ten years, while software rots and someone has to modify it next quarter with a working model of why it is the way it is (so in this narrow sense software needs comprehension more urgently than mathematics does, even though our artifacts are a good deal less profound 😅).
The other serious objection is that the bottleneck may be temporary. Moshkovitz spent the next day asking an AI model to explain the UGC construction, and within roughly a day Aaronson reported that she “now mostly understands the proof” and wants to give talks about it, which is a remarkable turnaround. But a commenter on the same thread wondered how much of the clarity people perceive when they press a model for explanations “is just their own informational contribution reflected back at them.” That stuck with me, since an explanation can help you comprehend without comprehending on your behalf.
norms worth borrowing
What I find really useful in the math community’s response is that it’s trying to write the norms down, which software mostly hasn’t. The advisory group asks producers to have “the model, or some other model” write each proof readably and to disclose prompts, compute and formalization status, and translated into engineering that gives me three questions any team merging agent PRs should answer for itself (my answers are proposals worth trying, not settled practice).
What evidence should come with an agent’s PR? Making a change reviewable should count as part of the agent’s job, so I’d want the intent in the reviewer’s terms, a ledger of what was machine-checked and against which revision (and, just as explicitly, what wasn’t), a reproducer where one makes sense, the dead ends it abandoned, and a named human who has read all of that and can answer why.
Who needs to understand the change, and how deeply? Not everyone, and not always all the way down. Every reviewer needs boundary depth, meaning what the change does, which invariants it relies on and what the checks cover, while anything headed for the core of the system also needs one person at modification depth, able to change it next quarter without the agent and explain it to a teammate without opening the transcript. A generated API client probably needs only the first and payment logic needs both, and I’d rather a team decide which is which ahead of time than during an incident.
How should a team measure the cost? Rather than PRs opened, I’d look at reviewer hours per resolved issue and break time-to-merge down by change size and by each reviewer’s open load, since if review slows even for small changes on lightly loaded reviewers, comprehension becomes a much more plausible culprit. For maintenance I’d watch how often agent-written code is reverted or reworked within a few weeks and how many people have touched each module (a crude bus factor for understanding). None of this measures comprehension directly, only the cost of not having it, and when agent output keeps outrunning a team’s capacity to understand it, I think the honest response is to slow the output rather than thin the review.
the slow part
I still want the agents to cook, and if better checks let them produce more useful work, I’m very much in favour. What interests me is whether we can increase the amount of good work a team ships without quietly handing everyone a second job reconstructing why it works, which means treating “understandable” as part of “done” rather than something the next person has to extract from a transcript. Not every reviewer needs to understand every line, but someone should understand enough to change the thing when the requirements move.
I’d still like the extra output. I’d also like us to know what we’re getting. ✨
things worth reading
- Sharing AI progress in mathematics, OpenAI’s announcement, and the repository with its history file.
- The Mathocalypse, Scott Aaronson’s post and its very long comment thread.
- The AHM statement, as reposted (not authored) on Terence Tao’s blog, and the original.
- Responsible Release of AI-Generated Mathematics, the advisory group’s guidelines, which read surprisingly well as a code review policy.
- Chen and Stratton’s paper on bottlenecks in software production, and Ars Technica’s summary.
- Anthropic’s OSS Scanner post, read alongside Two Labs, Opposite Answers to the Same Triage Bottleneck.