Skip to content
Vale, home

Let the Agents Cook. Make Them Prove It.

13 min read

On Lauren Tan’s 2,500 PRs, the engineering job description that’s quietly changing, and why the most interesting AI breakthrough might be a compiler that knows when to say no.

I had to reread the number: 2,500 pull requests merged in a single month. When I came across Lauren Tan (poteto)‘s account of what she’d been building, and then her conversation with Matt Pocock about the setup behind it, I was genuinely amazed. 😅 I’ve been increasingly fascinated by agentic workflows myself, but seeing someone push the idea this far made me wonder whether we’re getting a glimpse of what software engineering might look like in a few years, once orchestration and verification become just as central to our work as implementation.

I’ve never considered the number of merged PRs a particularly useful measure of engineering quality on its own, since it tells you very little about the nature of the work or the value it ultimately creates. But the scale of what Lauren describes is remarkable, and what caught my attention was how deliberately she’s approached the problem of maintaining quality while increasing throughput. If agents can help us ship more and make the resulting software more reliable, I’m absolutely here for it. I wanted to understand how she’d built enough confidence in her process to allow that many changes to move forward.

Lauren Tan and Matt Pocock discuss her agentic engineering setup

what makes that level of autonomy possible?

An agent can produce a plausible implementation remarkably quickly these days, but there’s quite a difference between seeing something work in a chat window and knowing it’s safe to merge into a codebase that other people depend on. A change may pass its own tests and still break a neighboring feature, introduce an architectural inconsistency, or quietly make something harder to maintain. At scale, the question becomes how you detect those problems without personally scrutinizing every line of every diff.

That’s where Lauren’s setup becomes particularly interesting. She combines Grok Bot, Cursor projects and cloud agents with pstack, a collection of engineering playbooks that spells out how agents should investigate, implement, test, review and ship changes. There’s no reason everyone should reproduce her exact toolchain, but the separation of responsibilities is worth studying.

For instance, in the full-autopilot playbook, an agent owns an individual PR while separate verifiers examine the committed revision and run checks against the actual behavior. The merge requires a clean verdict applying to the relevant patch, and a material change can invalidate earlier verification. I find that last detail especially important because evidence about one version of the code doesn’t automatically tell you anything about the next.

The pstack README makes a point of prioritizing code quality over sheer output, which helps explain why Lauren can make a credible case for parallelism in the first place. You start by creating a process capable of catching mistakes, learn which kinds of work it handles reliably, and only then give more agents enough independence to operate concurrently. I find the prospect of scaling that kind of engineering discipline much more appealing than simply multiplying the number of coding sessions.

what happens to the engineer?

I lead a web engineering team while still spending a substantial part of my time building things myself, so I find this transition fascinating from a very practical perspective. On a typical day I’m already moving between implementation details, architecture, prioritization and product decisions, sometimes within the same hour, and I can imagine that mix becoming increasingly common even among engineers who don’t formally manage anyone.

You start thinking a little more like an orchestrator: which work can safely happen in parallel, where do dependencies make that a terrible idea, what evidence would convince you a change is correct, and when is the right decision to stop working on a feature altogether? There’s a product-management dimension to those questions, although I don’t think that makes software engineers interchangeable with PMs. The technical understanding needed to define meaningful constraints, recognize subtle failures, and decide how to partition a system remains extremely important.

I touched on this in Agentic Engineering Is Not a Vibe, where I was thinking about the move from writing implementations to owning architecture and correctness. Lauren’s workflow has made me think more about what comes afterwards, when we begin designing the processes that produce and verify whole sequences of changes. It’s a different kind of engineering work, and one I suspect we’re all going to have to learn rather than assume we’re naturally good at it.

learning to trust without becoming careless

The more autonomy I’d like to give an agent, the more I need to understand the conditions under which I can actually trust its work. An LLM’s confidence is a fairly terrible substitute for a test result, and asking it to be careful doesn’t establish a meaningful boundary. I’d rather give it an explicit definition of done, the permissions appropriate to the task, a safe place to work, and a way to fail or roll back without dragging everything else down with it.

I got a rather memorable lesson in this when My AI Agent Installed Malware on My Server (And Then Caught Itself Doing It). The agent’s autonomy was genuinely useful, but it also expanded the consequences of one poor decision. After that experience, I became much more interested in the design of the environment surrounding an agent, especially where permissions, independent inspection and recovery can limit what goes wrong.

The pstack guide to unattended runs describes a sensible progression: establish how a task should be verified while you’re still supervising it, make those checks repeatable, inspect the agent’s failures, and gradually extend its independence. I like that there’s no pretense of a magical moment when an agent becomes trustworthy forever. Trust has to be earned in relation to a particular task and set of constraints, and reconsidered as either changes.

🔒 Autonomy becomes much easier to grant when verification is part of the workflow itself, rather than something we hope to find time for after the agent finishes.

perhaps our definition of good code needs another look

Lauren made another observation in a recent post on X that I found particularly thought-provoking. We’ve spent a lot of time worrying about the imperfect code AI might introduce, but our codebases were hardly pristine before agents arrived. Software changes continuously, and the implementation that made sense six months ago may no longer be the one we would choose today.

I think there’s a lot of truth to that. I’ve certainly worked with code that could have been more elegant, or better structured, yet still did exactly what the product needed at the time. Sometimes you accept a compromise because you need to learn from real users before investing heavily in an abstraction, or because the cost of eliminating an unlikely failure outweighs the harm it would cause. None of that means standards are unimportant. It means that engineering quality has always been contextual, and deciding where to spend effort is part of the job.

Lauren proposes an intentionally demanding ideal of code that is correct, bug-free and efficient to run, while acknowledging how expensive achieving all those qualities has traditionally been. I wouldn’t interpret that as a universal checklist. Correctness depends on a specification, performance has to be evaluated against actual constraints, and a sufficiently complex, evolving application rarely admits a convincing blanket claim of being bug-free. Still, the idea leads somewhere interesting: what if agents could reduce the cost of improving reliability enough that some compromises we’d previously accepted no longer made sense?

That’s where my optimism comes from. I don’t expect every website or internal dashboard to receive the same assurance process as safety-critical software, nor would I want the pursuit of theoretical perfection to prevent teams from experimenting and learning. But if the repetitive work of implementing changes becomes cheaper, perhaps we’ll have more opportunities to test the awkward edge cases, simplify an unnecessarily complicated design, or revisit something we’d always meant to improve. It’s an appealing possibility, provided we use the additional capacity deliberately rather than filling it with more work simply because we can.

Lauren calls learning to work this way a new craft, and I quite like that expression. What interests me most is the technical and organizational judgment required to make the gains real. And one experiment in particular makes me wonder how far we could take the idea of making correctness cheaper.

Bend, and the possibility of making some guarantees explicit

I’ve been fascinated by Bend 2, a young programming language that combines an ambition for fast execution with the ability to express formal properties and check proofs about them. The part that interests me most in the context of agents is its approach to making certain requirements much less ambiguous.

Natural language is convenient because I can describe an intention without specifying every implementation detail, but that flexibility can also conceal assumptions I didn’t even realize I was making. Bend introduces a file called LAWS.bend for stating invariants and a PROOF.bend containing proofs that the implementation respects them. Its README gives examples such as preserving a balance invariant or preventing invalid game states.

Imagine asking an agent to improve a transaction system. “Make sure the transaction logic remains correct” is a perfectly understandable request between people who share enough context, but it doesn’t give an independent verifier very much to work with. A law specifying that the relevant balances must always sum to zero, together with a machine-checkable proof obligation, expresses one precise part of what correctness requires.

The project’s game demonstration is a lovely illustration. An agent is asked to implement wraparound movement in a game where a formal law says the player must never be able to win. That innocuous-looking feature can accidentally introduce a path to the winning state; with the invariant enforced, an acceptable implementation has to preserve the law. The important part is that the checker doesn’t have to take the agent’s word for it.

I keep thinking about what it would mean to build more repositories in which rules gradually move from tribal knowledge and vague instructions into checks we can actually execute. We’d still use ordinary tests and runtime observation for many properties, perhaps formal proofs for the ones that justify the effort, and human judgment for the aspects that can’t sensibly be reduced to a formal condition. But the agent would have much clearer boundaries within which it could explore different implementations. Bend describes the idea as AGENTS.md backed by proof, and I think it’s a particularly compelling direction.

what a proof can and cannot tell us

There’s an important distinction here, though, because I wouldn’t want my enthusiasm for formal methods to turn into a promise they can’t possibly fulfill. A proof establishes that a property follows from the specification and assumptions under which it was checked. It doesn’t tell us whether the property captures everything users expect, or whether we were asking the right question to begin with.

A requirement can be perfectly formalized and still be incomplete. Worse, if an agent is allowed to rewrite the rules whenever they become inconvenient, the entire exercise risks becoming rather elaborate theater. The soundness of the checking system matters too, including how source code is translated and what the runtime actually executes.

Bend is very open about being young and about the limitations of its toolchain. Its documentation describes an optional --verdict path using a proven kernel, while also acknowledging that other compiler components and the translation into that kernel are not fully proved, along with substantial gaps in the current language ecosystem. That’s an important distinction between an intriguing architectural direction and something I’d be comfortable recommending as a universal solution for production systems today.

I can see enormous value in formalizing particular financial or security invariants, for example, but a proof won’t establish that a new user flow is intuitive, that a design feels coherent, or that the feature was worth building. Those questions require observation, taste and product judgment, which are precisely the parts of engineering I hope we get to spend more time on as implementation becomes less laborious.

a codebase that remembers why

There’s also an organizational consequence to all this that I find almost as interesting as the tooling. So much of what we know about a codebase is distributed across old PRs, conversations, half-maintained documents, and whichever engineer still remembers why an apparently awkward abstraction was introduced. I encounter this constantly when helping other people navigate existing systems.

An agent can’t reliably infer the whole history behind a decision from the current code alone. If we’re going to ask it to make more decisions independently, it would help enormously to make the constraints and reasoning behind them easier to discover, whether that’s through architectural rules, tests that explain otherwise surprising behavior, or explicit policies for which changes require closer review.

I like how pstack approaches repeated corrections: rather than repeatedly telling an agent not to make the same mistake, try to encode the lesson in an enforceable part of the environment. A type can make an invalid state unrepresentable; a regression test can preserve something we learned painfully; a formal invariant can protect a particularly important property. None of these replaces context or judgment, but together they make that judgment available to the next engineer or agent who touches the code.

I’d love to see repositories become better at carrying their own rationale in this way. It would make them easier for humans to maintain, too, which seems like a rather good outcome regardless of where agentic tooling goes next.

measuring progress without mistaking activity for value

This brings me back to those 2,500 PRs. The number caught my attention and I’m still impressed by the system required to coordinate that work, but I wouldn’t want any team to turn it into a target. PR counts have never told us enough about quality, difficulty, or net impact, and autonomous agents make it easier than ever to generate activity that looks productive on a dashboard.

What I’d actually want to know is whether useful changes reach users sooner, whether defects become less frequent or easier to detect, and whether the code remains understandable as it evolves. The 2025 DORA research is useful context here because it treats AI as an amplifier of the engineering environment around it. A team with poor feedback loops doesn’t acquire good ones merely by producing changes faster.

It would be genuinely exciting if the extra throughput came with more reliable software and less repetitive work for the people maintaining it. We should measure that outcome rather than assuming it’s been achieved because a counter increased.

the kind of engineering I’d like to do more of

I find the prediction that developers will simply become PMs supervising a fleet of bots a little too simplistic, partly because it overlooks the engineering depth involved in determining whether work is correct and partly because it makes the future sound rather boring. 😅 What I can imagine is a role in which we still understand systems deeply but spend a larger share of our attention defining meaningful problems, choosing which guarantees matter, shaping how different pieces of work interact, and evaluating the outcomes.

There’s a difficult educational question attached to that vision. Much of the judgment I rely on today came from actually implementing things, debugging failures, and watching seemingly reasonable choices become problems a few months later. How do we help newer engineers develop that intuition if agents increasingly handle the implementation? It’s not enough to become excellent at supervising a process if you can’t recognize when the process is confidently producing the wrong thing.

I don’t have an especially neat answer to that yet, although I suspect learning to work with agents responsibly will require knowing when not to delegate, particularly while you’re still developing an understanding of the underlying systems.

Anyhow, what stays with me from Lauren’s setup and the work happening around tools like Bend is the possibility of making engineering both more ambitious and more disciplined. I’d love to spend less time on repetitive implementation while becoming better at articulating what a system is supposed to do, what it must never do, and why any of it matters to the people using it. That sounds like a fascinating direction for our profession, and one I’d be very happy to keep exploring.

things worth reading (and watching)

← all posts