Bios Sphere Talk

There’s a lot of talk about how we can make better agents and design a system that fits our wishes better. But these agents aren’t exactly reliable. If they were, software engineers would’ve been replaced already. It seems like math has been solved, but we still have to check agent outputs in code reviews. You can ask other agents to review, but these clankers still output slop and junk. But we’re not going back to the stone age where we code by hand. So how do we maintain quality when we write none of the code?

To find out, I went out looking for the hardest problems that I could find. The issues were on github, and PRs needed to be reviewed by maintainers on real projects. Could coding agents pass the quality bar in open source?

Experiment Setup:

I set up a coding factory that scans Github for other peoples’ issues, fix them, then open PRs. It would find an appropriately complex issue, solve it, then make a PR. I was tokenmaxxing.

I didn’t really know how Open Source worked, so this is how my first PRs came across. I was asking maintainers to trust that my code is legit because it works! It’s their job to do the rest of the work after “I” write the code, right? No. If you are throwing stuff like this over the fence and hoping for the best, stop. Get some help.

This approach doesn’t work because these PRs are hard to review. It’s a net negative contribution on your team. The merge rate on these shotgun PRs were about 40%.

It turns out that sometimes the code is wrong! So I set up a code review agent that checked over the production agent. I wanted to hop onto the future of software. Until the review agents started messing up, too. I mandated that tests fail before on main, and pass on branch. But it turns out that the review agents never ran the tests! So I set up another agent who was supposed to do software QA, even using checklists. I even asserted that it produce a hash of the test output as a part of its run. But sometimes it would fabricate the hash out of nowhere, just to pass the deterministic gate that I set up as a part of the pipe.

Yes, my software factory was catching more mistakes but I could never really be sure if they were fully reliable. I am babysitting these agents. What was I doing wrong? Could I ever trust these ephemeral coding agents who pat themselves on the back when they think they’re done? Is there any way to put them to work without needing to trust them?

Agents confidently lie, cheat, make mistakes, then disappear. They will never be fully deterministic genies that do the job perfectly. The only way to get reliable results is to rely on the results and not the agent. This was the lesson: the work should stand on its own. Demand that your agent produce evidence and proof of its work rather than a confidence score. Confidence scores are lies.

What’s the difference between trust and evidence? Let’s talk about it.

Trust is fragile because it’s only as strong as its weakest link. You might trust your coworkers, but do you trust their trust in agents? Trust decays. And when stuff breaks in prod, investigations have to be repeated and attribution turns into a blame game.

When you have evidence, you no longer have a chain of attestations. If anybody can verify it independently, it doesn’t matter who said it, so you can leave them out of the picture. If your agents can’t provide evidence to back up their claims, it’s worse than useless.

So what is evidence?

Evidence is a tuple of a claim and its procedure. You need both. A claim without verification steps is untrue, and a procedure without a claim is meaningless. When a claim is presented, it must be falsifiable and verifiable. Think of the scientific method and why it works better than a vibe check. It includes a procedure for reproducibility so that the conclusion can be drawn if the same procedures are followed. I call this verifiable knowledge.

Here’s an example of verifiable knowledge. I got over 100 PRs merged to other peoples’ github repos since May. You can either believe me or ask your agent to verify. My handle kimjune01. Github has an API that your agent can query. Your agent can help you write this query and you can copy paste it into a terminal so that there is no agent involved when it’s run.

Before you checked it, you didn’t know for sure whether my claim is true or not. It’s true to me, but that didn’t matter to you. I’m just a guy talking on stage. Knowledge is subjective. My job as a stranger is to give you the means to check for yourself. Just like me, your agent is a stranger, too. You should ask your agent to provide a procedure to verify its work independently, to assemble the evidence tuple. That’s how you know your agent is done. Never trust your agents. They don’t care about you.

So hopefully by now I’ve convinced you that evidence is a better measure of agent output than a vibe check. Since I adopted it, my merge rate has increased to about 80%. For evidence, check OpenHands Github repo. They are the most AI-forward open source repo that I know of. Every one of their PRs contains both the claim and its verification procedure.

You may think that this is common sense, like how we learned the scientific method in high school. But providing verifiable knowledge with every PR is a bar raiser for the whole team. When you expect evidence from each other, you’ll have higher quality and less rework. You will make mistakes easier to catch, and merge fewer mistakes to prod.

So what did we learn? Instead of trusting your agents, make them produce results that are good enough to trust on its own. Our human role in the loop is better allocated to picking the right problems to solve.

I wish I could talk to you more about agents, open source, epistemology and methodology, but I’d rather save time for questions.