Guardrails for Agentic Coding

A note on where this comes from: none of this is original thinking assembled in a vacuum. Vibe Coding, by Gene Kim and Steve Yegge, is what first shifted me from wary of AI-assisted coding to convinced it's worth doing right. The shape of "guardrails" specifically came from Gary Tan's gstack and from Spencer Burleigh's writing on treating AI-assisted coding as a governance problem, not a prompting problem, including the swiss-cheese framing from aviation that runs through a lot of this post: independent gates, different blind spots, so a miss in one gets caught by another.  Matt Pocock’s talk was also a heavy influence on this thinking. Beyond those, this setup has been shaped by conversations, other people’s posts, and things I read once and internalized without remembering the source. If something here looks familiar and I didn’t credit it, that’s a gap in my memory, not an intent to claim it. Tell me and I’ll fix it.

To start, here is a (kind of) vanity metric: I’ve been able to peak at 31 equivalent FTE engineers with this new agentic coding setup I’ve had. I really started to focus on this new system in the beginning of June.

Now I’m not going to claim that this is 100% accurate, but anecdotally I feel much more productive and find myself more and more confident in the code that gets pushed out. The graph below gives you an idea.

The assumptions above are that 1 engineer would produce 5 PRs per week. Not a perfect metric, but avoids the lines of code issue because it is too verbose. Also, the beginning of this chart is when I was setting up the repo. So there were a ton of PRs being opened to get the infra and everything set up. So the fact that post June, as I started to hit a more mature product, velocity dramatically increased, is pretty cool to see. I’ve also had a couple of “slower weeks” the past weeks. Really just product testing, getting some more guardrails set up, looking for bugs and such. So the “slower weeks” now are somewhat equivalent to the fastest weeks before this switch.

The way I have achieved this is to stop really focusing on implementation details as much. I have started to view this much more as a systems engineering problem. My goal is to make it nearly impossible to ship bad code. If that is true, then you can continue to increase velocity arbitrarily. That also means finding a way to constantly look for and address the next bottleneck.

If I were just to be moving faster without focusing on the system, then we would just get spaghetti code faster, a common trap of “vibe coding”. I guess I differentiate between vibe coding and agentic coding by really focusing on the system and making sure there are enough common check points.

About 3 months ago, my strategy (see LinkedIn Post below) was to go through a step by step scoping session, build session, review session, etc. That worked well, but I was still a bottleneck. With more external checks, I have taken a step away from planning even and more so on quality checks here and there, which means seeing if the product is continuing to work as expected.

As I have focused more so on making sure the system holds, velocity and quality have been increasing. I used to have noisy error channels where I figured 90% of them were just noise. This let real bugs slip through the cracks easily. Now every error that comes in is a high priority error and gets addressed immediately.

This is not some strict set of rules that are set in stone. These are just ones that seem to have resulted in the best results so far.

There are a few things that fall under this category, so let’s dig in.

The stack underneath the build-philosophy doc is Django and DRF on the backend, Next.js and React on the frontend, Tailwind and shadcn for UI. None of it is novel, and that’s deliberate. Mainstream frameworks have an enormous amount of public documentation and an enormous footprint in whatever a model was trained on: years of Stack Overflow answers, official docs, idiomatic open source examples. An agent generating a DRF view is drawing on all of that. An agent generating code for something obscure is drawing on much less, and it shows. More invented APIs that don’t exist, more code that looks plausible and isn’t, more confidently wrong output with nothing behind it to be confident about.

That’s really just a continuation of the argument “boring tech has lower maintenance risk.” A model that already knows Django well writes more idiomatic Django. A model working somewhere it has thin training signal is guessing, and guessing looks like code right up until it doesn’t run.

We all know clean code standards, but it’s really easy to ignore them when coding yourself. Why not just use any for every type? It would take too long to do this manually (at least that’s what I’ve told myself in the past).

LLMs can knock this out in no time. It is a very low cost to do. Unit Tests? Same thing. Here is a list of some of these checks I do that is nearly zero cost now.

Here’s the actual list of checks that are helping before a PR is merged:

My agents.md file also has some guardrails to help prevent the LLMs from going off the rails. I collect “build philosophy” ideas over time that I think will add for a more disciplined codebase. Here are some things that have made it in:

Each one has some extra description in the agents.md but the overall philosophy is straightforward - make clean, consistently structured code. Don’t try to be a hero, stay grounded in the product, etc.

This build philosophy is important to keep concise. The goal is to make usre there are really strong signals for the LLM to know how to code. If you have a list of 100, then they kind of become meaningless. I have NO data to back this up other than just my own experience, but I think 10 is the upper limit to keep it useful.

I don’t just use one tool of everything. I use a combination of Claude Code (via Conductor), Devin and Greptile. Using these three helps avoid any blind spots in one provider or tool.

For new code, I will create a session with either Devin or Claude. Bouncing back and forth helps me make sure that I’m not overindexing on one’s biases. If one of them touches the code that was written by the other, there are “implicit” reviews there because it will have to fix the existing code or make changes if it needs its feature to work.

Almost all PRs are reviewed by Greptile (outside of very, very small non application code changes). There is a loop I have until it gives me a high confidence 5/5 or a “safe to merge 4/5”. Usually those are small things that can be quickly fixed before merging.

Depending on the size of the PR, I will ask Devin to give it a second pair of eyes with their review feature. I also have a skill locally that helps me understand code better. So, if I want to manually review something in depth, I can actually have a skill grill me so that I have an understanding of how the system works. That helps me really make sure I understand how this works.

We have set up extreme logging - like a TON of logging. In my agents.md, we have a line that says “if it isn’t logged, it didn’t happen.” This is amazing for debugging, particularly for agents. If there is a bug, there is a complete breadcrumb trail to help figure it out. I connect Devin to our Posthog MCP, which has access to product analytics, logs, data warehouse (or database) and more so that we can actually root cause bugs instead of just guessing.

I have some automated scans and monitors that act as another guardrail against bad code.

First, I have a Devin automation that looks at our logs every night. It will look at all error logs and open up a pr if it needs. That avoids a noisy error channel.

Second, I’ve tried Devin’s security scanning feature. They do in depth security scans, looking for vulnerabilities. So far its been pretty helpful and has made me feel more confident in shipping code.

You can’t just go from 0 to 100 right out of the gate. That will likely end up in tears. And not from your LLM (even though it may express some sorrow, I wouldn’t believe its sincerity).

For example, I still don’t give any AI access to merging PRs, no matter how small. Every single one is merged by me. It will take some time before I am completely confident.

None of these layers explains the productivity on its own. Not the build philosophy doc, not the mechanical checks, not the auto-review loop, not staged trust. What explains it is that the layers don't fail together. A gap the build philosophy doc misses gets caught by mypy. A gap mypy can't see, because it's a design decision, not a type error, gets caught by an independent reviewer trained differently than the model that wrote the code. A gap even that reviewer misses waits behind a human before anything ships. And a gap none of the current layers can see at all is exactly what the next one, still unproven, is being built to close.

The 38% fix rate says the system isn't clean. It's holding. Those aren't the same claim, and I don't want to conflate them just because the first number is more flattering than the second. What I'd actually stand behind is narrower: velocity that isn't backed by a system built the same way, correction pushed upstream, gates that don't share blind spots, trust staged instead of granted, doesn't stay velocity for long. It just becomes debt with better throughput.