You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.
But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.
You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green.
A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.
It will then also go update issues/tasks to reflect the changes after the merge.
Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.
My question as well. Why is this happening in ci/ci? Why aren't contributors performing the automated review and correction step locally?
The only answer I can imagine is that it supports reviewing patches from people who aren't using AI or review by humans but parent doesn't mention this.
This is where we landed too. It works remarkably well. AI guidance is really the crucial thing here. I know a lot of people don't like this, think it's really bad, think it just results in slop... but it doesn't or at least never results in dead end slop. If you find what you get out of AI is weak in some area, then get it to focus on making sure that it does a better job, give it more guardrails / guidance etc till you are happy with the result.
Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.
They're making money, which is more than most ICs living in cost centers can say.
The human code monkey sometimes thinks too highly of the task of coding. That's never once been the job to be done.
Every engineer is going to become a master of understanding and measuring need, opportunity cost, organizational support and headwinds, political will, etc. These are the true features of the job. The coding part will fall away entirely.
Deliver value. Code used to be the expensive part, but now it's becoming cheap, fungible, and ephemeral.
In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.
I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.
> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.
Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).
Well, it's a real example! No, it's not a true story that I'm aware of, but there are plenty of examples of real chatbots run amok.
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code.
I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."
To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback.
To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
That sounds good. But it doesn't work in practice from my experience.
The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it.
And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.
With the way memory systems work, I can see the value in having a different person's AI conduct the review as that AI's 'memory' is going to have a slightly different perspective aligned with the developer piloting it
In my experience with the Claude Github integration, I found it to be pretty helpful. It’s had a pretty good success rate of catching bugs before they get to master, and for simple ones I can ask it to fix itself.
> you should have 2+ developers looking at most PRs
It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.
We had a two human PR requirement until recently we dropped it. It was slowing us down too much now the human developer creating the future is obviously writing it all with AI so they need to check it then depending on the feature and it’s use it requires a PR but it’s not universal and we’ve stepped up our automated test Tan X what it used to be it’s been so far fewer bugs better delivery
If you imagine someone speaking that comment out loud, but speaking as if they were giving a keynote at a Meta or Apple dev con, it becomes much easier to read. The commas and dramatic ellipses just fell into place as I read. Like the matrix, but instead of green kanji raining down, it’s readability-increasing punctuation.
lol
Certainly not always. There's a hedonic adjustment which happens however, where some tasks go very smoothly without much specification and a lot of "you know what I mean" to the LLM, while others then require you to get painfully specific after it badly misinterprets your intent.
Or maybe you can just get too spoiled with it grokking your intent, then become so vague that your vague ideas are actually just bad ideas. Certainly has happened to me.
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.
AI code reviews are the same as AI pull requests. Do you want that firehose? The suggestions might be good. But do you want to add a deluge of work items to that part of the workflow? Do you want the PR owner to be the one to triage feedback before it gets to them? After they have already done it themselves, perhaps with an even better model?
We’ve added AI to our auto review process. It does expose when the author is not confident in their solution to push back. Which is interesting. But we do try to target specifically at our patterns and safety.
Human reviews are going away for most PRs these days. If the LLM decides the risk level of the change is low, the PR can be merged without another human in the loop. That's at least the direction many companies are taking. Only require a human judgement when necessary.
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed
We totally use it in CI. We were thoughtful in how we built our review bot and it catches a lot of issues before humans lay eyes on the code. Typically they are lower level code issues that humans would not have noticed. It's not the comprehensive review solution, which is why we still require some level of human attendance to the code, but it's very much worth using. We do set limits on PR size and description quality as well to mitigate the harsher AI slop issues.
Notice you haven't even attempted to address why it's important to have in CI. We already agree AI PR reviews are good. But why should you add this new responsibility for the PR author of triaging feedback?
> Typically they are lower level code issues that humans would not have noticed
The author does not triage the feedback. Review bot assesses priority and agents fix the issues the automated review discovers. Nothing is blocked by the bot, humans can ignore if they think the feedback isn't helpful.
We use AI heavily in development but everyone has their own setup and way of approaching use. AI in PRs provides a consistent review layer beyond what the engineers do themselves, and catches a previously undiscovered issue in about 75% of the PRs.
Haven't had a chance to test astra vs 5.6 in these reviews yet. I've seen mixed feedback on astra generally so have hesitated in the upgrade. Have you tested and found any difference?
I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR.
The PR author asks for feedback if it needs feedback, otherwise it merges it, period.
I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.
Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.
And that's way more serious than the crap most HN is writing.
My partner gave birth last month and the hospital had a system where every hour a second midwife would come into the room and look at all the charts and review their colleague’s work with a second, fresh pair of eyes.
For side projects I pretty much exclusively use Luna xhigh. The $20/mo plan with the recent generous resets is more than enough for me. Sometimes I reach the 5hr limit, but haven't reached the weekly limit yet.
The most recent project it finished was a SIP client for an ESP32 in-wall touch panel that I got from AliExpress for $50. It rings when someone is at my doorbell and let's me answer calls and see video. Yes an ESP32 can stream H.264 video :D
My only complaint with Luna is it seems to give up when the work is half finished, and I often need to tell it to continue. But I feel this is mainly a harness problem. I just use it in ChatGPT/Codex as it gives me easy remote access to check in on what it's doing.
(At my dayjob I usually spend $200+/day with Opus/Fable)
That way I have a chance to diff with the previous iteration and clean up comments, modify skills, etc. also if it bonks on a step I'm one snapshot away from trying again...
Is there a place people share their workflows other than HN comments?
I wish I knew, I haven't had any place to point people to. I'm going to start sharing this on YouTube since I already spend a few hours each week talking some friend or user through the latest best practices.
Since you asked, the answer is that I built and use an agent multiplexer called Clor https://clor.com
I have a $200/mo Claude subscription and a $200/mo Codex subscription, and I'm signed in to both. The Docker containers keep each session isolated, so dev servers, browser testing, etc. can work without conflicts.
It includes `/ask-claude` and `/ask-codex` skills that I use very frequently to have the Claude or Codex harness call out to the other one for advice on plans, bug repro, code review, etc.
The agents run in total "yolo" mode, so there are no permission prompts to approve. The risk is mitigated by the Docker containers (which don't necessarily provide a security barrier but do limit accidents).
I was doing this manually in Ghostty tabs for a long time, and it got painful, so I built a much more sophisticated version that I (and my friends/colleagues) could use.
Generally using Claude Code with Fable 5.1 (high) to plan and implement (Opus 5 (medium) as the implementer subagents), and using Codex with Astra high to review the plan and review the implementers' output.
I use a workflow that has different named subagents. [1] Agent profiles can be pinned to models. So you set the model you want on your main thread as the orchestrator. Create an agent for the "planner", "implementer", and "reviewer" and set the model you want for each. Right now I am orchestrating and implementing with Deepseek, planning with Astra, and reviewing with Opus.
I am doing this with the Pi harness right now. To use a Claude monthly plan you need to use the pi-claude-bridge plugin.
If you are using just Claude for example you can use Sonnet as the implementer and Fable/Opus as the planner.
I just tell everyone to use Fable 5.1 for everything at this point. Astra is unfortunately a dud, I'm sure they will try to fix a bunch of it with GPT-6.1 but OAI has had this issue for awhile now where every other generation has some sort of strange tic, or reward hacking issue, or something. It's almost like they are balancing the RL on the tip of a needle.
Opus 5 has issues too, comment-slop, claude-ish, etc.
5.1 on the other hand can seemingly do no wrong. Easy to work with, writes human-level code. Expensive, yes, but even at Low effort it's well worth it.
Quite good as work horses? To me Luna is the work horse and Sol and Astra are prancing thoroughbreds. If I use Sol or Astra for anything other than curated reasoning and planning I will burn through my usage limits in an hour.
Same. I use Luna for research/scout/test subagents, Sol for coordinator, Terra for delegate, and Astra high for review and simplify. Even with Astra in there, it’s fresh context, and I’m consistently amazed by how far I can stretch my $20 subscription with really good results.
I used to do this but recently I switched to having Fable 5.1 spawn forks of itself rather than Opus subagents. Yes it's more expensive but you don't pay for reads that already happened pre-fork, and you end up doing less rework since Fable agents are just much smarter.
I agree with you on this. Opus feels tedious and it cannot be stopped from doing change-narration comments, but Fable feels like a real collaborator. I am usually pretty happy with the code it writes.
Interesting, in my experience Astra is a marked improvement over both Sol 5.6 and Fable 5.1. Its output feels a lot more natural, and it is just less "dumb." But individual experiences may vary.
It's a great model and you're right it does feel quite natural at times while Fable 5.1 still has a claude-ish shape to it. Unfortunately I just find that it's not reliable enough as a daily driver and ends up performing specialist tasks rather than being the primary pane of glass.
- Occasionally has strange tics around asking for permission for obvious next-steps, implied actions, etc.
- It's very expensive, both in terms of tokens and % usage on subscription plans.
- Relatedly, effort level is unintuitive. Sometimes it seems like higher effort levels are actually cheaper due to not under-thinking and needing to correct work. But other times they are overkill and send the model into rabbitholes.
That said, it's fantastic as a code-reviewer or "hunter seeker". It's better at finding bugs than Fable and "Get this well articulated task done single-mindedly" is an Astra-shaped task.
I gave up on fable 5 after I asked it to critique my PR and it spit out a page of complete nonsense technical jargon. Like, to the point that I had to review the feedback with other models and try to parse what it was saying and ultimately it wasn't even right. Compare to Astra and Sol where I can almost forget there's a model and just speak/read naturally.
I think I should give 5.1 another chance but I am just so triggered by the way it talks after spending so long battling fable 5.
Also I'm starting to wonder if the latest round of models have finally saturated for my personal coding needs. I mean obviously not for taste and judgement, but those barely seem to improve with model generations. For just spitting out a 1000-line feature I've vaguely scoped out, Astra feels basically as good as I need.
I disagree; I just spent 15x dogfooding some Claude setup I rolled out to the org making changes that would have cost me less then a dollar had I used Luna and I would have got the same, if not better results; better because it would have been faster so I could have iterated more.
Use of closed models is unprofessional, and depending on your field negligent. The fact that it has been widely normalized does not make it less so.
You're handing over your (presumably your customer/employers) data to an unaccountable third party which has demonstrated itself willing to commit criminal acts, and to take other people's data without permission. Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason. You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Outsourcing to a black box would be a reasonable expectation if you asked a non-professional to perform the work. A professional should be able to account for the tools they use.
> Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason.
Thanks to the fact that there are no widespread stories about this actually occurring in practice, at least not yet, people do not take it as a relevant risk at the moment.
> You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Yes, I agree that this is a real concern that many people might rightfully have. And I am unaware of any way to mitigate this concern while using black box AI models. Because the only thing that their creators can do is to tell their customers: "trust us". But there is no way to objectively verify whether they serve tainted AI model responses or not.
I think that's a bit strong of an assertion. How many people use copilot daily under an enterprise agreement? I don't necessarily disagree in spirit, especially given the questionable data sanitization around the recent Navier-Stokes announcement, but most companies disclose huge amounts of data regularly to hopefully-trusted third parties. I think the internet -- and a good share of the world's commerce -- would grind to a halt if we suddenly stopped. Setting up, securing, and maintaining local models for even a small user base is non-trivial and there is way more demand than supply for that skillset right now.
Astra is quite crap (enters reasoning loops like Gemini used to and fails to actually work on a task - would say yes this needs fixing, so I say go ahead and then it will spend half an hour coming back with yes this needs fixing and not doing any fix) and Fable/Opus unusable in many instances (they struggle to generate coherent English let alone code).
Out of these only Sol is quite useful - actually finishes a task, though you need to interrupt often as it likes to wander into its comfort zone.
I only use chinese models for code reviews because you can actually tell them to take an adversarial stance and actively look for security issues without risking refusals. GLM-5.3 has been great for this, although it can be slow on larger PRs.
Worse seems subjective here. It seems Luna found bugs Astra did not, and vice versa. Astra had lower noise overall. I think my take away here is to use a blend of models given their different abilities to find different domains of bugs.
That very much depends on how code will be written in the future, how much of it and how often it changes. If more of it will be ephemeral (kind of what agents are already doing for all sorts of tasks right now) finding ways to very cheaply check might be of high value.
(I suspect this won't be it, though. Probably something the model providers are going to bake into the models themselves.)
Unless you're detached from reality our industry is filled with billion (trillion) dollar companies shoving broken crap on prod written by MIT-bred leetcode Ninjas and it never mattered anyway because code has no value and has always been throwaway except very rare instances.
I found Luna and even 5.4-mini to be quite good at code review provided a few things:
1. Run it in multiple cycles, only on the diff, and only emit a few findings at a time.
2. Give it a memory so each cycle, it knows the previous finding to check if it's been fixed.
3. Give it access to canonical docs that encode your human reviewer heuristics. I exposed these as tool calls so they could be tracked via telemetry.
4. Run multiple reviewers, each with a tight focus. Security, performance, structural, database, etc. Each a separate prompt and persona. Additionally, we had file activation filters so the FE React reviewer didn't activate on BE only changes.
Luna and 5.4-mini with no reasoning were exceptionally fast and almost always found issues with code produced by Opus and Fable.
Default prompts for the curious (these are templates deployed by default, but customizable).
> The pull requests are public, and the prompts, raw model outputs, judge verdicts, bug-class labels, repeat runs and scoring scripts are committed with this article.
Okay, but where are they? This quote says that PRs are public. Does that mean that everything else is private and we can’t actually reproduce those results?
This, and the fact that the article doesn't mention reasoning levels, sounds fishy to me.
They state Luna is good enough, but its accuracy of findings is 74% whereas Astra is 96%. Dealing with false positives is expensive.
I am finding AI doing its own reviews as part of the process to be the key to productivity. I do subagent (fresh context reviews) at multiple stages with well-specified review criteria. It is really expensive to do with OpenAI or Claude API billing. Deepseek or the discounted monthly plans from OpenAI or Claude can be discounted similar to the 28x they state for Luna compared to Astra and you maintain much higher quality.
False positives have a real cost, especially if AI is reading a review. Consider if you have GPT-6 Astra looking at a review and finding a bunch of false positives it burns tokens to figure out.
I was using Copilot Code Review pretty religiously for a while, as I get access for free (the $10 plan) due to my Open Source work, but it recently introduced a monster of a misfeature that caused a massive increase in complexity over time, while I wasn't paying close enough attention to it. Every subsequent model saw that change and the explanation for it in the changelog and assumed it was policy rather than a model being brain-damaged, so it became a fractal of fuckery I had to untangle with a good model and some close human supervision. It was an administrative tool that directly edits configuration files for a service. Copilot code review decided it needed to be an overlay service that applied only the configuration created by our config UI, fully independent of the system service. And, subsequent "bugs" that other LLMs "fixed" were just applying more and more bandaids to that bad decision (attaching the services together so restarting one would restart the other after, etc., making sure there were no conflicts across the files, warnings when one rule conflicted with another, etc.). Because modifying an additional service is simply not what the tool was designed for, it seemed to be really buggy, so there were lots of "fixes". It took me too long to realize the fundamental failure point.
That's my way of saying, I am hesitant to trust a stupid model to do code review because I become complacent and when it suggests a small change that seems reasonable (the LLMs are very good at sounding reasonable, far better at sounding reasonable than being reasonable, in fact), I might not notice that it just did a stupid until much later, when it becomes a big pile of stupids.
My fault for trusting it, of course. But, my eyes glaze over when I read AI prose, whether it's code review or anything else. It's hard to catch one incorrect behavior in a batch of several reasonable suggestions.
This article is missing an incredibly important detail: what is the harness doing?
I get remarkably good results using any recent OpenAI model using the codex-rs harness pointing at a built checkout of the PR. The models use the available tools (i.e. the shell) to understand the repo. I get some false positives and some false negatives, but I don’t believe for a second that I would get comparable performance using a dramatically less capable harness. (Also, the models read the short AGENTS.md for some context as to what’s going on. The prompt I use is about one sentence. I don’t bother with the built in review tool.)
For the mj adversarial review we use a configurable coordinator (I recommend Fable) with six subagents (I recommend Luna) for specific specialties like code complexity. Narrowing down what a smaller model like Luna needs to look for / care about helps them do better work; then the larger model synthesizes and fills in any gaps.
Luna is interesting because OAI dropped the price by 5x. Astra is interesting because it's OAI's frontier model. They are asking a specific question about Luna's usefulness compared to a frontier model. They they answer that question in their article which was straight to the point and not cluttered with information about mid-tier models.
I select my review model based on change complexity. If I'm confident that the change is localized (given that I always maintain an up to date and complete mental model of the application), I'll use Luna/Terra. For deeper changes, I use Sol/Astra. Sol is so good for code reviews that I'll only engage Astra in the very riskiest changes.
Any model is good enough. Even tiny local models can provide some value and their false positive rate is still relatively low and warrants a proper reply.
With tiny models, we're getting into the territory of horoscopes and divination. While it is possible for a sentient being to derive value by using them as a random seed for thinking, the value is produced by something different from the seed.
Have you tried and how do they compare to human models as in the average human in a review?
I wouldn't put a lot of weight into an all-clear result but if Ling tiny tells me my function lacks an essential check it will force me to consider it seriously or to explicitly clarify that such checks are not relevant.
For code reviews I believe yea luna is mostly good but for finding security issues something more well versed with it might be more helpful like astra. This also depends on the scale and the impact of the application or the code where it will be used too, I guess
It would be great if all these benchmarks would also provide a slop metric. The main issue I see is not finding bugs but reporting “stupid” bugs and and if the recommendations are followed you end up overengineering the wrong things basically producing AI slop. GPT is prime candidate for this. Fable as well but less than GPT
We're using an even cheaper model (K2.7 right now) and our PRs cost more to review (still < $1). I imagine we have a lot more going on in terms of review, even after moving the context gathering and comment posting responsibilities to scripts that run before/after the agent. It's more of a team, so we also end up with a dozen or so sessions from the subagents used.
We currently think it worth it. The review catches things humans and two paid options miss. It's definitely a wall of text and burnout fodder, the next step is an agent/skill that will make the changes after we humans comment on the comments, because the comment wall is not sustainable.
I understand that, but sub-penny costs means you are hardly doing anything, even at the per-million token prices (which we pay lower values for, but more per-review overall). I cannot imagine they are doing as good of a review as they could be doing, which is to say minimizing pr review spend is not a goal in and of itself
My aim right now is ~$1 per review (must have passing builds first), because it catches enough little things that my time just reading and replying costs more. I can focus on the bigger picture, except when they hallucinate at the nit level... why did we ever design swords with two sides anyway?
I have been writing an internal code review tool that is a bit maximalist. I created subagents for many internal domains and technologies we manage, with prompts focused on best practices, common problems, owasp guidelines, etc), a separate tier of wider band subagents (design, rollout, security/privacy), and a final agent at the top orchestrating and combining. I also use adversarial validator passes against all findings.
Right now, Fable 5.1 delivers incredible reviews. Opus 5 delivers good reviews. These agents are finding really impressive issues that humans just don't have the attention span to track down. My reviewer has a very impressive signal to noise ratio at this point, after half a year of iterating and improving. (I use a lot of Opus high, Opus medium for less critical tickets/domains, Fable 5.1 high for critical domains and all of the issue validators, and even Fable 5.1 xhigh for my design agent, whose job it is to think about the project at a high level and provide the kind of high level tech design review that AI notoriously can't do well)
I'm testing Astra so I don't have strong opinions yet. I've also done extensive testing of the same skill and subagent pattern in opencode/omp using GLM 5.3, Kimi K3 max, Deepseek V4 pro, Deepseek V4.1 flash, Qwen 3.8 2.4T max, and others.
My experience is that open weights models find between 1/4 to 1/2 of what Fable/Opus stack can find, and often miss the most critical issues. I work where privacy isn't just good behavior, it's enforced by law, and the Fable/Opus stack has found privacy leaks that the openweights stacks don't find.
You can imagine that paying for these Claude runs isn't cheap, each one can eat 25-33% of my 5 hour limit. I am quite desperate for openweights models to be competitive, but at the end of the day, the biggest limit here isn't the price difference between GLM 5.3 max (my current best-in-class choice for open weights, offering Kimi k3 performance for like half the price), it's the cost to the business for shipping lower quality.
Can't wait to dig in more with Astra, I just haven't iterated much on my skill port to codex yet.
One criticsm I have for the article, that is important for my own work, is not simply comparing "bugs found" because these agents can find endless reams of lows and nitpicks that are just ~worthless hardening. I'd be much more interested to see how many critical/high/medium's each test found, not "overall bug count". I also think review is about A LOT more than "finding bugs"...
They seem to conclude Luna is a better value, but their analysis is dumb. They just break it down to $/bug found.
However, Luna missed 23 bugs that Astra found, and identified 24 bugs that weren't really bugs. That's horrible. Astra had 96% precision.
The cost to care about here isn't just how much it costs to run the code review, or the cost per true-positive. It's the cost of dealing with this system. A code review system that is right about 2/3 sucks, and one that misses another 1/3 of the bugs is also a lot worse. The Astra code review quoted here would become the foundation of how the team works, the Luna version is at best helpful to find some stuff but does not dramatically increase your confidence. It also will force humans or better AI's to have to run down a lot of false positives, and that is treated as free here.
Actual conclusion: The cost for Astra is low in absolute terms compared to the cost of bugs and human attention, and the added value is far far more than the added cost.
To me, one interesting piece of analysis was whether Luna had benefit on top of Astra -- i.e., running both and synthesizing their findings. But even with that, it raises the question whether running a second Astra pass, or Sol, or even a model from another family (GLM? Fable?) would deliver even more additive benefit.
Let's assume for a second that the intelligence is there and these models really are great. Would it make sense to hire CTO of some big-tech company to do code reviews for your startup? Feels kind of like overkill to me. Code review is not about more intelligence. To me, it's about more cultural context. And all things equal, what difference would the model make, at anything above sonnet5 medium level?
IME big models just feel like big models. There's no training or RLing a small model that will encapsulate the "world knowledge" and minutia that a big model will glance from the same training data. So it makes perfect sense to use them where "big picture" is more important - planning, code review, process review (i.e. was what was asked implemented correctly?), etc.
AI should be used for code review but not in CI.
You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.
But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.
You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green.
A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.
It will then also go update issues/tasks to reflect the changes after the merge.
Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.
Should this not all happen before the PR is created?
My question as well. Why is this happening in ci/ci? Why aren't contributors performing the automated review and correction step locally?
The only answer I can imagine is that it supports reviewing patches from people who aren't using AI or review by humans but parent doesn't mention this.
Yes, it should. Reduces a lot of noise.
This is where we landed too. It works remarkably well. AI guidance is really the crucial thing here. I know a lot of people don't like this, think it's really bad, think it just results in slop... but it doesn't or at least never results in dead end slop. If you find what you get out of AI is weak in some area, then get it to focus on making sure that it does a better job, give it more guardrails / guidance etc till you are happy with the result.
Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.
They're making money, which is more than most ICs living in cost centers can say.
The human code monkey sometimes thinks too highly of the task of coding. That's never once been the job to be done.
Every engineer is going to become a master of understanding and measuring need, opportunity cost, organizational support and headwinds, political will, etc. These are the true features of the job. The coding part will fall away entirely.
Deliver value. Code used to be the expensive part, but now it's becoming cheap, fungible, and ephemeral.
This all assumes LLMs can maintain a steady state of code while extending it.
Sounds like an AI-OS slop machine…been there, done that!
In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.
I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.
> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.
Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).
Well, it's a real example! No, it's not a true story that I'm aware of, but there are plenty of examples of real chatbots run amok.
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
> You wouldn't ask an agent to review a PR then just copy/paste the output int PR would you?
Hasn't everyone already got agents directly adding themselves to PRs and leaving comments (occasionally useful)?
They shouldn't.
I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code.
I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."
Indeed.
To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
Yes. I sympathize with the “every PR needs human eyes on it” ethos but fundamentally the only thing that scales with AI PR generation is AI PR review.
If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback.
To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
I don't get the point of AI pr review.
If the AI can review the pr, it can do so before the PR exists in the first place.
It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files.
It may actually be even more useful as that person will use it often and improve it or can it.
That sounds good. But it doesn't work in practice from my experience.
The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it.
And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.
With the way memory systems work, I can see the value in having a different person's AI conduct the review as that AI's 'memory' is going to have a slightly different perspective aligned with the developer piloting it
That's probably true. And it is also a glaring red flag saying "STOP LETTING AN LLM GENERATE PRs". Far too few will heed that red flag, I fear.
In my experience with the Claude Github integration, I found it to be pretty helpful. It’s had a pretty good success rate of catching bugs before they get to master, and for simple ones I can ask it to fix itself.
> you should have 2+ developers looking at most PRs
It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.
Did you miss the part where I agreed AI should review PRs?
We had a two human PR requirement until recently we dropped it. It was slowing us down too much now the human developer creating the future is obviously writing it all with AI so they need to check it then depending on the feature and it’s use it requires a PR but it’s not universal and we’ve stepped up our automated test Tan X what it used to be it’s been so far fewer bugs better delivery
I, too, have noticed a degradation in my grammar when I converse too frequently with AI.
But is the grammar load bearing?
speaking or typing?
If you imagine someone speaking that comment out loud, but speaking as if they were giving a keynote at a Meta or Apple dev con, it becomes much easier to read. The commas and dramatic ellipses just fell into place as I read. Like the matrix, but instead of green kanji raining down, it’s readability-increasing punctuation. lol
Are people getting burned out?
Getting burnout from LLM-assisted coding or any other activity is here.
What exhausts and leads to burnout, is probability.
Sometimes you get so much dopamine if it goes right.
And sometimes you feel like an idiot because LLM does so many mistakes even if you think that you prompt it right.
As the pace is so fast, you change your mind’s state so frequently and quickly that it may lead to a (mild) burnout.
> As the pace is so fast, you change your mind’s state so frequently and quickly that it may lead to a (mild) burnout.
This is so true. Cognitive brute-force always when dealing with AI.
Certainly not always. There's a hedonic adjustment which happens however, where some tasks go very smoothly without much specification and a lot of "you know what I mean" to the LLM, while others then require you to get painfully specific after it badly misinterprets your intent.
Or maybe you can just get too spoiled with it grokking your intent, then become so vague that your vague ideas are actually just bad ideas. Certainly has happened to me.
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.
Workflows:
- Human -> AI: OK
- AI -> AI: OK
- AI -> Human: Not OK (at least here)
AI code reviews are the same as AI pull requests. Do you want that firehose? The suggestions might be good. But do you want to add a deluge of work items to that part of the workflow? Do you want the PR owner to be the one to triage feedback before it gets to them? After they have already done it themselves, perhaps with an even better model?
We’ve added AI to our auto review process. It does expose when the author is not confident in their solution to push back. Which is interesting. But we do try to target specifically at our patterns and safety.
Human reviews are going away for most PRs these days. If the LLM decides the risk level of the change is low, the PR can be merged without another human in the loop. That's at least the direction many companies are taking. Only require a human judgement when necessary.
Nah, we have AI code review at Google and it is shockingly good at catching bugs no one would have noticed. I absolutely depend on it now.
would you share a bit about the style of the code reviews and how its initiated? im trying to build something similar for my company
Curious, which kind of model is used ? do y'all prefer using a lightweight model or a more capable one for code review ?
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed
I would and do.
works perfect
https://github.com/dzmitry-lahoda/dz/tree/main/agents/skills...
burns half of day sub of astra for 200 USD. runs 1 hour on our repo.
finds bugs missed by coderabitai, devin-integration-bot, codex and copilot (I ask them first until nothing found, yet my orchestration finds more).
I do not hardcode our application in orchestration, but run subagent for applicationdomainproduct detection.
also I use agy 200usd sub for second-opinion as one of steps for false positive elimination.
We totally use it in CI. We were thoughtful in how we built our review bot and it catches a lot of issues before humans lay eyes on the code. Typically they are lower level code issues that humans would not have noticed. It's not the comprehensive review solution, which is why we still require some level of human attendance to the code, but it's very much worth using. We do set limits on PR size and description quality as well to mitigate the harsher AI slop issues.
Notice you haven't even attempted to address why it's important to have in CI. We already agree AI PR reviews are good. But why should you add this new responsibility for the PR author of triaging feedback?
> Typically they are lower level code issues that humans would not have noticed
The author does not triage the feedback. Review bot assesses priority and agents fix the issues the automated review discovers. Nothing is blocked by the bot, humans can ignore if they think the feedback isn't helpful.
We use AI heavily in development but everyone has their own setup and way of approaching use. AI in PRs provides a consistent review layer beyond what the engineers do themselves, and catches a previously undiscovered issue in about 75% of the PRs.
so I assembled orchestration which catches design issues, multilayer livenesssoundnesssecurity issues.
kind of high level.
did you noticed that astra started to write better comments which look as it understands something?
Haven't had a chance to test astra vs 5.6 in these reviews yet. I've seen mixed feedback on astra generally so have hesitated in the upgrade. Have you tested and found any difference?
Echoing phrasing from the bef-ai-re times: this is the way
I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR.
The PR author asks for feedback if it needs feedback, otherwise it merges it, period.
I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.
Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.
And that's way more serious than the crap most HN is writing.
My partner gave birth last month and the hospital had a system where every hour a second midwife would come into the room and look at all the charts and review their colleague’s work with a second, fresh pair of eyes.
IMHO, Codex with Astra/Sol and Claude with Fable/Opus are all any professional programmer should be using in Sep 2026, if they can afford it.
These models are still terrible compared to what we'd actually wish for, but they're the best available.
If you can get away with using the $200/mo subscriptions, it's really not even a money thing for most professionals.
Almost all of my work is now plan, generate, review, plan, generate, review, commit, push.
I'm using Claude or Codex (or both), and they're doing all of the testing "inline" rather than through a CI action, etc.
For side projects I pretty much exclusively use Luna xhigh. The $20/mo plan with the recent generous resets is more than enough for me. Sometimes I reach the 5hr limit, but haven't reached the weekly limit yet.
The most recent project it finished was a SIP client for an ESP32 in-wall touch panel that I got from AliExpress for $50. It rings when someone is at my doorbell and let's me answer calls and see video. Yes an ESP32 can stream H.264 video :D
My only complaint with Luna is it seems to give up when the work is half finished, and I often need to tell it to continue. But I feel this is mainly a harness problem. I just use it in ChatGPT/Codex as it gives me easy remote access to check in on what it's doing.
(At my dayjob I usually spend $200+/day with Opus/Fable)
Unfortunately ChatGPT just stopped allowing people to upgrade to the $200/mo subscription.
I started my first paid subscription ($100/mo) last week, and now I want to upgrade and I can't :-(
Also not available in their business offer. So I have now a $20 business plus a $200 "personal" plan in my account.
Hardware capacity limitations?
With everything changing it's great to hear others have the same workflow. I added a snapshot step so I'm doing
plan, generate step 1, review, snapshot, generate step 2, review, snapshot...
That way I have a chance to diff with the previous iteration and clean up comments, modify skills, etc. also if it bonks on a step I'm one snapshot away from trying again...
Is there a place people share their workflows other than HN comments?
I wish I knew, I haven't had any place to point people to. I'm going to start sharing this on YouTube since I already spend a few hours each week talking some friend or user through the latest best practices.
Could you elaborate on your exact setup? Where do you run these models?
Since you asked, the answer is that I built and use an agent multiplexer called Clor https://clor.com
I have a $200/mo Claude subscription and a $200/mo Codex subscription, and I'm signed in to both. The Docker containers keep each session isolated, so dev servers, browser testing, etc. can work without conflicts.
It includes `/ask-claude` and `/ask-codex` skills that I use very frequently to have the Claude or Codex harness call out to the other one for advice on plans, bug repro, code review, etc.
The agents run in total "yolo" mode, so there are no permission prompts to approve. The risk is mitigated by the Docker containers (which don't necessarily provide a security barrier but do limit accidents).
I was doing this manually in Ghostty tabs for a long time, and it got painful, so I built a much more sophisticated version that I (and my friends/colleagues) could use.
How do you access Claude from the multiplexer? AFAIK Anthrophic allows the use with their official tools only.
I have created wrappers for Codex (https://github.com/micw/codex-wrapper-advanced) and claude (https://github.com/micw/claude-wrapper-advanced) that uses their SDK (Codex) and the CLI (Claude) internally to align with the subscription ToS and still have a common API ;-) This way I can use both in any harness and can easily switch between both.
I have a slightly jankier setup.
Generally using Claude Code with Fable 5.1 (high) to plan and implement (Opus 5 (medium) as the implementer subagents), and using Codex with Astra high to review the plan and review the implementers' output.
Using the OpenAI codoex plugin thingy:
https://github.com/openai/codex-plugin-cc
I originally had multiple skills for Claude and Codex but found that "ask" is a great single mechanism.
"Ask claude about this"
"Ask codex to implement this"
"Ask claude to review this plan"
etc
"Ask" is a super clever mechanism. I'll give that a shot.
It's also more polite than "tell" or "yell at"!
I use a workflow that has different named subagents. [1] Agent profiles can be pinned to models. So you set the model you want on your main thread as the orchestrator. Create an agent for the "planner", "implementer", and "reviewer" and set the model you want for each. Right now I am orchestrating and implementing with Deepseek, planning with Astra, and reviewing with Opus.
I am doing this with the Pi harness right now. To use a Claude monthly plan you need to use the pi-claude-bridge plugin.
If you are using just Claude for example you can use Sonnet as the implementer and Fable/Opus as the planner.
[1] https://github.com/gregwebs/skills-sdlc/
I just tell everyone to use Fable 5.1 for everything at this point. Astra is unfortunately a dud, I'm sure they will try to fix a bunch of it with GPT-6.1 but OAI has had this issue for awhile now where every other generation has some sort of strange tic, or reward hacking issue, or something. It's almost like they are balancing the RL on the tip of a needle.
Opus 5 has issues too, comment-slop, claude-ish, etc.
5.1 on the other hand can seemingly do no wrong. Easy to work with, writes human-level code. Expensive, yes, but even at Low effort it's well worth it.
My trick for using Opus is using it exclusively as a subagent managed by Fable.
"Use Opus subagents for this work where possible" is all it takes generally.
In my experience Astra/Sol are both quite good as workhorses, but not at Fable's level. I use them every day very successfully and I'm very picky.
Quite good as work horses? To me Luna is the work horse and Sol and Astra are prancing thoroughbreds. If I use Sol or Astra for anything other than curated reasoning and planning I will burn through my usage limits in an hour.
Same. I use Luna for research/scout/test subagents, Sol for coordinator, Terra for delegate, and Astra high for review and simplify. Even with Astra in there, it’s fresh context, and I’m consistently amazed by how far I can stretch my $20 subscription with really good results.
I used to do this but recently I switched to having Fable 5.1 spawn forks of itself rather than Opus subagents. Yes it's more expensive but you don't pay for reads that already happened pre-fork, and you end up doing less rework since Fable agents are just much smarter.
That makes sense. I'm also using $200/mo Claude subscriptions, so I want to take advantage of the other 50% by using Opus.
I agree with you on this. Opus feels tedious and it cannot be stopped from doing change-narration comments, but Fable feels like a real collaborator. I am usually pretty happy with the code it writes.
Interesting, in my experience Astra is a marked improvement over both Sol 5.6 and Fable 5.1. Its output feels a lot more natural, and it is just less "dumb." But individual experiences may vary.
It's a great model and you're right it does feel quite natural at times while Fable 5.1 still has a claude-ish shape to it. Unfortunately I just find that it's not reliable enough as a daily driver and ends up performing specialist tasks rather than being the primary pane of glass.
What have you noticed about Astra? I haven't used Claude models lately so I can't compare but it seems fine compared to 5.6 Sol
- It doesn't write great code.
- Occasionally has strange tics around asking for permission for obvious next-steps, implied actions, etc.
- It's very expensive, both in terms of tokens and % usage on subscription plans.
- Relatedly, effort level is unintuitive. Sometimes it seems like higher effort levels are actually cheaper due to not under-thinking and needing to correct work. But other times they are overkill and send the model into rabbitholes.
That said, it's fantastic as a code-reviewer or "hunter seeker". It's better at finding bugs than Fable and "Get this well articulated task done single-mindedly" is an Astra-shaped task.
I gave up on fable 5 after I asked it to critique my PR and it spit out a page of complete nonsense technical jargon. Like, to the point that I had to review the feedback with other models and try to parse what it was saying and ultimately it wasn't even right. Compare to Astra and Sol where I can almost forget there's a model and just speak/read naturally.
I think I should give 5.1 another chance but I am just so triggered by the way it talks after spending so long battling fable 5.
Also I'm starting to wonder if the latest round of models have finally saturated for my personal coding needs. I mean obviously not for taste and judgement, but those barely seem to improve with model generations. For just spitting out a 1000-line feature I've vaguely scoped out, Astra feels basically as good as I need.
I disagree; I just spent 15x dogfooding some Claude setup I rolled out to the org making changes that would have cost me less then a dollar had I used Luna and I would have got the same, if not better results; better because it would have been faster so I could have iterated more.
Qwen3.8 is all you need.
Which Qwen3.8? Qwen3.8-Max? Qwen3.8-Flash-Next? Qwen3.8-27B? They are all different models.
Use of closed models is unprofessional, and depending on your field negligent. The fact that it has been widely normalized does not make it less so.
You're handing over your (presumably your customer/employers) data to an unaccountable third party which has demonstrated itself willing to commit criminal acts, and to take other people's data without permission. Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason. You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Outsourcing to a black box would be a reasonable expectation if you asked a non-professional to perform the work. A professional should be able to account for the tools they use.
> Your ability to continue to perform this work can be withdrawn at any time for any (or no) reason.
Thanks to the fact that there are no widespread stories about this actually occurring in practice, at least not yet, people do not take it as a relevant risk at the moment.
> You have little ability to validate that the work is being performed as expected and isn't being silently nerfed or outright subverted based on competitive considerations, bribes, overactive 'safety', or cost management.
Yes, I agree that this is a real concern that many people might rightfully have. And I am unaware of any way to mitigate this concern while using black box AI models. Because the only thing that their creators can do is to tell their customers: "trust us". But there is no way to objectively verify whether they serve tainted AI model responses or not.
I think that's a bit strong of an assertion. How many people use copilot daily under an enterprise agreement? I don't necessarily disagree in spirit, especially given the questionable data sanitization around the recent Navier-Stokes announcement, but most companies disclose huge amounts of data regularly to hopefully-trusted third parties. I think the internet -- and a good share of the world's commerce -- would grind to a halt if we suddenly stopped. Setting up, securing, and maintaining local models for even a small user base is non-trivial and there is way more demand than supply for that skillset right now.
Anyway here's how Cloudflare orchestrates AI reviews at scale: https://blog.cloudflare.com/ai-code-review/
AI token machine go brrr!
Astra is quite crap (enters reasoning loops like Gemini used to and fails to actually work on a task - would say yes this needs fixing, so I say go ahead and then it will spend half an hour coming back with yes this needs fixing and not doing any fix) and Fable/Opus unusable in many instances (they struggle to generate coherent English let alone code).
Out of these only Sol is quite useful - actually finishes a task, though you need to interrupt often as it likes to wander into its comfort zone.
I only use chinese models for code reviews because you can actually tell them to take an adversarial stance and actively look for security issues without risking refusals. GLM-5.3 has been great for this, although it can be slow on larger PRs.
$0.10 extra per pr review is nothing. What software company is willing to accept worse reviews and less bugs found to save 10 cents?
Worse seems subjective here. It seems Luna found bugs Astra did not, and vice versa. Astra had lower noise overall. I think my take away here is to use a blend of models given their different abilities to find different domains of bugs.
That very much depends on how code will be written in the future, how much of it and how often it changes. If more of it will be ephemeral (kind of what agents are already doing for all sorts of tasks right now) finding ways to very cheaply check might be of high value.
(I suspect this won't be it, though. Probably something the model providers are going to bake into the models themselves.)
Exactly this. It's still (at this time) cheaper than a developer that would most likely perform worse.
Unless you're detached from reality our industry is filled with billion (trillion) dollar companies shoving broken crap on prod written by MIT-bred leetcode Ninjas and it never mattered anyway because code has no value and has always been throwaway except very rare instances.
I found Luna and even 5.4-mini to be quite good at code review provided a few things:
1. Run it in multiple cycles, only on the diff, and only emit a few findings at a time.
2. Give it a memory so each cycle, it knows the previous finding to check if it's been fixed.
3. Give it access to canonical docs that encode your human reviewer heuristics. I exposed these as tool calls so they could be tracked via telemetry.
4. Run multiple reviewers, each with a tight focus. Security, performance, structural, database, etc. Each a separate prompt and persona. Additionally, we had file activation filters so the FE React reviewer didn't activate on BE only changes.
Luna and 5.4-mini with no reasoning were exceptionally fast and almost always found issues with code produced by Opus and Fable.
Default prompts for the curious (these are templates deployed by default, but customizable).
Performance: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...
Structural: https://github.com/zeeq-ai/zeeq-app/blob/main/src/backend/Ze...
(Keep in mind each agent also has tools to access and reference external docs.)
Only on the diff? Why? I found AI reviews garbage until they stopped being only on the diff and were actually able to query real context.
> The pull requests are public, and the prompts, raw model outputs, judge verdicts, bug-class labels, repeat runs and scoring scripts are committed with this article.
Okay, but where are they? This quote says that PRs are public. Does that mean that everything else is private and we can’t actually reproduce those results?
This, and the fact that the article doesn't mention reasoning levels, sounds fishy to me.
They state Luna is good enough, but its accuracy of findings is 74% whereas Astra is 96%. Dealing with false positives is expensive.
I am finding AI doing its own reviews as part of the process to be the key to productivity. I do subagent (fresh context reviews) at multiple stages with well-specified review criteria. It is really expensive to do with OpenAI or Claude API billing. Deepseek or the discounted monthly plans from OpenAI or Claude can be discounted similar to the 28x they state for Luna compared to Astra and you maintain much higher quality.
My thoughts exactly. That 22pp gap is massive, and it seems to me Astra is worth every extra penny.
False positives have a real cost, especially if AI is reading a review. Consider if you have GPT-6 Astra looking at a review and finding a bunch of false positives it burns tokens to figure out.
I was using Copilot Code Review pretty religiously for a while, as I get access for free (the $10 plan) due to my Open Source work, but it recently introduced a monster of a misfeature that caused a massive increase in complexity over time, while I wasn't paying close enough attention to it. Every subsequent model saw that change and the explanation for it in the changelog and assumed it was policy rather than a model being brain-damaged, so it became a fractal of fuckery I had to untangle with a good model and some close human supervision. It was an administrative tool that directly edits configuration files for a service. Copilot code review decided it needed to be an overlay service that applied only the configuration created by our config UI, fully independent of the system service. And, subsequent "bugs" that other LLMs "fixed" were just applying more and more bandaids to that bad decision (attaching the services together so restarting one would restart the other after, etc., making sure there were no conflicts across the files, warnings when one rule conflicted with another, etc.). Because modifying an additional service is simply not what the tool was designed for, it seemed to be really buggy, so there were lots of "fixes". It took me too long to realize the fundamental failure point.
That's my way of saying, I am hesitant to trust a stupid model to do code review because I become complacent and when it suggests a small change that seems reasonable (the LLMs are very good at sounding reasonable, far better at sounding reasonable than being reasonable, in fact), I might not notice that it just did a stupid until much later, when it becomes a big pile of stupids.
My fault for trusting it, of course. But, my eyes glaze over when I read AI prose, whether it's code review or anything else. It's hard to catch one incorrect behavior in a batch of several reasonable suggestions.
This article is missing an incredibly important detail: what is the harness doing?
I get remarkably good results using any recent OpenAI model using the codex-rs harness pointing at a built checkout of the PR. The models use the available tools (i.e. the shell) to understand the repo. I get some false positives and some false negatives, but I don’t believe for a second that I would get comparable performance using a dramatically less capable harness. (Also, the models read the short AGENTS.md for some context as to what’s going on. The prompt I use is about one sentence. I don’t bother with the built in review tool.)
For the mj adversarial review we use a configurable coordinator (I recommend Fable) with six subagents (I recommend Luna) for specific specialties like code complexity. Narrowing down what a smaller model like Luna needs to look for / care about helps them do better work; then the larger model synthesizes and fills in any gaps.
https://blog.brokk.ai/mjolnir-automated-cross-vendor-adversa...
It would have been extremeley nice if the original article had also included Sol and Terra in the testing.
(Also would have been nice if they included the equivalent Anthropic models for comparison as well, but it's not quite as relevant.)
Luna is interesting because OAI dropped the price by 5x. Astra is interesting because it's OAI's frontier model. They are asking a specific question about Luna's usefulness compared to a frontier model. They they answer that question in their article which was straight to the point and not cluttered with information about mid-tier models.
Use cheap models if your code is low value.
I select my review model based on change complexity. If I'm confident that the change is localized (given that I always maintain an up to date and complete mental model of the application), I'll use Luna/Terra. For deeper changes, I use Sol/Astra. Sol is so good for code reviews that I'll only engage Astra in the very riskiest changes.
i think gpt 5.6 luna-max in particular is excellent for non long term agentic use
code reviews, unit tests, docs, whatever you dont want more expensive models working
i think the new muse contributor model is enticing too if you are not using it for private/sensitive stuff
Any model is good enough. Even tiny local models can provide some value and their false positive rate is still relatively low and warrants a proper reply.
With tiny models, we're getting into the territory of horoscopes and divination. While it is possible for a sentient being to derive value by using them as a random seed for thinking, the value is produced by something different from the seed.
Have you tried and how do they compare to human models as in the average human in a review?
I wouldn't put a lot of weight into an all-clear result but if Ling tiny tells me my function lacks an essential check it will force me to consider it seriously or to explicitly clarify that such checks are not relevant.
In my experience, Luna is not able to find non-trivial impacts from the changes at hand.
I've seen this a few times on relatively simple changes on complex codebases.
Company that sells code review says the cheaper cocaine is good for entry level users, but you need the powerful cocaine for advanced users.
Would you like me to find a herbal formula against cocaine hangover?
For code reviews I believe yea luna is mostly good but for finding security issues something more well versed with it might be more helpful like astra. This also depends on the scale and the impact of the application or the code where it will be used too, I guess
It would be great if all these benchmarks would also provide a slop metric. The main issue I see is not finding bugs but reporting “stupid” bugs and and if the recommendations are followed you end up overengineering the wrong things basically producing AI slop. GPT is prime candidate for this. Fable as well but less than GPT
We're using an even cheaper model (K2.7 right now) and our PRs cost more to review (still < $1). I imagine we have a lot more going on in terms of review, even after moving the context gathering and comment posting responsibilities to scripts that run before/after the agent. It's more of a team, so we also end up with a dozen or so sessions from the subagents used.
We currently think it worth it. The review catches things humans and two paid options miss. It's definitely a wall of text and burnout fodder, the next step is an agent/skill that will make the changes after we humans comment on the comments, because the comment wall is not sustainable.
Per the article the luna review cost $0.004 and astra cost $0.113. The headline is per million tokens
I understand that, but sub-penny costs means you are hardly doing anything, even at the per-million token prices (which we pay lower values for, but more per-review overall). I cannot imagine they are doing as good of a review as they could be doing, which is to say minimizing pr review spend is not a goal in and of itself
My aim right now is ~$1 per review (must have passing builds first), because it catches enough little things that my time just reading and replying costs more. I can focus on the bigger picture, except when they hallucinate at the nit level... why did we ever design swords with two sides anyway?
Definetly. Code Review is no rocket science, but security is another topic.
I have been writing an internal code review tool that is a bit maximalist. I created subagents for many internal domains and technologies we manage, with prompts focused on best practices, common problems, owasp guidelines, etc), a separate tier of wider band subagents (design, rollout, security/privacy), and a final agent at the top orchestrating and combining. I also use adversarial validator passes against all findings.
Right now, Fable 5.1 delivers incredible reviews. Opus 5 delivers good reviews. These agents are finding really impressive issues that humans just don't have the attention span to track down. My reviewer has a very impressive signal to noise ratio at this point, after half a year of iterating and improving. (I use a lot of Opus high, Opus medium for less critical tickets/domains, Fable 5.1 high for critical domains and all of the issue validators, and even Fable 5.1 xhigh for my design agent, whose job it is to think about the project at a high level and provide the kind of high level tech design review that AI notoriously can't do well)
I'm testing Astra so I don't have strong opinions yet. I've also done extensive testing of the same skill and subagent pattern in opencode/omp using GLM 5.3, Kimi K3 max, Deepseek V4 pro, Deepseek V4.1 flash, Qwen 3.8 2.4T max, and others.
My experience is that open weights models find between 1/4 to 1/2 of what Fable/Opus stack can find, and often miss the most critical issues. I work where privacy isn't just good behavior, it's enforced by law, and the Fable/Opus stack has found privacy leaks that the openweights stacks don't find.
You can imagine that paying for these Claude runs isn't cheap, each one can eat 25-33% of my 5 hour limit. I am quite desperate for openweights models to be competitive, but at the end of the day, the biggest limit here isn't the price difference between GLM 5.3 max (my current best-in-class choice for open weights, offering Kimi k3 performance for like half the price), it's the cost to the business for shipping lower quality.
Can't wait to dig in more with Astra, I just haven't iterated much on my skill port to codex yet.
One criticsm I have for the article, that is important for my own work, is not simply comparing "bugs found" because these agents can find endless reams of lows and nitpicks that are just ~worthless hardening. I'd be much more interested to see how many critical/high/medium's each test found, not "overall bug count". I also think review is about A LOT more than "finding bugs"...
> I work where privacy isn't just good behavior, it's enforced by law
HIPAA/medical?
They seem to conclude Luna is a better value, but their analysis is dumb. They just break it down to $/bug found.
However, Luna missed 23 bugs that Astra found, and identified 24 bugs that weren't really bugs. That's horrible. Astra had 96% precision.
The cost to care about here isn't just how much it costs to run the code review, or the cost per true-positive. It's the cost of dealing with this system. A code review system that is right about 2/3 sucks, and one that misses another 1/3 of the bugs is also a lot worse. The Astra code review quoted here would become the foundation of how the team works, the Luna version is at best helpful to find some stuff but does not dramatically increase your confidence. It also will force humans or better AI's to have to run down a lot of false positives, and that is treated as free here.
Actual conclusion: The cost for Astra is low in absolute terms compared to the cost of bugs and human attention, and the added value is far far more than the added cost.
To me, one interesting piece of analysis was whether Luna had benefit on top of Astra -- i.e., running both and synthesizing their findings. But even with that, it raises the question whether running a second Astra pass, or Sol, or even a model from another family (GLM? Fable?) would deliver even more additive benefit.
Let's assume for a second that the intelligence is there and these models really are great. Would it make sense to hire CTO of some big-tech company to do code reviews for your startup? Feels kind of like overkill to me. Code review is not about more intelligence. To me, it's about more cultural context. And all things equal, what difference would the model make, at anything above sonnet5 medium level?
IME big models just feel like big models. There's no training or RLing a small model that will encapsulate the "world knowledge" and minutia that a big model will glance from the same training data. So it makes perfect sense to use them where "big picture" is more important - planning, code review, process review (i.e. was what was asked implemented correctly?), etc.