We were playing around with automating, standardizing, and optimizing more of our software development process, but didn’t have trustworthy priors for deciding what to use. Public benchmarks get aggressively optimized against, and their tasks don’t necessarily resemble what we consider a unit of work. So we built infrastructure to test combinations of models, harnesses, skills, reasoning effort, and ticket styles against work we had already completed.
The combinations mattered. A harness or skill could help one model and hurt another, more reasoning sometimes reduced quality, and rewriting the ticket changed which setups performed best. We also monitored agent behavior using a simplified version of https://openai.com/index/how-we-monitor-internal-coding-agen....
We have a gripe with hidden-test pass rates as a measure of code quality. Philosophically, we’re closer to FrontierCode’s focus on mergeability (https://cognition.com/blog/frontier-code): we assess the mergeability of the contribution through deterministic checks and specialized agents examining specific requirements.
The article shows a small sample from our codebase and how a router driven by those findings held up on additional tasks, achieving substantial cost savings at similar quality. These are directional, low-N findings specific to our work. We’re sharing the approach because we think other teams could benefit from building their own benchmarks.
More broadly, we think engineering teams should be able to build their own ways of working from composable tools. We’d like the software factory itself to be something engineers can shape, inspect, and improve, including what they automate and where they retain human judgment. There’s room for engineering taste there, too, and we think it starts with being able to assess the component parts.
we had the same observation regarding a skill being highly dependent on the model that is running ot, so very interesting to see the harness is actually also a factor.
great take on the benchmarks results can offset its work in the long run in dollar & time cost.
We were playing around with automating, standardizing, and optimizing more of our software development process, but didn’t have trustworthy priors for deciding what to use. Public benchmarks get aggressively optimized against, and their tasks don’t necessarily resemble what we consider a unit of work. So we built infrastructure to test combinations of models, harnesses, skills, reasoning effort, and ticket styles against work we had already completed.
The combinations mattered. A harness or skill could help one model and hurt another, more reasoning sometimes reduced quality, and rewriting the ticket changed which setups performed best. We also monitored agent behavior using a simplified version of https://openai.com/index/how-we-monitor-internal-coding-agen....
We have a gripe with hidden-test pass rates as a measure of code quality. Philosophically, we’re closer to FrontierCode’s focus on mergeability (https://cognition.com/blog/frontier-code): we assess the mergeability of the contribution through deterministic checks and specialized agents examining specific requirements.
The article shows a small sample from our codebase and how a router driven by those findings held up on additional tasks, achieving substantial cost savings at similar quality. These are directional, low-N findings specific to our work. We’re sharing the approach because we think other teams could benefit from building their own benchmarks.
More broadly, we think engineering teams should be able to build their own ways of working from composable tools. We’d like the software factory itself to be something engineers can shape, inspect, and improve, including what they automate and where they retain human judgment. There’s room for engineering taste there, too, and we think it starts with being able to assess the component parts.
we had the same observation regarding a skill being highly dependent on the model that is running ot, so very interesting to see the harness is actually also a factor.
great take on the benchmarks results can offset its work in the long run in dollar & time cost.