This morning we used 'AGI' to build 3 features for our CRM. We then ran abloh.dev on the PR and found that 64 of the bugs that abloh planted had passed Astra's test suite. This is what slipped past Astra and the (avoided) consequences:
1. A blind spot only abloh's AI found
The test data Astra used gives every fake company a name identical to its web domain.
When abloh's AI swapped name and domain in Astra's query, all tests still passed.
2. A feature that switches itself off
Astra hooked its reminders into an hourly job no test runs.
Deleting Astra's 3 lines caused reminders to stop forever and the CI remained green.
3. Emails from a paused mailbox
The app pauses a mailbox when bounces or spam complaints spike.
We flipped a || to && and emails held overnight go out from it anyway
abloh ranked it the most severe of 27 findings on that PR, in both runs.
I firmly believe that having an independent checker in one's stack will grow in importance over time, in line with model improvement, as diff sizes exceed human oversight capacity under shipping deadlines. Not to mention devs becoming more complacent as trust in models increases. Trusting anything to one shot implementation is unrealistic and poor engineering discipline.
This morning we used 'AGI' to build 3 features for our CRM. We then ran abloh.dev on the PR and found that 64 of the bugs that abloh planted had passed Astra's test suite. This is what slipped past Astra and the (avoided) consequences:
1. A blind spot only abloh's AI found The test data Astra used gives every fake company a name identical to its web domain. When abloh's AI swapped name and domain in Astra's query, all tests still passed.
2. A feature that switches itself off Astra hooked its reminders into an hourly job no test runs. Deleting Astra's 3 lines caused reminders to stop forever and the CI remained green.
3. Emails from a paused mailbox The app pauses a mailbox when bounces or spam complaints spike. We flipped a || to && and emails held overnight go out from it anyway abloh ranked it the most severe of 27 findings on that PR, in both runs.
I firmly believe that having an independent checker in one's stack will grow in importance over time, in line with model improvement, as diff sizes exceed human oversight capacity under shipping deadlines. Not to mention devs becoming more complacent as trust in models increases. Trusting anything to one shot implementation is unrealistic and poor engineering discipline.