This stance is a breath of fresh air. In my experience change management is the first thing to slap on when a bad release happens.
I've worked at a large enterprise that have a "Change Advisory Board", that you need to convince when you want to bump the major version on your linter. It has the effect of velocity slowing down to a crawl. Changes have to large, since otherwise it wouldn't be approved by the CAB. A slow mess.
At my current place we have to loudly declare "I CONFIRM COMPLIANCE" in every PR description. I'm not sure that anyone knows why, but it keeps the bureaucrats happy. Shrug
My favorite thing about CABing things that shouldn't be CABd is that it then bunches up changes that all happen at the same time... after the CAB meeting. Now if something breaks, it could be any of the dozens of things that just got pushed in the 15 minutes after the meeting.
Youre required to have a policy. That policy may be throwing bananas at the wall, but if it's documented and you follow it, you're compliant with policy.
The better way is that for each policy you look at what do you actually want to do and how you want to do it, and then write that down as the policy. Now the policy makes sense because it's how you wanted to do it anyway.
I've set up policies and processes from the ground up for SOC2 audits in startups, that's how I do it.
I was in a high trust environment that didn't use dual auth on some things. They lost $250k to embezzlement.
High trust is high trust until someone exploits it, and the likelihood of encountering an exploit (embezzlement, fraud, political speech, etc.) approaches 100 as the total number of staff hours increase.
This is easy for Amp Code to do but not so easy for other people with different types of businesses.
I started at a business doing millions of customer transactions worth 100s of millions of dollars and their whole process was a complete YOLO Playground. As a result they lost money, people (it is high stress if everything breaks all the time) and most importantly: customer confidence.
What brought back some sanity? The basics for a mature business: deployment reviews and approvals, removing prod access, code reviews, QA in the loop. Release notes. All these things come up in audits. In SOC1 because the customer wants assurances, in SOC2 because we need to show prospects that we are serious and mature enough to get their business.
Please don't think it is just theatre. If you get away with bending the rules to the bare minimum to get a checkmark then good for you and all the power to you.
But if you are running a business where you are dealing with critical systems, PII, customer data, money, etc. Well maybe being very strict about having good change management policies in place will actually help you mature.
That maturity will also allow you to get customers. Because guess what, most customers who are going to trust you with their business have some expectations about how you run your ship.
I wish my company was as "simple" as Amp. No disrespect, I love what they do. But it doesn't really look like the kind of business where you get in serious trouble for using a minimal process with high trust in people.
Think about that when you read this. What could go wrong in your business. Don't project Amp's process on yours if your business is different.
> What they ask for is that changes are authorized, tested, approved, and recorded — and pull requests are just one way of doing that.
I'm pretty unclear how approvals are ensured in your system?
Authorisation and recording are pretty much out-of-the-box with a version control system, CI can guarantee testing, but how are we demonstrating approval without any evidence of change reviews?
Restricted push access. Access to main follows business function: every engineer at Amp can push, and most of Amp is engineers. But the percentage of people with access matters less than being able to explain exactly who has it and why.
So like you know exactly who can push the code and let's say random from marketing can't push code because he is not authorized.
This. My understanding is that SOC2 does require approvals (eg 2nd person reviews/approves the code change). This approach doesn't seem to include that step. Is that not a requirement?
SOC2 requires basically nothing beyond that you follow the procedures that you say you're going to follow. The relevant text of CC8.1 is:
> The entity authorizes, designs, develops or acquires, configures, documents, tests, approves and implements changes to its infrastructure, data, software, and procedures to meet its objectives.
You'll note that this doesn't actually say much of anything.
It's very common for companies which are working on SOC 2 compliance to write down that all changes will be reviewed and once you do that you're required to actually follow through and do it, but you don't have to write down that all changes will be reviewed. The point of SOC2 is mostly that if you are vibing slop into production you have to document that fact (or rather, document your lack of change controls that make it possible) and so your customers can be aware of that. Large conservative customers may insist on more rigid processes as a condition of buying from you.
Every company is different, with different risks but here is the change management i have implemented successfully.
Minor, medium, major change category depending on potential impact with documented guidelines when each apply.
First release of a service always recommend to be major. (They are few vs daily changes and can be redtaped more to get many stakeholders requirements. Functional and non functional) Author of the change and his team decide the category.
- Minor need one more reviewer outside author. Team decide everything.
- Medium need min one more, and usually QA team or other affected team.
- Major change need CAB meeting with all major stakeholders to sign off.
CAB is responsible for the process and high level monitor the program, see if any cheating goes on, performance metrics such as number of changes per category etc. Help support people what could be considered minor vs major changes.
Overall rel flexible and balanced. Pushes people towards making smaller changes with less risk and at the same time increase velocity while still protect reasonable against bad actors, mistakes etc.
It is not perfect, medium changes tends to be more difficult to coordinate, some changes are missclassified etc but that is okey. It is not only about the process, communication matters and ci/cd have to be robust etc.
Engineers often over-think compliance. SOC2 is regulatory capture, and an audit that tests your controls. You define the appropriate controls. As long as you do what you say you're going to do, you should pass the audit. It's not rocket science. It's a feature of a correct system that it is auditable. How easy it is to audit is really a function of your maturity. SOC2 and its ilk such as ISO27001 are just maturity signalling mechanisms. Stop overthinking it or applying black and white rules -- in actual practice its always shades of grey and most auditors are just happy to have an engaged and switched on team to be auditing, vs someone who treats it as adversarial. You're paying them!
The problem is it's not the engineers that overthink it. The requirement for soc2 usually comes with the first "serious" customer. It is usually a big blocker on some fat contract and now the business makes it your problem for the next 6 months.
So what do you do? You engage and some 3rd party 1800-need-soc2 clowns which will hold your hand and implement all the cookie cutter solutions they know will make auditor happy (oh and btw, they know the auditor personally).
I went into reading this not knowing what amp makes, and at the end of the article was like "see now this sounds like a company who absolutely wouldn't fuck with vibe coding"...
Then I realized what amp makes and had an extremely visceral reaction.
Y'all just running your business on hopes and dreams out here.
Does there exist any tool to analyze PR feedback quality and usefulness?
I've been trying to explain to compliance employees that if the majority of PR reviews anyway is just "LGTM - Just merge" (I don't care), what's the actual value of PRs? It's just facade.
On a different note, we're looking into artifact attestations and admissions through Sigstore, which is solves many of the same challenges but through different means.
So an engineer who knows your codebase and tests can sneak in malicious code/backdoors because you're high trust.
And I am guessing you'll extend that high trust to your agents/'orbs' next. And I am sure you'll find an auditor who'll go with it coz frankly most don't care.
I am not surprised there are folks willing to do this, but I am surprised that you feel you must brag about it. And you ARE bragging when you title the post the way you did. Good luck.
No, that's not a brag but a suggestion to take things into context and not apply soc2 as a cookie cutter solution where a 20k employee enterprise and a 20 people startup must share commonalities.
In a 20 people startup it's very likely that most engineers have access to production anyway and can inject malicious stuff directly, so PRs stop no one really.
> So an engineer who knows your codebase and tests can sneak in malicious code/backdoors because you're high trust.
What is the alternative? Bumping a dependency in a PR and getting a LGTM can also introduce a backdoor.
That's another form of high trust: your trusting the publisher of the dependency.
High trust comes with high responsibility, which is easier to enforce when the trusted party is an individual on your team with generally aligned incentives, rather than an organization or unpaid individual on the internet serving many.
Maybe if your code is modular enough, you can have module owners and organize things like a free software projects with the owners as maintainers. The maintainers could merge without PRs while others would go through a PR process with the maintainer doing the review.
This stance is a breath of fresh air. In my experience change management is the first thing to slap on when a bad release happens.
I've worked at a large enterprise that have a "Change Advisory Board", that you need to convince when you want to bump the major version on your linter. It has the effect of velocity slowing down to a crawl. Changes have to large, since otherwise it wouldn't be approved by the CAB. A slow mess.
At my current place we have to loudly declare "I CONFIRM COMPLIANCE" in every PR description. I'm not sure that anyone knows why, but it keeps the bureaucrats happy. Shrug
My favorite thing about CABing things that shouldn't be CABd is that it then bunches up changes that all happen at the same time... after the CAB meeting. Now if something breaks, it could be any of the dozens of things that just got pushed in the 15 minutes after the meeting.
A declaration of fault from the executives' perspective if anything goes wrong
Youre required to have a policy. That policy may be throwing bananas at the wall, but if it's documented and you follow it, you're compliant with policy.
That's a good way to kill off motivated employees.
"We don't know why we're doing it - it's just mandatory".
Only if you write a bad policy, so don't do that.
The better way is that for each policy you look at what do you actually want to do and how you want to do it, and then write that down as the policy. Now the policy makes sense because it's how you wanted to do it anyway.
I've set up policies and processes from the ground up for SOC2 audits in startups, that's how I do it.
Smart employees totally understand: "we do it because it makes it easier to explain to auditors, customers, new employees, non technical managers".
The policy can be changed.
I was in a high trust environment that didn't use dual auth on some things. They lost $250k to embezzlement.
High trust is high trust until someone exploits it, and the likelihood of encountering an exploit (embezzlement, fraud, political speech, etc.) approaches 100 as the total number of staff hours increase.
This is easy for Amp Code to do but not so easy for other people with different types of businesses.
I started at a business doing millions of customer transactions worth 100s of millions of dollars and their whole process was a complete YOLO Playground. As a result they lost money, people (it is high stress if everything breaks all the time) and most importantly: customer confidence.
What brought back some sanity? The basics for a mature business: deployment reviews and approvals, removing prod access, code reviews, QA in the loop. Release notes. All these things come up in audits. In SOC1 because the customer wants assurances, in SOC2 because we need to show prospects that we are serious and mature enough to get their business.
Please don't think it is just theatre. If you get away with bending the rules to the bare minimum to get a checkmark then good for you and all the power to you.
But if you are running a business where you are dealing with critical systems, PII, customer data, money, etc. Well maybe being very strict about having good change management policies in place will actually help you mature.
That maturity will also allow you to get customers. Because guess what, most customers who are going to trust you with their business have some expectations about how you run your ship.
I wish my company was as "simple" as Amp. No disrespect, I love what they do. But it doesn't really look like the kind of business where you get in serious trouble for using a minimal process with high trust in people.
Think about that when you read this. What could go wrong in your business. Don't project Amp's process on yours if your business is different.
> What they ask for is that changes are authorized, tested, approved, and recorded — and pull requests are just one way of doing that.
I'm pretty unclear how approvals are ensured in your system?
Authorisation and recording are pretty much out-of-the-box with a version control system, CI can guarantee testing, but how are we demonstrating approval without any evidence of change reviews?
I think you mistake 'authorized' with 'approved'.
Authorized is:
Restricted push access. Access to main follows business function: every engineer at Amp can push, and most of Amp is engineers. But the percentage of people with access matters less than being able to explain exactly who has it and why.
So like you know exactly who can push the code and let's say random from marketing can't push code because he is not authorized.
No, I understand that. Authorisation is a basic feature of pretty much any hosted version control system.
What is less clear here is approvals.
> What is less clear here is approvals
This. My understanding is that SOC2 does require approvals (eg 2nd person reviews/approves the code change). This approach doesn't seem to include that step. Is that not a requirement?
SOC2 requires basically nothing beyond that you follow the procedures that you say you're going to follow. The relevant text of CC8.1 is:
> The entity authorizes, designs, develops or acquires, configures, documents, tests, approves and implements changes to its infrastructure, data, software, and procedures to meet its objectives.
You'll note that this doesn't actually say much of anything.
It's very common for companies which are working on SOC 2 compliance to write down that all changes will be reviewed and once you do that you're required to actually follow through and do it, but you don't have to write down that all changes will be reviewed. The point of SOC2 is mostly that if you are vibing slop into production you have to document that fact (or rather, document your lack of change controls that make it possible) and so your customers can be aware of that. Large conservative customers may insist on more rigid processes as a condition of buying from you.
Amp will probably claim the agent wrote the code and the human approved it
Every company is different, with different risks but here is the change management i have implemented successfully.
Minor, medium, major change category depending on potential impact with documented guidelines when each apply.
First release of a service always recommend to be major. (They are few vs daily changes and can be redtaped more to get many stakeholders requirements. Functional and non functional) Author of the change and his team decide the category.
- Minor need one more reviewer outside author. Team decide everything.
- Medium need min one more, and usually QA team or other affected team.
- Major change need CAB meeting with all major stakeholders to sign off.
CAB is responsible for the process and high level monitor the program, see if any cheating goes on, performance metrics such as number of changes per category etc. Help support people what could be considered minor vs major changes.
Overall rel flexible and balanced. Pushes people towards making smaller changes with less risk and at the same time increase velocity while still protect reasonable against bad actors, mistakes etc.
It is not perfect, medium changes tends to be more difficult to coordinate, some changes are missclassified etc but that is okey. It is not only about the process, communication matters and ci/cd have to be robust etc.
This balanced approach works for many.
Engineers often over-think compliance. SOC2 is regulatory capture, and an audit that tests your controls. You define the appropriate controls. As long as you do what you say you're going to do, you should pass the audit. It's not rocket science. It's a feature of a correct system that it is auditable. How easy it is to audit is really a function of your maturity. SOC2 and its ilk such as ISO27001 are just maturity signalling mechanisms. Stop overthinking it or applying black and white rules -- in actual practice its always shades of grey and most auditors are just happy to have an engaged and switched on team to be auditing, vs someone who treats it as adversarial. You're paying them!
The problem is it's not the engineers that overthink it. The requirement for soc2 usually comes with the first "serious" customer. It is usually a big blocker on some fat contract and now the business makes it your problem for the next 6 months.
So what do you do? You engage and some 3rd party 1800-need-soc2 clowns which will hold your hand and implement all the cookie cutter solutions they know will make auditor happy (oh and btw, they know the auditor personally).
I went into reading this not knowing what amp makes, and at the end of the article was like "see now this sounds like a company who absolutely wouldn't fuck with vibe coding"...
Then I realized what amp makes and had an extremely visceral reaction.
Y'all just running your business on hopes and dreams out here.
Does there exist any tool to analyze PR feedback quality and usefulness?
I've been trying to explain to compliance employees that if the majority of PR reviews anyway is just "LGTM - Just merge" (I don't care), what's the actual value of PRs? It's just facade.
On a different note, we're looking into artifact attestations and admissions through Sigstore, which is solves many of the same challenges but through different means.
[dead]
AICPA should have spent 5 seconds on a web search to find out that SoC means System on a Chip, and chosen something else.
So an engineer who knows your codebase and tests can sneak in malicious code/backdoors because you're high trust.
And I am guessing you'll extend that high trust to your agents/'orbs' next. And I am sure you'll find an auditor who'll go with it coz frankly most don't care.
I am not surprised there are folks willing to do this, but I am surprised that you feel you must brag about it. And you ARE bragging when you title the post the way you did. Good luck.
No, that's not a brag but a suggestion to take things into context and not apply soc2 as a cookie cutter solution where a 20k employee enterprise and a 20 people startup must share commonalities.
In a 20 people startup it's very likely that most engineers have access to production anyway and can inject malicious stuff directly, so PRs stop no one really.
Yeah, saying it's verified with tests is kinda moot when the code change can just modify the same tests or verifications that will be run on it.
> So an engineer who knows your codebase and tests can sneak in malicious code/backdoors because you're high trust.
What is the alternative? Bumping a dependency in a PR and getting a LGTM can also introduce a backdoor.
That's another form of high trust: your trusting the publisher of the dependency.
High trust comes with high responsibility, which is easier to enforce when the trusted party is an individual on your team with generally aligned incentives, rather than an organization or unpaid individual on the internet serving many.
If you have a malicious engineer I bet he will come up with idea to make a PR at a time or in a way that someone will just LGTM the PR.
Soc2 is pointless.
But how do they review each other's work? Prs are indispensable for collaboration...
Working at Amp
We...just read the commits, and talk to each other.
We're also pretty trigger happy with the Huddle button in Slack.
Nobody on the team would go back to mandatory PRs
Do you read them before someone pushes them?
We‘ve had code reviews before GitHub existed. PRs are a tool. And just one tool among many.
They (companyin the article) don't, though, hence the question. Is there even one set of eyes on this code, given the amount of agents they use?
Sure, which is why i asked
Maybe if your code is modular enough, you can have module owners and organize things like a free software projects with the owners as maintainers. The maintainers could merge without PRs while others would go through a PR process with the maintainer doing the review.
You can pair program.
there goes the neighborhood