That's like 100,000 USD. I keep thinking about making a rap video wearing a 10 TB gold chain surrounded by big booty girls with their naughty bits covered in m.2 SSD's while dissing the AI industry. Though I cant afford the RAM :-/
It’s sad that I can’t interpret if you mean to actually shoot your rap video on film, or have an AI generate it lol. Either way you’re going to need RAM.
Haha, reminds me off bringing down office mail servers by accidentally creating loops of emails back in the day... What is old is new again, but this time with probabilities :)
Companies are already using on-premise GitHub server, if they are using GitHub in the first place. There are many other self hosted solutions which are quite common in enterprise environment.
What is the quality-first, high uptime alternative to GitHub? My employer uses both GitHub and GitLab, and while I think GitLab is better, its quality also frankly sucks. It's riddled with bugs that have just been marinating on the issue tracker for years, and the most common "fix" for gnarly bugs in the CI platform is "revise the documentation to reflect the existing (broken) behavior".
It's amazing, before we even had ChatGPT, GitLab was building so much endless slop halfbaked crap in their pursuit of ever more "enterprise checkboxes". Now they have slowed right down, no doubt collapsing under the escalating maintenance weight of all the nonsense that was created, like the canaries in the vibe coding mines telling us of impending doom.
Now you go to their blog, theres a banner at the top announcing "GitLab Agentic AI whatever is GA (GENERAL AVAILABILITY)" and you try to click it its literally a fucking 404 not found. That's the level of their stability and quality. Try it for yourself:
*Stupid question*: What is so hard about self hosting one's own repo? I get it must be difficult for a mega corporation, but for companies like us, who have hundreds of repos but only 20 of them are regularly used, and concurrent read/write is relatively light -- considering our largest team is less than 20 persons, so even if all of them are reading/writing from the repo, it doesn't seem to be a huge issue.
Even for a bigger company, say 5x developers (we have about 100+ SWEs and maybe 10-20 other titles who use GitHub), is it really a big thing to self host their own repos? External applications are definitely on another level because you could have hundreds of concurrent visits easily.
> What is so hard about self hosting one's own repo?
Maybe nothing! I was genuinely asking. I still don't know what Actually Good™ forges are out there these days, generally suitable for corporate use in place of the likes of GitHub or GitLab. Forgejo? Something not based on Git?
I guess self-hosting GitHub is the easiest second step for companies that use GitHub? It does have a lot of niceties built around git, which is very crude.
Not really. I've been around a while. Git for about 15 years. Subversion before that. Perforce before that. rcs before that (back down to sun3 machines). Mostly Fossil now for personal things.
What I am saying is that people learn as much as they need to. They generally don't need to know any more git than is required to interact with github. If anything problematic comes up, they go in with a wrecking ball because they don't truly understand what they are doing. And git has a lot of wrecking balls available.
If you threw them at raw git and asked them to collaborate with someone they'd be up shit creek. They have no idea how SSH or email works for example.
Your favorite search engine or LLM will show you in a second, it's really easy.
The problem is that it's not enough. The fact that Github uses Git specifically is a technical detail; it could use mercurial equally easily, as Bitbucket used to. Github Actions, OWNERS files, PRs and review tools, issue tracker, wiki are all not Git features.
Not a chance. I think you need to spend some time in low ball corporate IT. It's just monkeys throwing faeces at the wall. We only just levered them off subversion...
(I use Fossil 100% offline for personal projects for ref)
True, workers can still commit to their local git.
I've been looking into having a separate git server that we can commit to and add plain ole git hooks to, and just having it be synced with github as a clone.
I'd rather recommend Forgejo (a fork of Gitea developed under the auspices of Codeberg e.V.) instead. The way in which Gitea broke the trust of the community seems like it probably should be avoided nowadays.
I mean, there are solutions, but none of them seems to have a large enough mindshare and efficiency. (Even though Github's code review tools are pretty spartan.)
They can be. A PR can be made and code review conducted by submitting a patch to a mailing list. That's how the kernel and, I think, git itself is developed.
CI/CD is really a methodology. It just means integrating/deploying stuff as soon as its ready. So you just need maintainers to be able to run the test suite and deploy, which seems like a really basic thing.
If GH has an issue, it seems to always be around 4pm or 5pm GMT. I'm starting to think that i should avoid any planned production releases around this time.
I did this in 2019, it avoided so many headaches. CI is better too since there's a nice clean mapping of build -> pod for everything and I can just exec in if something's borked.
It's not as bad as you think, I run the helm upgrade when patches come out, the backing store is S3 or managed SQL, it runs a nightly k8s cron called gitlab-backup which tarballs the whole thing into an s3 bucket with a single command restore should disaster strike. (This is part of the product, not a thing I wrote.)
I probably only babysit it for 30 minutes per year, including all the upgrades.
It depends how high you value your headaches, and how high, your org's downtime. Github not working accrues over the hourly rate of every developer affected, which is likely $70-$100 a hour. 10 hours of outage in a year affecting a team of 10 would cost north of $70k, enough to hire a part-time SRE dedicated just to tend to your Gitlab installation.
^ this. the last thing i want is to add to my workload. take my money and make my life easier, even if it means that for one hour every couple months i can't do anything
It's the cloud managed one, they have an "enterprisey" license that gives you more features/limits but you don't have to run infra, upgrades, patches etc
We did hire some, boss! Soshie, Vizzy and Dexter. They're AI, but they're supposed to be way better than a human SRE. At least that's what the Sintra salesguy told us.
Why hire anyone to fix a problem when you can make an AI agent to "fix" it, tell investors about it to pump the price, and not fix anything knowing that you have a monopoly?
Yes we did hire SREs, unfortunately they are in another continent and they only know how to pull others into the chat. We also have some AI too, do you want to try them? They are pretty good SREs, one of them wrote 100K lines of code in a week while another one reviews every line along the way. It was fantastic! Fantastic!! FANTASTIC!!!
You've heard of infrastructure as code, now presenting air strikes as code!
Need a new secret offensive operation? Create a new JSON file with the coordinates, make a merge request and get Commander approval, merge it, and our new proprietary GitHub action runner will deploy a drone in seconds!
This is far too simple. The correct way is to generate an NFT that's a screenshot from Google Maps of where you'd like to hit, and a blockchain-watching AI will spot it, figure out where you probably mean and send the coordinates to the fire control system.
Perhaps the gemini-cli bot arguing with itself is taking its toll
https://github.com/google-gemini/gemini-cli/issues/16750
https://github.com/google-gemini/gemini-cli/issues/16723 is even worse, GitHub shows `5195 remaining items` in the collapsed timeline.
Wow. If you look at all the issues this seems pretty common
https://github.com/google-gemini/gemini-cli/issues?q=is%3Ais...
Wow that's whole a lot of yapping
Jeez, what a mess. Some of those issues have over 5000 events on them.
I really hope that didn't send emails out to people.
I could not resist to put my sarcastic comment about RAM price increases serving a good cause in there.
Having just had to buy 4TB of RAM, I appreciate this.
That's like 100,000 USD. I keep thinking about making a rap video wearing a 10 TB gold chain surrounded by big booty girls with their naughty bits covered in m.2 SSD's while dissing the AI industry. Though I cant afford the RAM :-/
It’s sad that I can’t interpret if you mean to actually shoot your rap video on film, or have an AI generate it lol. Either way you’re going to need RAM.
Shooting on film doesn't need any RAM. Unfortunately the price of silver is also through the roof.
Yep that much. 64Gb DDR5 ECC sticks (128Gb don't exist at the moment apparently). They declined the PO 6 months ago. That'll teach 'em.
I was pissed that there weren't any sticks heading to the recycling out of the nodes otherwise I would make myself that chain :)
Use Veo
like most rap videos do with cars/jets/mansions, just rent the ram sticks for a few hours!
And the cinema equipment to make the video itself.
Maybe the bots need rule of ko.
https://en.wikipedia.org/wiki/Rules_of_Go#Ko
Haha, reminds me off bringing down office mail servers by accidentally creating loops of emails back in the day... What is old is new again, but this time with probabilities :)
Wonderful, lmao.
I suspect the migration to Azure is continuing to go well
Yes indeed. 6 years of non-stop outages across the platform every month.
Even self-hosting would have been more stable than sitting on GitHub as predicted more than half a decade ago. [0]
Now there is no 'CEO of GitHub' to contact this time (Satya does not care).
[0] https://news.ycombinator.com/item?id=22867803
This feels more like Copilot-as-platform-engineer to me
Github's been running on vibe code for a while now and it's starting to show
I did not come to hacker news expecting comedy gold but you have done it my friend!
Github's recent reliability has honestly been abysmal. Not surprised.
Unless some major customers are moving away, I don't think they are going to seriously care about it.
I suspect some companies may already be considering it. Especially with the wealth of alternatives today.
Companies are already using on-premise GitHub server, if they are using GitHub in the first place. There are many other self hosted solutions which are quite common in enterprise environment.
In my experience companies are moving into GitHub for Copilot and GHA.
GHA maybe, but copilot is just another mid tier player in a congested space.
Doesn't stop folks from wanting to buy the MS brand. Execs are really out of touch these days.
These days? lol
It's a cheap mid tier player. You get more tokens per dollar.
What kind of alternatives do you see as viable for large(ish) commercial users?
GitHub on-prem. Officially called GitHub Enterprise Server. You can have GitHub, but hosted on your own servers.
So you still pay them, you do the hosting work, and you get a product with worse features than gitlab?
Yeah, at that point why would anyone choose GitHub?
And it costs you more money than GitLab.
but you can be smug when theres a github incident, and thats hard to put a price on
You can do that with gitlab.
What is the quality-first, high uptime alternative to GitHub? My employer uses both GitHub and GitLab, and while I think GitLab is better, its quality also frankly sucks. It's riddled with bugs that have just been marinating on the issue tracker for years, and the most common "fix" for gnarly bugs in the CI platform is "revise the documentation to reflect the existing (broken) behavior".
It's amazing, before we even had ChatGPT, GitLab was building so much endless slop halfbaked crap in their pursuit of ever more "enterprise checkboxes". Now they have slowed right down, no doubt collapsing under the escalating maintenance weight of all the nonsense that was created, like the canaries in the vibe coding mines telling us of impending doom.
Now you go to their blog, theres a banner at the top announcing "GitLab Agentic AI whatever is GA (GENERAL AVAILABILITY)" and you try to click it its literally a fucking 404 not found. That's the level of their stability and quality. Try it for yourself:
https://about.gitlab.com/blog/
Maybe it's GU already.
*Stupid question*: What is so hard about self hosting one's own repo? I get it must be difficult for a mega corporation, but for companies like us, who have hundreds of repos but only 20 of them are regularly used, and concurrent read/write is relatively light -- considering our largest team is less than 20 persons, so even if all of them are reading/writing from the repo, it doesn't seem to be a huge issue.
Even for a bigger company, say 5x developers (we have about 100+ SWEs and maybe 10-20 other titles who use GitHub), is it really a big thing to self host their own repos? External applications are definitely on another level because you could have hundreds of concurrent visits easily.
What did I miss?
> What is so hard about self hosting one's own repo?
Maybe nothing! I was genuinely asking. I still don't know what Actually Good™ forges are out there these days, generally suitable for corporate use in place of the likes of GitHub or GitLab. Forgejo? Something not based on Git?
I guess self-hosting GitHub is the easiest second step for companies that use GitHub? It does have a lot of niceties built around git, which is very crude.
[dead]
Good thing git is a distributed system
Virtually no one knows how to do anything with it outside of github.
You might be surprised, but that's not true at all.
I once read someone commenting "Nobody writes code by hand without looking syntax up".
Man, you are just outing yourself as a complete beginner, the field is way deeper than you imagine and it's not even close.
Not really. I've been around a while. Git for about 15 years. Subversion before that. Perforce before that. rcs before that (back down to sun3 machines). Mostly Fossil now for personal things.
What I am saying is that people learn as much as they need to. They generally don't need to know any more git than is required to interact with github. If anything problematic comes up, they go in with a wrecking ball because they don't truly understand what they are doing. And git has a lot of wrecking balls available.
If you threw them at raw git and asked them to collaborate with someone they'd be up shit creek. They have no idea how SSH or email works for example.
Your favorite search engine or LLM will show you in a second, it's really easy.
The problem is that it's not enough. The fact that Github uses Git specifically is a technical detail; it could use mercurial equally easily, as Bitbucket used to. Github Actions, OWNERS files, PRs and review tools, issue tracker, wiki are all not Git features.
Not a chance. I think you need to spend some time in low ball corporate IT. It's just monkeys throwing faeces at the wall. We only just levered them off subversion...
(I use Fossil 100% offline for personal projects for ref)
That's a them problem.
i still find insightful ways to use git every day. amazing tool. it's a shame for those who only see it as "how to sync my repo with my coworkers"
True, workers can still commit to their local git.
I've been looking into having a separate git server that we can commit to and add plain ole git hooks to, and just having it be synced with github as a clone.
Check out Gitea. Its kind of a clone of github but you can self host.
I'd rather recommend Forgejo (a fork of Gitea developed under the auspices of Codeberg e.V.) instead. The way in which Gitea broke the trust of the community seems like it probably should be avoided nowadays.
Git is!
PRs and code review are not. CI/CD is not.
I mean, there are solutions, but none of them seems to have a large enough mindshare and efficiency. (Even though Github's code review tools are pretty spartan.)
> PRs and code review are not. CI/CD is not.
They can be. A PR can be made and code review conducted by submitting a patch to a mailing list. That's how the kernel and, I think, git itself is developed.
CI/CD is really a methodology. It just means integrating/deploying stuff as soon as its ready. So you just need maintainers to be able to run the test suite and deploy, which seems like a really basic thing.
You'd be surprised how far a lot of places got just using git notes and jenkins for a very long time.
If GH has an issue, it seems to always be around 4pm or 5pm GMT. I'm starting to think that i should avoid any planned production releases around this time.
helm repo add gitlab https://charts.gitlab.io/ && helm upgrade --install gitlab gitlab/gitlab
I did this in 2019, it avoided so many headaches. CI is better too since there's a nice clean mapping of build -> pod for everything and I can just exec in if something's borked.
Things would have to get really bad before I considered managing my own repositories. Trading someone else's headaches for my own.
It's not as bad as you think, I run the helm upgrade when patches come out, the backing store is S3 or managed SQL, it runs a nightly k8s cron called gitlab-backup which tarballs the whole thing into an s3 bucket with a single command restore should disaster strike. (This is part of the product, not a thing I wrote.)
I probably only babysit it for 30 minutes per year, including all the upgrades.
It depends how high you value your headaches, and how high, your org's downtime. Github not working accrues over the hourly rate of every developer affected, which is likely $70-$100 a hour. 10 hours of outage in a year affecting a team of 10 would cost north of $70k, enough to hire a part-time SRE dedicated just to tend to your Gitlab installation.
>10 hours of outage in a year affecting a team of 10 would cost north of $70k
10 hours x 10 developers x $70 per hour = $7000, not $70000.
Thank you for the correction! This indeed completely changes the picture :-\
[dead]
^ this. the last thing i want is to add to my workload. take my money and make my life easier, even if it means that for one hour every couple months i can't do anything
Have you ever actually hosted gitlab?
Not only have I hosted it, I've helped migrate two gitlab instances to github enterprise, because we didn't want to maintain it anymore
Don't you still have to maintain github enterprise?
It's the cloud managed one, they have an "enterprisey" license that gives you more features/limits but you don't have to run infra, upgrades, patches etc
Microsoft, it's time to hire some SREs.
We did hire some, boss! Soshie, Vizzy and Dexter. They're AI, but they're supposed to be way better than a human SRE. At least that's what the Sintra salesguy told us.
So that's what the Tay, and Zoe AI bots were doing all this time after they were cancelled and banned off of Twitter.
Working on the GitHub Azure migration and for years it's gone so well so far.
I hope someone’s been reviewing their work in case they’ve been adding certain german related Easter eggs
Why hire anyone to fix a problem when you can make an AI agent to "fix" it, tell investors about it to pump the price, and not fix anything knowing that you have a monopoly?
Microsoft doesn't pay well enough to attract good SRE talent.
Clippy to the rescue! :-)
Yes we did hire SREs, unfortunately they are in another continent and they only know how to pull others into the chat. We also have some AI too, do you want to try them? They are pretty good SREs, one of them wrote 100K lines of code in a week while another one reviews every line along the way. It was fantastic! Fantastic!! FANTASTIC!!!
OK I have no idea about MSFT SREs, just to be /s.
Days since last GitHub incident: 0.2
14 incidents this month. So far.
And it is January 16. Jeez.
I believe it is an Azure outage or some type of MS service - everything on Azure is down.
My az services seem to be up.
having no issues on azure here, seeing no azure incidents on the status page or any of my admin panels
> seeing no azure incidents on the status page
… in all seriousness, that is hardly proof that Azure isn't having an outage.
if i thought it alone was proof enough, i wouldnt have also included the bit about how i was actively using azure.
its one signal, among others. and in any case, i wasn't trying to prove the parent commenter wrong. i was offering my own signal to the crowd.
I second this. Not experiencing any Azure issues at this time.
This is why companies should host their own source code on-prem.
Angry unicorns seem to be over.
Fixed in about 30m to an hour.
Definitely annoying, but I'll try the hot take that, contrary to popular belief, GH is not critical infrastructure - or so I hope.
Please tell me no part of the Ukrainian air defense system depends on a gh action hook.
You've heard of infrastructure as code, now presenting air strikes as code!
Need a new secret offensive operation? Create a new JSON file with the coordinates, make a merge request and get Commander approval, merge it, and our new proprietary GitHub action runner will deploy a drone in seconds!
This is far too simple. The correct way is to generate an NFT that's a screenshot from Google Maps of where you'd like to hit, and a blockchain-watching AI will spot it, figure out where you probably mean and send the coordinates to the fire control system.
When millions of man-hours are lost waiting for your service to be back up, I think that deserves a bit of resiliency.
It's not critical, but there's still a lot of reliance on it.
It's also the only reason why I still need IPv4.
The status page says things are still not fixed.