It's actually fun to see this. Running systems in a lot of different ways are just interesting. I do however get kinda sad at the hate at k8s because it's really good at what it does.
I've seen so many projects bending over backwards to avoid k8s and pay large cloud bills to avoid it at all costs. (ECS and app services are hopelessly expensive and bothersome)
K8s is really good, pretty easy to maintain, but a bit hard to understand. Mostly because distributed, zero down time systems are a bit hard to get by nature. But if you have someone that wants to take it on I've managed k8s clusters, solo, without incident, while doing lots of other stuff too (working with larger teams now though). Not to mention there's a lot of competence out there that can take over if I'd move on. Most of the deep complexity comes with more advance use cases, that wont show up for smaller deployments.
That said, no h8 towards going your own way! If your a solo developer (or small team) for a smallish project, don't feel the absolute need. If you get to the point you need it you should be earning enough to start paying someone to help ya get your app to a distributed system like k8s.
I think it's good to invest the time in understanding k8s though as a professional. Even if you won't directly run it it teaches you a lot about how to think about distributed, zero downtime systems. And what requirements that puts on an app.
Balanced takes like this are the only reason I still come back to HN. A shot at K3S in my homelab is in my backlog but young kids have set me back. If you have any material that touches on what you said above I'd appreciate a link. So much crap out there which is just Hello World blogspam.
I see stuff like Azure Container Applications like an easy introduction to the world of k8s. Sane defaults, looks managed but you can still get familiar with all the important concepts, and sooner or later you can move to a proper Kubernetes setup while having mastered the ideas behind it. If needed, of course.
I was surprised that there was no mention of two things:
1. Retrying a non-idempotent request on a failure type that does indicate that no action was taken is not necessarily safe.
2. It’s possible and actually fairly common to design a backend that can do a clean shutdown: it stops accepting new requests, completes old requests, then exits. I sincerely hope that Docker’s tooling is good enough for a service to unregister itself before it actually stops accepting requests, but I’m not actually very familiar with using Docker to manage HTTP routing. (I use a home grown tool that is far simpler.)
I used to be critical of almost everyone who used it. Now that I've been learning the ecosystem for a couple years, I'm not as angry about it, and it doesn't affect me as much personally.
I still think almost all of my complaints were valid:
- not just container orchestration, more like a whole AWS solution (its own routing)
- has single-points of failure that would be avoided if you just used aws
- unnecessary complexity for almost all use cases
- ecosystem easily allows you to 2x-8x that complexity (helm+argocd+istio+carpenter+dozens more=hundreds of new failure modes)
- fundamentally moves many teams to a "I don't understand why my service is crashing, so let's just bring up more nodes every time rather than ever learning to debug it" mentality.
- All the feature teams that are supposed to "own their own kubernetes implementation for their own services" never do.
Of course a lot of that frustration is more at working at a place with a poorly-operated SOA where it takes a half-dozen services to send an email to a client or send a text or something silly. It sure is a waste of runway to obsess over a SOA when you aren't profitable yet.
But at this point it's sort of a sunk cost because it's become the industry standard. And AI can help with 90% of the complexity, which is it's own yellow-flag, but here we are.
I’m one of those top commenters who used to be a K8 naysayer. I’m also definitely not young blood.
The reason my position has changed is because:
1. The tooling has gotten better for setting up and managing K8.
2. In two of the last 3 jobs where we opted for a simplified alternative to k8, we came to regret that decision within a couple of years of that decision being made. If you’re core architecture is changing on a timescale of months (not years) then you picked the wrong foundations to build from.
That all said, I still think there is a pragmatic decision that needs to be made. And if I were in the author of this articles position I probably wouldn’t have picked k8s for this task either, despite what I said above. But, and as I said in my comment dismissing this article, they are dealing with low traffic and none of the problems that lend themselves to the benefits of k8. So my criticism of this article is that it’s misleading because their problem is easy but they’re writing as if they’re having to deal
With problems of scale when they’re actually not.
I‘ve built this anti-k8s stance pre-LLMs, and just realized that, actually, agents should be pretty helpful in dealing with it? Is avoiding kubernetes still advisable for projects that will likely never use its full complexity, given how easy it is to maintain now?
Kube and helm are ideal for LLM usage, they make what is a fragile kind of thing into declarative, same as terraform for infra at the slightly lower level.
That isn’t a lot. You could easily run that from one host. The reason people reach for Kubernetes (and similar) is because they need to scale past that single host dependency.
You could, but they don't, meaning their argument is still sound (whether they _could_ use a single host is besides the point, they're not doing that).
They’re multi-region, but that doesn’t mean they’re running across multiple hosts in each region.
Docker compose doesn’t support pooling multiple hosts, so if they are running multiple hosts per region then there’s a lot more complexity to their setup than they’re documenting in that blog. Even if that complexity is human toil managing each host as a separate entity.
The reason most people reach for Kubernetes is because it's cool. The entire infra the vast majority of Kubernetes users have could run on a single bare metal machine with a second one for redundancy.
To be fair: using Kubernetes anyways builds the skill just in case you become one of the 0.1% who actually need it down the line.
You can hire an Azure or Google Kubernetes devops guy and he will be equally comfortable on your AWS EKS kubernetes cluster. And when he leaves, you don't have a six week onboarding process with the new guy to learn all the ins and outs of your totally bespoke, non-standard container orchestration system that was cobbled together by two devs with no operations experience.
K3S takes about 5 minutes to setup the first time and you instantly have an entire universe of standardized operational tooling. I wouldn't touch docker compose with a 20 foot pole for production work.
As soon as you work in a team, it’s irrelevant whether the project actually needs it. There will be someone who convinces stakeholders that it is necessary and then you just have to fall in line and learn the skills knowing that it is most likely one of the 99.9% of projects where it is just overkill.
Until your project has some success, and it turns out all those "complex" features actually turn out to be extremely useful.
Which is exactly what is happening with us, too bad we didn't choose K8S from the get-go and stuck with a "simpler" tool (gaining very little in the process).
Your rebuttal to the parent claiming that almost nobody needs k8s is to bring up a workload almost nobody runs? It seems to me like your argument reinforces the parent's, not undermines it.
> There's a mass delusion in the industry that you need Kubernetes to run a serious production service. You don't. At StatusDude, we serve thousands of monitoring checks per minute, run multi-region workers, and deploy multiple times a day
This is pretty small scale, Kubernetes comes in when you've got a larger workload.
“We don’t know how to scale Traefik so we went with haproxy”
Well doh. Haproxy is designed for this. You can make haproxy serve copious amount of traffic on a single arm core and a little bit of ram. Imagine what you can do with a few replicas on your large clusters.
This has nothing to do with the choice of CI/CD or docker versus kubernetes.
While I agree with you I'm not sure the rest of the world does.
Over the past decade, I'm seeing k8s used everywhere for everything, companies setting up clusters to run literally one simple app with couple of hundred requests per hour.
In theory. I read so many times now where people report they use it and don't really need it, and I've seen it myself now too. Still very anecdotal, but it seems somethings there
>P.S Nginx would do too, I just felt like getting haproxy up this time :)
I think this line summarizes better than anything. Perfect example of how move fast and break things begins gloriously at first then inevitably, the breaking you thought you were doing hasn't even started and you find out what that part means.
I've always been the one saying "this is going to be a problem in a couple months" then I get shot down for "being negative." Then in a couple of of months when it fails I start getting aggression thrown at me "oh i know you want to say I told you so" and such even though I've never said such a thing when something fails. No. I would just like you to hear out my thoughts even when they may not be what you want to hear. We are all working towards the same goal.
It's exhausting reading about this stuff because there is inevitably a barrage of comments about "you don't need kubernetes, you can run your app out of a single vm you dumb trend chaser" in this style.
Like, sorry, no, not to a point. Yes, if you have a small app without a lot of scale, and it doesn't need to be uber reliable and have very little if almost 0 downtime, okay, sure. Most use cases are like that! This is correct but applying it as a generality is just plain wrong and displays the type of arrogance people accuse kubernetes users of having.
What happens if a container in the VM goes down or the app inside of it crashes, how do you recover? Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly overstate how painful it is.
So, people usually just go with kubernetes because 1) it is operationally not that hard to deal with compared to the things I just mentioned, and has a massive ecosystem and 2) the risk of the VM + container spiraling into complexity is perceived as way more than going more complex at the start.
I mean dude, most of us did high-availability before kubernetes existed. Kubernetes didn't solve any unsolved problem, it just moved it from a gui to yaml.
You just used a load-balancer (an AWS one preferably) with a few machines behind it, optionally an auto-scale group. AWS has multi-region support built in.
Re: file descriptors -- that's not something handled by kubernetes, if anything you just have more layers now.
You don't need service discovery, you never did, Host your services at a private zone {service-name}.{env}.company.com zones which points to the loadbalancer.
You need monitoring and should use an observability solution for that, has 0 to do with kubernetes.
You're just taking a bunch of random entirely solved problems and for some reason suggesting kubernetes is somehow helpful for them.
“we” is me. Kubernetes is easier. yes, file descriptors are handled by managed k8s. that was a small example. I could go much further with it. the ecosystem is far easier and I’d never go back to the way you’re describing because it is just rearchitecting the wheel.
You do need service discovery at some point at scale if you are using any kind of distributed system. that is quite a strong claim. what do you have to support that claim?
You sound like you’ve never really seen scale and I’m not sure how to respond in a way that is polite. It is heavily dependent on your use case and industry. if you can get by with docker containers, good for you! My only point is you are probably reinventing the wheel here.
> You sound like you’ve never really seen scale and I’m not sure how to respond in a way that is polite.
Bro I've worked at meta, I've worked at places that handled 1billion requests a day with a team of 20 people. I have a lot of questions myself about your experience that might be impolite as well. The fact that you linked kubernetes and file descriptors makes me think you really don't actually have the baseline fundamentals at all here.
meta only uses inhouse tooling as far as I’m aware, so what exactly is your expertise you’re claiming here?
I didn’t claim kubernetes handles file descriptors, lol. I was saying single vm solutions hit these OS limits faster than you’d think. Try understanding the argument you’re responding to. You haven’t responded with anything of substance, so I can safely ignore you, especially given your comment history seems to have an overwhelming bias against kubernetes, which is weird. Have a good day.
Because plenty of people share your POV and kinda - a little bit - behave like there was no life before k8s, I will try to address your points.
>What happens if a container in the VM goes down or the app inside of it crashes, how do you recover?
Docker will restart container automatically. You don't have to do anything.
Docker-compose will restart after VM restart. You don't have to do anything.
If a VM goes down - I do have a HA (another VM at another provider) and DNS load balancing.
>Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
While I don't like systemd, it does this automatically, while, it's not really used here.
> What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
This has been pretty much answered already but, upgrades does not affect containers (unless docker engine upgrade).
Restarts - docker will handle these automatically - nothing to do here.
> What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
This actually handles way better w/o managed kubernetes, as it's usually a single region and your cluster and workloads would simply be completely down, while mine would work, because of provider redundancy.
> What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
Kubernetes with autoscaling wins hands down here, but, it's not automatic, nor hassle free.
You are also assuming overprovisiong which is usually not the case for traffic spikes.
> What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
This also affects k8s, exactly the same way.
> What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
I do have service discovery and network isolation built into docker, thanks.
> What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Believe it or not, it's the default with docker.
> Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly understate how painful it is.
All my workloads are containerized and I can just move them to a k8s cluster whenever I want, if needed.
2) the risk of the VM + container spiraling into complexity is perceived. as way more than going more complex at the start.
The risk of your k8s ecosystem spiraling into operators madness and argoapps over helmfiles all while trying to accommodate for ci/cd and costs offing the chart is - IMHO - way higher.
Thanks. People are really acting like we were cavemen before kubernetes, but I guess that those people just never tried to run anything without k8s and because that's the only thing they know they are biased toward it
There's a mass delusion in the industry that using Kubernetes has to be hard, grossly over complex, and is always wrong if you are not Netflix scale. Is the system as described by the author significantly less complex and better than a small Kubernetes system? Sounds like they went through a lot of work to get it to their desired state.
Rolling your own zero-downtime deployments is as about as a good idea as rolling your own security... it's not a good idea.
I run our small system on a single EC2 instance with K3s. It runs a half-dozen or so services and does it quite well. I don't think it is particularly complex or over engineered. I like how easy it is to maintain the configuration in a helm package and quickly deploy it in different environments.
There is a learning curve for doing K8s well, but that's true for any non-trivial system.
so op just recreated with sticks and tape a very basic feature what k8s does out of the box, and nobody else would be able to support his creation, because its handrolled adhoc with sparse documentation.
Please try to be less snarky and dismissive in comments on HN. The guidelines make it clear we're trying for something better here, and this is meant to be a place where we can appreciate building for its own sake. https://news.ycombinator.com/newsguidelines.html
Perhaps we should look the other way: why use K8s if podman-compose can do the same? Maybe we should deprecate it and move towards simpler and more robust solutions?
If this creation stays as is, then it's not very complicated and pretty easy to understand and support. But that is a big IF, and very likely won't be true. Over time people will add more useful features to it, then it becomes another Kubernetes (and if you don't have a strong engineering team, it will probably be much worse than Kubernetes).
It's totally fine to not to use k8s. Personally, I think I have made several good decisions in my career to use/avoid k8s in different scenarios.
But if someone wrote a blog to brag about not using k8s, they can't stop people from wanting to compare their work against k8s. If there's any arrogance in the air, it feels stronger on the other side.
I worked at a place (a big name in a given vertical!), where the SRE looked at K8S and said "hold my beer".
Out came Docker, dnsmasq, miles of duct tape and a whole lot of swearing. Just to come nowhere close to reinventing something better folks were doing years prior.
Just because you can (or think you can) doesn't mean you should. I sure do hope no one is maintaining that NIH monstrosity now!
There was an article a while back on HN about why web designers choose light grey on white background. Basically, it looks fine on their own monitor which has the contrast turned way up
The css is ".prose-invert" and there's a ".prose" that looks better, I wonder if something threw a switch to make it "invert" when it should be ".prose" because you're right, this is unreadable as-is. Interesting read though.
For me it’s Dark Reader (https://github.com/darkreader/darkreader) which can be installed on at least chrome and Firefox desktop browsers, and safari and Kagi on iPhones.
I use it to keep from getting flashbanged by my monitor. In this case it also fixes the above site, however some websites need the color filter mode changed to work better, so realistically I’m not ending up with less fixing of websites, just easier fixing.
Its black background and white text on my screen. So either the OP saw the comments and fixed it or some people in this thread have weird settings. Or maybe I have weird settings...hmmm.
It's actually fun to see this. Running systems in a lot of different ways are just interesting. I do however get kinda sad at the hate at k8s because it's really good at what it does.
I've seen so many projects bending over backwards to avoid k8s and pay large cloud bills to avoid it at all costs. (ECS and app services are hopelessly expensive and bothersome)
K8s is really good, pretty easy to maintain, but a bit hard to understand. Mostly because distributed, zero down time systems are a bit hard to get by nature. But if you have someone that wants to take it on I've managed k8s clusters, solo, without incident, while doing lots of other stuff too (working with larger teams now though). Not to mention there's a lot of competence out there that can take over if I'd move on. Most of the deep complexity comes with more advance use cases, that wont show up for smaller deployments.
That said, no h8 towards going your own way! If your a solo developer (or small team) for a smallish project, don't feel the absolute need. If you get to the point you need it you should be earning enough to start paying someone to help ya get your app to a distributed system like k8s.
I think it's good to invest the time in understanding k8s though as a professional. Even if you won't directly run it it teaches you a lot about how to think about distributed, zero downtime systems. And what requirements that puts on an app.
Balanced takes like this are the only reason I still come back to HN. A shot at K3S in my homelab is in my backlog but young kids have set me back. If you have any material that touches on what you said above I'd appreciate a link. So much crap out there which is just Hello World blogspam.
KodeKloud is worth a subscription. I'm an older learner too and their courseware works well for me.
I see stuff like Azure Container Applications like an easy introduction to the world of k8s. Sane defaults, looks managed but you can still get familiar with all the important concepts, and sooner or later you can move to a proper Kubernetes setup while having mastered the ideas behind it. If needed, of course.
I was surprised that there was no mention of two things:
1. Retrying a non-idempotent request on a failure type that does indicate that no action was taken is not necessarily safe.
2. It’s possible and actually fairly common to design a backend that can do a clean shutdown: it stops accepting new requests, completes old requests, then exits. I sincerely hope that Docker’s tooling is good enough for a service to unregister itself before it actually stops accepting requests, but I’m not actually very familiar with using Docker to manage HTTP routing. (I use a home grown tool that is far simpler.)
I remember a time where HN was quite critical to the complexity of k8s. After reading top comments, I can see the tide has shifted.
CTOs have found reasons to standardize on k8s and it's not just for technical reasons. This was recently discussed last week:
https://notnotp.com/notes/what-job-interviews-taught-me-abou...
https://news.ycombinator.com/item?id=48546428
I used to be critical of almost everyone who used it. Now that I've been learning the ecosystem for a couple years, I'm not as angry about it, and it doesn't affect me as much personally.
I still think almost all of my complaints were valid: - not just container orchestration, more like a whole AWS solution (its own routing)
- has single-points of failure that would be avoided if you just used aws
- unnecessary complexity for almost all use cases
- ecosystem easily allows you to 2x-8x that complexity (helm+argocd+istio+carpenter+dozens more=hundreds of new failure modes)
- fundamentally moves many teams to a "I don't understand why my service is crashing, so let's just bring up more nodes every time rather than ever learning to debug it" mentality.
- All the feature teams that are supposed to "own their own kubernetes implementation for their own services" never do.
Of course a lot of that frustration is more at working at a place with a poorly-operated SOA where it takes a half-dozen services to send an email to a client or send a text or something silly. It sure is a waste of runway to obsess over a SOA when you aren't profitable yet.
But at this point it's sort of a sunk cost because it's become the industry standard. And AI can help with 90% of the complexity, which is it's own yellow-flag, but here we are.
This or there's a lot of fresh blood ;-)
I’m one of those top commenters who used to be a K8 naysayer. I’m also definitely not young blood.
The reason my position has changed is because:
1. The tooling has gotten better for setting up and managing K8.
2. In two of the last 3 jobs where we opted for a simplified alternative to k8, we came to regret that decision within a couple of years of that decision being made. If you’re core architecture is changing on a timescale of months (not years) then you picked the wrong foundations to build from.
That all said, I still think there is a pragmatic decision that needs to be made. And if I were in the author of this articles position I probably wouldn’t have picked k8s for this task either, despite what I said above. But, and as I said in my comment dismissing this article, they are dealing with low traffic and none of the problems that lend themselves to the benefits of k8. So my criticism of this article is that it’s misleading because their problem is easy but they’re writing as if they’re having to deal With problems of scale when they’re actually not.
Mandatory mention: https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...
But yeah, pretty cool DNS resolving features in HAProxy, that's nifty
I‘ve built this anti-k8s stance pre-LLMs, and just realized that, actually, agents should be pretty helpful in dealing with it? Is avoiding kubernetes still advisable for projects that will likely never use its full complexity, given how easy it is to maintain now?
Kube and helm are ideal for LLM usage, they make what is a fragile kind of thing into declarative, same as terraform for infra at the slightly lower level.
> thousands of monitoring checks per minute
That isn’t a lot. You could easily run that from one host. The reason people reach for Kubernetes (and similar) is because they need to scale past that single host dependency.
100%. And a shared mental model. I love how I can scale up all my services the same way, across clouds.
It's great.
You could, but they don't, meaning their argument is still sound (whether they _could_ use a single host is besides the point, they're not doing that).
Are you sure they’re not?
They’re multi-region, but that doesn’t mean they’re running across multiple hosts in each region.
Docker compose doesn’t support pooling multiple hosts, so if they are running multiple hosts per region then there’s a lot more complexity to their setup than they’re documenting in that blog. Even if that complexity is human toil managing each host as a separate entity.
> The reason people reach for Kubernetes (and similar) is because they need to scale past that single host dependency.
I have some stuff on single-node k3s. Because it's standard so I don't have to care.
The reason most people reach for Kubernetes is because it's cool. The entire infra the vast majority of Kubernetes users have could run on a single bare metal machine with a second one for redundancy.
To be fair: using Kubernetes anyways builds the skill just in case you become one of the 0.1% who actually need it down the line.
You can hire an Azure or Google Kubernetes devops guy and he will be equally comfortable on your AWS EKS kubernetes cluster. And when he leaves, you don't have a six week onboarding process with the new guy to learn all the ins and outs of your totally bespoke, non-standard container orchestration system that was cobbled together by two devs with no operations experience.
K3S takes about 5 minutes to setup the first time and you instantly have an entire universe of standardized operational tooling. I wouldn't touch docker compose with a 20 foot pole for production work.
Docker compose is hardly "totally bespoke".
Setting up K8s isn't rocket science, but maintaining it are offputting, to say the least.
As soon as you work in a team, it’s irrelevant whether the project actually needs it. There will be someone who convinces stakeholders that it is necessary and then you just have to fall in line and learn the skills knowing that it is most likely one of the 99.9% of projects where it is just overkill.
Until your project has some success, and it turns out all those "complex" features actually turn out to be extremely useful.
Which is exactly what is happening with us, too bad we didn't choose K8S from the get-go and stuck with a "simpler" tool (gaining very little in the process).
> The reason most people reach for Kubernetes is because it's cool.
This shittake was probably valid 10y ago, I would have agreed with you back then
> The entire infra the vast majority of Kubernetes users have could run on a single bare metal machine
Where are you pulling this out of? A large number of k8s users don't need it, but the alternative you have sounds hyperbolic.
Okay, I'll bite. What if your workload genuinely doesn't fit on one machine? Like load balancing or clustering 20+ nodes for LLM inference?
Your rebuttal to the parent claiming that almost nobody needs k8s is to bring up a workload almost nobody runs? It seems to me like your argument reinforces the parent's, not undermines it.
What workloads are other people running?
I wouldn't say my career is out of the ordinary, however most software I've built/maintained need more than one host.
..and HA
docker-rollout also works well: https://github.com/wowu/docker-rollout
The readme covers connection draining with Traefik which should solve one of the issues the author mentions
Are you monitoring resource utilization per container? Do notifications get sent out when container(s) become unhealthy? How are you handling secrets?
These are things I'm trying to figure out at work using Podman. Would love to hear about any experience in these areas.
> There's a mass delusion in the industry that you need Kubernetes to run a serious production service. You don't. At StatusDude, we serve thousands of monitoring checks per minute, run multi-region workers, and deploy multiple times a day
This is pretty small scale, Kubernetes comes in when you've got a larger workload.
The whole reasoning behind this is flawed.
“We don’t know how to scale Traefik so we went with haproxy”
Well doh. Haproxy is designed for this. You can make haproxy serve copious amount of traffic on a single arm core and a little bit of ram. Imagine what you can do with a few replicas on your large clusters.
This has nothing to do with the choice of CI/CD or docker versus kubernetes.
While I agree with you I'm not sure the rest of the world does.
Over the past decade, I'm seeing k8s used everywhere for everything, companies setting up clusters to run literally one simple app with couple of hundred requests per hour.
We do this because we need redundancy. Are there non kubernetes alternatives for HA/redundancy that are worth it?
In theory. I read so many times now where people report they use it and don't really need it, and I've seen it myself now too. Still very anecdotal, but it seems somethings there
>P.S Nginx would do too, I just felt like getting haproxy up this time :)
I think this line summarizes better than anything. Perfect example of how move fast and break things begins gloriously at first then inevitably, the breaking you thought you were doing hasn't even started and you find out what that part means.
I've always been the one saying "this is going to be a problem in a couple months" then I get shot down for "being negative." Then in a couple of of months when it fails I start getting aggression thrown at me "oh i know you want to say I told you so" and such even though I've never said such a thing when something fails. No. I would just like you to hear out my thoughts even when they may not be what you want to hear. We are all working towards the same goal.
>>zero-downtime
the site is down for me.
It's exhausting reading about this stuff because there is inevitably a barrage of comments about "you don't need kubernetes, you can run your app out of a single vm you dumb trend chaser" in this style.
Like, sorry, no, not to a point. Yes, if you have a small app without a lot of scale, and it doesn't need to be uber reliable and have very little if almost 0 downtime, okay, sure. Most use cases are like that! This is correct but applying it as a generality is just plain wrong and displays the type of arrogance people accuse kubernetes users of having.
What happens if a container in the VM goes down or the app inside of it crashes, how do you recover? Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly overstate how painful it is.
So, people usually just go with kubernetes because 1) it is operationally not that hard to deal with compared to the things I just mentioned, and has a massive ecosystem and 2) the risk of the VM + container spiraling into complexity is perceived as way more than going more complex at the start.
I mean dude, most of us did high-availability before kubernetes existed. Kubernetes didn't solve any unsolved problem, it just moved it from a gui to yaml.
You just used a load-balancer (an AWS one preferably) with a few machines behind it, optionally an auto-scale group. AWS has multi-region support built in.
Re: file descriptors -- that's not something handled by kubernetes, if anything you just have more layers now.
You don't need service discovery, you never did, Host your services at a private zone {service-name}.{env}.company.com zones which points to the loadbalancer.
You need monitoring and should use an observability solution for that, has 0 to do with kubernetes.
You're just taking a bunch of random entirely solved problems and for some reason suggesting kubernetes is somehow helpful for them.
“we” is me. Kubernetes is easier. yes, file descriptors are handled by managed k8s. that was a small example. I could go much further with it. the ecosystem is far easier and I’d never go back to the way you’re describing because it is just rearchitecting the wheel.
You do need service discovery at some point at scale if you are using any kind of distributed system. that is quite a strong claim. what do you have to support that claim?
You sound like you’ve never really seen scale and I’m not sure how to respond in a way that is polite. It is heavily dependent on your use case and industry. if you can get by with docker containers, good for you! My only point is you are probably reinventing the wheel here.
> You sound like you’ve never really seen scale and I’m not sure how to respond in a way that is polite.
Bro I've worked at meta, I've worked at places that handled 1billion requests a day with a team of 20 people. I have a lot of questions myself about your experience that might be impolite as well. The fact that you linked kubernetes and file descriptors makes me think you really don't actually have the baseline fundamentals at all here.
bRo I wOrKeD aT mEtA
meta only uses inhouse tooling as far as I’m aware, so what exactly is your expertise you’re claiming here?
I didn’t claim kubernetes handles file descriptors, lol. I was saying single vm solutions hit these OS limits faster than you’d think. Try understanding the argument you’re responding to. You haven’t responded with anything of substance, so I can safely ignore you, especially given your comment history seems to have an overwhelming bias against kubernetes, which is weird. Have a good day.
You were the one who brought my experience, you were wrong. You can stop publicly embarrassing yourself at any point. I'm done reading it either way.
Because plenty of people share your POV and kinda - a little bit - behave like there was no life before k8s, I will try to address your points.
>What happens if a container in the VM goes down or the app inside of it crashes, how do you recover?
Docker will restart container automatically. You don't have to do anything. Docker-compose will restart after VM restart. You don't have to do anything. If a VM goes down - I do have a HA (another VM at another provider) and DNS load balancing.
>Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.
While I don't like systemd, it does this automatically, while, it's not really used here.
> What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.
This has been pretty much answered already but, upgrades does not affect containers (unless docker engine upgrade). Restarts - docker will handle these automatically - nothing to do here.
> What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.
This actually handles way better w/o managed kubernetes, as it's usually a single region and your cluster and workloads would simply be completely down, while mine would work, because of provider redundancy.
> What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.
Kubernetes with autoscaling wins hands down here, but, it's not automatic, nor hassle free. You are also assuming overprovisiong which is usually not the case for traffic spikes.
> What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.
This also affects k8s, exactly the same way.
> What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.
I do have service discovery and network isolation built into docker, thanks.
> What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.
Believe it or not, it's the default with docker.
> Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly understate how painful it is.
All my workloads are containerized and I can just move them to a k8s cluster whenever I want, if needed.
2) the risk of the VM + container spiraling into complexity is perceived. as way more than going more complex at the start.
The risk of your k8s ecosystem spiraling into operators madness and argoapps over helmfiles all while trying to accommodate for ci/cd and costs offing the chart is - IMHO - way higher.
Thanks. People are really acting like we were cavemen before kubernetes, but I guess that those people just never tried to run anything without k8s and because that's the only thing they know they are biased toward it
This just feels like mostly a complaint of missing features in Traefik.
There's a mass delusion in the industry that using Kubernetes has to be hard, grossly over complex, and is always wrong if you are not Netflix scale. Is the system as described by the author significantly less complex and better than a small Kubernetes system? Sounds like they went through a lot of work to get it to their desired state.
Rolling your own zero-downtime deployments is as about as a good idea as rolling your own security... it's not a good idea.
I run our small system on a single EC2 instance with K3s. It runs a half-dozen or so services and does it quite well. I don't think it is particularly complex or over engineered. I like how easy it is to maintain the configuration in a helm package and quickly deploy it in different environments.
There is a learning curve for doing K8s well, but that's true for any non-trivial system.
so op just recreated with sticks and tape a very basic feature what k8s does out of the box, and nobody else would be able to support his creation, because its handrolled adhoc with sparse documentation.
sounds like ghetto engineering
Please try to be less snarky and dismissive in comments on HN. The guidelines make it clear we're trying for something better here, and this is meant to be a place where we can appreciate building for its own sake. https://news.ycombinator.com/newsguidelines.html
Perhaps we should look the other way: why use K8s if podman-compose can do the same? Maybe we should deprecate it and move towards simpler and more robust solutions?
If all you have is a hammer, everything looks like a nail.
This is just the derogatory version of "picking tech which aligns with existing expertise and resource availability".
If this creation stays as is, then it's not very complicated and pretty easy to understand and support. But that is a big IF, and very likely won't be true. Over time people will add more useful features to it, then it becomes another Kubernetes (and if you don't have a strong engineering team, it will probably be much worse than Kubernetes).
[dead]
[flagged]
It's totally fine to not to use k8s. Personally, I think I have made several good decisions in my career to use/avoid k8s in different scenarios.
But if someone wrote a blog to brag about not using k8s, they can't stop people from wanting to compare their work against k8s. If there's any arrogance in the air, it feels stronger on the other side.
Please don't reply to a bad comment with another bad comment on HN. That's how we get flamewars, which is exactly what we want to avoid. https://news.ycombinator.com/newsguidelines.html
I am not sure the argument "muhh k8s is too complex, I will roll docker instead" flies well in the age of cloud managed k8s offerings
EKS is literally one click of a button away and you dont need to handroll this.
even if you dont know AWS console nor terraform, claude code with aws mcp can do that for you
The problem I have with Kunernetes in general is that everything is slow and configuring everything seems needlessly complex in common use cases.
I worked at a place (a big name in a given vertical!), where the SRE looked at K8S and said "hold my beer".
Out came Docker, dnsmasq, miles of duct tape and a whole lot of swearing. Just to come nowhere close to reinventing something better folks were doing years prior.
Just because you can (or think you can) doesn't mean you should. I sure do hope no one is maintaining that NIH monstrosity now!
Are you implying that leaning on standard tooling is more arrogant than "hold my beer"?
[flagged]
What's the thought behind having white and light grey text on a light grey background?
There was an article a while back on HN about why web designers choose light grey on white background. Basically, it looks fine on their own monitor which has the contrast turned way up
The css is ".prose-invert" and there's a ".prose" that looks better, I wonder if something threw a switch to make it "invert" when it should be ".prose" because you're right, this is unreadable as-is. Interesting read though.
Are the people here not looking at the article or bots? How on Earth does anyone read this?
Guessing it’s reading the system color scheme, because on my phone it’s white text on black.
(on iOS with dark mode enabled system wide)
Oh god, I just disabled dark mode and yeah, it looks awful. Looks great in dark mode though.
For me it’s Dark Reader (https://github.com/darkreader/darkreader) which can be installed on at least chrome and Firefox desktop browsers, and safari and Kagi on iPhones.
I use it to keep from getting flashbanged by my monitor. In this case it also fixes the above site, however some websites need the color filter mode changed to work better, so realistically I’m not ending up with less fixing of websites, just easier fixing.
I tweaked the css in my browser with developer tools.
It’s white text on black bg for me. But to answer your question- reader mode!
Looks like “dark mode” implementation attempt which missed setting the background to black.
Its black background and white text on my screen. So either the OP saw the comments and fixed it or some people in this thread have weird settings. Or maybe I have weird settings...hmmm.
It’s changed since I checked. I think op fixed it.
Their dark mode is busted if you don't have JS enabled.
[flagged]