Avoiding cyclic dependencies is good, sure. And they do name specific problems that can happen in counterexample #1.
However, the reasoning as to why it can't be a general DAG and has to be restricted to a polytree is really tenuous. They basically just say counterexample #2 has the same issues with no real explanation. I don't think it does, it seems fine to me.
The article doesn't make that claim. For example, the service n7 is used by multiple other nodes, namely n3 and n4. There is no cycle there, so it's okay.
Treating N4 as a service is fair. I think the article was leaning more toward that idea of N4 being a database, which is a legit bad idea with microservices (if fact defeating the point entirely). My takeaway is that if you're going to have a service that many other services depend on, you can do it but you need to be highly away of that brittleness. Your N4 service needs to be bulletproof. Netflix ran into this exact issue with their distributed cache.
I might have a different take. I think microservices should each be independent such that it really doesn't matter how they end up being connected.
Think more actors/processes in a distributed actor/csp concurrent setup.
Their interface should therefore be hardened and not break constantly, and they shouldn't each need deep knowledge of the intricate details of each other.
Also for many system designs, you would explicitly want a different topology, so you really shouldn't restrict yourself mentally with this advice.
Well, in practice you're likely to have hard dependencies between services in some respect, in that the service won't be able to do useful work without some other service. But I agree that in general it's a good idea to have a graceful degradation of functionality as other services become unavailable.
As we are talking about micro services, K8s has two patterns that are useful.
A global namespace root with sub namespaces will just desired config and current config will the complexity hidden in the controller.
The second is closer to your issue above, but it is just dependency inversion, how the kubelet has zero info on how to launch a container or make a network or provision storage, but hands that off to CRI, CNI or CSI
Those are hard dependencies that can follow a simple wants/provides model, and depending on context often is simpler when failures happen and allows for replacement.
E.G you probably wouldn’t notice if crun or runc are being used, nor would you notice that it is often systemd that is actually launching the container.
But finding those separation of concerns can be challenging. And K8s only moved to that model after suffering from the pain of having them in tree.
I think a DAG is a better aspirational default though.
> it really doesn't matter how they end up being connected.
I think you just mean that it should be robust to the many ways things end up being connected but it always does matter. There will always be a cost to being inefficient even if its ok to be.
I agree with this, and also I’m confused by the article’s argument—wouldn’t this apply equally to components within a monolith? Or is the idea that—within a monolith—all failures in any component can bring down the entire system anyway?
All sounds like a good plan, but there’s no easy way to enforce the lack of cycles. I’ve seen helper functions that call a service to look something up, called from a library that is running on the service itself. So a service calls itself. There was probably four or five different developers code abstractions stacked in that loop.
> Even without a directed cycle this kind of structure can still cause trouble. Although the architecture may appear clean when examined only through the direction of service calls the deeper dependency network reveals a loop that reduces fault tolerance increases brittleness and makes both debugging and scaling significantly more difficult.
While I understand the first counterexample, this one seems a bit blurry. Can anybody clarify why a directed acyclic graph whose underlying undirected graph is cyclic is bad in the context of microservice design?
Without necessarily endorsing the article's ideas....I took this to be like the diamond-inheritance problem.
If service A feeds both B and C, and they both feed service D, then D can receive an incoherent view of what A did, because nothing forces B and C to keep their stories straight. But B and C can still both be following their own spec perfectly, so there's no bug in any single service. Now it's not clear whose job it is to fix things.
the problem with "microservices" is the "micro". Why we thought we need so many tiny services is beyond me. How about just a few regular sized services?
At the time “microservices” was coined, “service oriented architecture” had drifted from being an architectural style to being associated with inplementation of the WS-* technical standards, and was frequently used to describe what were essentially monoliths with web services interfaces.
“Microservices” was, IIRC, more about rejecting that and returning to the foundations of SOA than anything else. The original description was each would support a single business domain (sometimes described “business function”, and this may be part of the problem, because in some later descriptions, perhaps through a version of the telephone game, this got shortened to “function” and without understanding the original context...)
I always took it to be a minimum and that "micro" meant "we don't need to wait for a service to have enough features to exist. They can be small." Instead, people see it as a maximum and services should be as small as possible, which ends up being a mess.
Kind of - AFAIK "micro" was never actually throughly defined. In my mind I think of it as mapping to one table (IE, users = user service, balances = balances service) but that might still be a "full service" worth of code if you need anything more than basic CRUD
The original sense was one business domain or business function (which often would include more than one table in a normalized relational db); the broader context was that, given the observation that software architecture tends to reflect software development organization team structure, software development organizations should parallel businesses organizations and that software serving different business functions should be loosely coupled, so that business needs in any area could be addressed with software change with only the unavoidable level of friction from software serving different business functions, which would be directly tied to the business impacts of the change on those connected functions, rather than having unrelated constraints from coupling between unrelated (in business function) software components inhibiting change driven by business needs in a particular area.
"Micro" refers to the economy, not the technology. A service in the macro economy is provided by another company. Think of a SaaS you use. Microservices takes the same model and moves it under the umbrella of a micro economy (i.e. a single company). Like traditional SaaS, each team is responsible for their own product, with communication between teams limited to sharing of documentation. You don't get to call up a developer when you need help.
It's a (human) scaling technique for large organizations. When you have thousands of developers they can't possibly keep in communication with each other. You have to draw a line between them. So, we draw the line the same way we do at the global scale.
This is a fair enough point, but you should also try to keep that tree as small as possible. You should have a damn good reason to make a new service, or break an existing one in two.
People treat the edges on the graph like they're free. Like managing all those external interfaces between services is trivial. It absolutely is not. Each one of those connections represents a contract between services that has be maintained, and that's orders of magnitude more effort then passing data internally.
You have to pull in some kind of new dependency to pass messages between them. Each service's interface had to be documented somewhere. If the interface starts to get complicated you'll probably want a way to generate code to handle serialization/deserialization (which also adds overhead).
In addition to share code, instead of just having a local module (or whatever your language uses) you now have to manage a new package. It either had to be built and published to some repo somewhere, it has to be a git submodule, or you just end up copying and pasting the code everywhere.
Even if it's well architected, each new services adds a significant amount of development overhead.
The explanation given makes sense. If they're operating on the same data, especially if the result goes to the same consumer, are they really different services? On the other hand, if the shared service provides different data to each, is it really one microservice or has it started to become a tad monolithic in that it's one service performing multiple functions?
I like that the author provides both solutions: join (my preferred) or split the share.
I don't understand this. Can you help explain it with a more practical example? Say that N1 (the root service) is a GraphQL API layer or something. And then N2 and N3 are different services feeding different parts of that API—using Linear as my example, say we have a different service for ticket management and one for AI agent management (e.g. Copilot integration). These are clearly different services with different responsibilities / scaling needs / etc.
And then N4 is a shared utility service that's responsible for e.g. performance tracing or logging or something similar. To make the dependency "harder", we could consider that it's a shared service responsible for authentication and authorization. So it's clear why many root services are dependent on it—they need to make individual authorization decisions.
How would you refactor this to remove an undirected dependency loop?
Yeah, a lot of cross-cutting concerns fall into this pattern: logging, authorization, metrics, audit trails, feature-flags, configuration distribution, etc
The only way I can see to avoid this is to have all those cross-cutting concerns handled in the N1 root service before they go into N2/N3, but it requires having N1 handle some things by itself (eg: you can do authorization early), or it requires a lot of additional context to be passed down (eg: passing flags/configuration downstream), or it massively overcomplicates others (eg: having logging be part of N1 forces N2/N3 to respond synchronously).
So yeah, I'm not a fan of the constraint from TFA. It being a DAG is enough.
I think it does indeed make a lot of sense in the particular example given.
But what if we add 2 extra nodes: n5 dependent on n2 alone, and n6 dependent on n3 alone? Should we keep n2 and n3 separate and split n4, or should we merge n2 and n3 and keep n4, or should we keep the topology as it is?
The same sort of problem arises in a class inheritance graph: it would make sense to merge classes n2 and n3 if n4 is the only class inheriting from it, but if you add more nodes, then the simplification might not be possible anymore.
Most components need to depend on an auth service, right? I don’t think that means it’s all necessarily one service (does all of Google Cloud Platform or AWS need to be a single service)?
That's immediately what I thought of. You'll never be able to satisfy this rule when every service has lines pointing to auth.
You'll probably also have lines pointing to your storage service or database even if the data is isolated between them. You could have them all
be separate but that's a waste when you can leverage say a big ceph cluster.
The trick I've used is the N1 (gateway) service handles all AuthN and proxies that information to the upstream services to allow them to handle AuthZ. N+ services only accept requests signed by N1 - the original authentication info is removed.
Service B initiates the connection to Service A in order to receive notifications, and Service B initiates the connection to Service A to query for changed data.
Service A never initiates a connection with Service B. If Service B went offline, Service A would never notice.
Requiring that no service is depended on by two services is nonsense.
You absolutely want the same identity service behind all of your services that rely on an identity concept (and no, you can't just say a gateway should be the only thing talking to an identity service - there are real downstream uses cases such as when identity gets managed).
Similarly there's no reason to have multiple image hosting services. It's fine for two different frontends to use the same one. (And don't just say image hosting should be done in the cloud --- that's just a microservice running elsewhere)
Same for audit logging, outbound email or webhooks, acl systems (can you imagine if google docs, sheets, etc all had distinct permissions systems)
Yeah even further, does that mean that SAAS like S3 shouldn't exist because it has multiple users?
I guess one possible solve would be to separate shared services into separate private deployments. Every upstream service gets its own imagine hosting service. Updates can roll out independently. I guess that would solve the blast radius/single source of failure problems but that seems really extreme.
The trick is to have your gateway handle authn, and then proxy authz data upstream so those services can decide how to handle it without needing to make a second call to the identity service.
I agree with you. Its interesting when I look at the examples you provide, that they are all non-domain services, so perhaps that is what codifies a potential rule.
If you look at this proposal and reject it, i question your experience. My experience is not doing this leads to codebases so intertwined that organizations grind to a halt.
My experience is in the SaaS world, working with orgs from a few dozen to several thousand contributors. When there are a couple dozen teams, a system not designed to separate out concerns will require too much coordinated efforts to develop against.
If a service n4 can't be called by separate services n2 and n3 in different parts of the tree (as shown in counterexample #2), then n4 isn't really a service but just a module of either n2 or n3 that happens to be behind a network interface.
Is there any way to actually enforce this in reality? Eventually some leaf service is going to need to hit an API on an upstream node or even just 2 leaf nodes that need to talk to each other.
Said less snarky, it should be trivial to define and restrict the dependencies of services (Although there are many ways to do that). If its not trivial, that's a different problem.
Ah, you don't mean enforce a novice making a mistake, you mean ensure from a design purity perspective?
I don't think its true that you need requests to flow both ways. For example, if a downstream API needs more context from an upstream one, one solution is to pass that data down as a parameter. You don't need to allow the downstream services to independently loop back to gather more info.
Restricting arbitrary east-west traffic should be table stakes... It should be the default and you opt into services being able to reach each other. So in that sense its already done.
The solution requires AWS since the gp thinks that's the only access control mechanism that matters. So I doubt there is going to be little cost about it.
Rule #2 sounds dumb. If there can't be a single source of truth, for let's say permission checking, that multiple other services relay on, how would you solve that? Replicate it everywhere? Or do you allow for a new business requirement to cause massive refactors to just create a new root in your fancy graph?
That implies that every service has a `user -> permissions` table, no? That seems to contradict the idea brought up elsewhere in the thread that microservices should all be the size of one table.
In reality their structure is much more like the Box with Christmas lights I just got from the basement. It would take a knot theory expert half a day to analyze what’s happening inside the box.
My main take on microservices at this point is that you only want microservices to isolate failure modes and for independent scaling. Most IO bound logic can live in a single monolith.
It is simpler than that. You only want microservices in the same cases you want services (i.e. SaaS). Meaning, when your team benefits from an independent third-party building and maintaining it. The addition of "micro" to "service" indicates that you are reaching out to a third-party that is paid by the same company instead of paying a separate company.
The author is not saying you should use a polytree but rather that the ideal graph of microservices should also be a polytree.
A polytree has the property that there is exactly one path that each node can be reached. If you think of this as a dependency graph, for each node in the graph you know that none of its dependencies have shared transitive dependencies.
I'll give it one though: if there are no shared transitive dependencies then there cannot be version conflicts between services, where two otherwise functioning services need disparate versions of the same transitive dependency.
Microservices should have clear owners reflected in the org chart, but the topology of dependencies should definitely not be isomorphic to your org chart.
The article is not wrong, but I feel like the polytree restraint is a bit forced, and perhaps not the most important concern.
You really need to consider why you want to use micro services rather than a monolith, and how to achieve those goals.
Here's where I'll get opinionated: the main advantage micro services have over a monolith is the unique failure modes they enable. This might sound weird at first, but bear with me. First of all, there's an uncomfortable fact we need to accept: your web service will fail and fall over and crash. Doesn't matter if you're Google or Microsoft or whatever, you will have failures, eventually. So we have to consider what those failures will look like, and in my book, microservices biggest strength is that, if built correctly, they fail more gracefully than monoliths.
Say you're targeted by a DDOS attack. You can't really keep a sufficiently large DDOS from crashing your API, but you can do damage control. To use an example I've experienced myself, where we foresaw an attack happening (it came fairly regularly, so it was easy to predict) and managed to limit the damage it did to us.
The DDOS targeted our login API. This made sense because most endpoints required a valid token, and without a token the request would be ignored with very little compute wasted on our end. But requests against /login had to hit a database pretty much every time.
We switched to signed JWT for Auth, and every service that exposed an external API had direct access to the public key needed to validate the signatures. This meant that if the Auth service went down, we could still validate tokens. Logged in users were unaffected.
Well, just add predicted, the Auth service got ddosed, and crashed. Even with auto scaling pods, and a service startup time of less than half a second, there was just no way to keep up with the sudden spike. The database ran out of connections, and that was pretty much it for our login service.
So, nobody could login for the duration of the attack, but everyone who was already logged in could keep using our API's as if nothing had happened. Definitely not great, but an acceptable cost, given the circumstances.
Had we used a monolith instead, every single API would've gone down, instead of just the Auth ones.
So, what's the lesson here? Services that expose external API's should be siloed, such that a failure in one, or it's dependencies, does not affect other API's. A polytree can achieve this, but it's not the only way to do it. And for internal services the considerations are different, I'd even go so far as to say simpler. Just be careful to make sure that any internal service than can be brought down by an attack on an external one, doesn't bring other external services down with it.
So rather than a polytree, strive for siloes, or as close to them as you can manage. When you can't make siloes, consider either merging services, or create deliberate weak-points to contain damage
Avoiding cyclic dependencies is good, sure. And they do name specific problems that can happen in counterexample #1.
However, the reasoning as to why it can't be a general DAG and has to be restricted to a polytree is really tenuous. They basically just say counterexample #2 has the same issues with no real explanation. I don't think it does, it seems fine to me.
An AuthN/Z system would probably end looking like counterexample #2, which immediately raised a red flag for me about the article.
Yeah if services can't be used by multiple other services, then what's the point?
The article doesn't make that claim. For example, the service n7 is used by multiple other nodes, namely n3 and n4. There is no cycle there, so it's okay.
Treating N4 as a service is fair. I think the article was leaning more toward that idea of N4 being a database, which is a legit bad idea with microservices (if fact defeating the point entirely). My takeaway is that if you're going to have a service that many other services depend on, you can do it but you need to be highly away of that brittleness. Your N4 service needs to be bulletproof. Netflix ran into this exact issue with their distributed cache.
I might have a different take. I think microservices should each be independent such that it really doesn't matter how they end up being connected.
Think more actors/processes in a distributed actor/csp concurrent setup.
Their interface should therefore be hardened and not break constantly, and they shouldn't each need deep knowledge of the intricate details of each other.
Also for many system designs, you would explicitly want a different topology, so you really shouldn't restrict yourself mentally with this advice.
Well, in practice you're likely to have hard dependencies between services in some respect, in that the service won't be able to do useful work without some other service. But I agree that in general it's a good idea to have a graceful degradation of functionality as other services become unavailable.
As we are talking about micro services, K8s has two patterns that are useful.
A global namespace root with sub namespaces will just desired config and current config will the complexity hidden in the controller.
The second is closer to your issue above, but it is just dependency inversion, how the kubelet has zero info on how to launch a container or make a network or provision storage, but hands that off to CRI, CNI or CSI
Those are hard dependencies that can follow a simple wants/provides model, and depending on context often is simpler when failures happen and allows for replacement.
E.G you probably wouldn’t notice if crun or runc are being used, nor would you notice that it is often systemd that is actually launching the container.
But finding those separation of concerns can be challenging. And K8s only moved to that model after suffering from the pain of having them in tree.
I think a DAG is a better aspirational default though.
Right, I don't mean that no service depends on each other, but that they can treat each other like a black box.
> it really doesn't matter how they end up being connected.
I think you just mean that it should be robust to the many ways things end up being connected but it always does matter. There will always be a cost to being inefficient even if its ok to be.
I agree with this, and also I’m confused by the article’s argument—wouldn’t this apply equally to components within a monolith? Or is the idea that—within a monolith—all failures in any component can bring down the entire system anyway?
> wouldn’t this apply equally to components within a monolith?
It's a nearly universal rule you'll want on every kind of infrastructure and data organization.
You can get away for some time with making things linked by offline or pre-stored resources, but it's a recipe for an eventual disaster.
All sounds like a good plan, but there’s no easy way to enforce the lack of cycles. I’ve seen helper functions that call a service to look something up, called from a library that is running on the service itself. So a service calls itself. There was probably four or five different developers code abstractions stacked in that loop.
> Even without a directed cycle this kind of structure can still cause trouble. Although the architecture may appear clean when examined only through the direction of service calls the deeper dependency network reveals a loop that reduces fault tolerance increases brittleness and makes both debugging and scaling significantly more difficult.
While I understand the first counterexample, this one seems a bit blurry. Can anybody clarify why a directed acyclic graph whose underlying undirected graph is cyclic is bad in the context of microservice design?
Without necessarily endorsing the article's ideas....I took this to be like the diamond-inheritance problem.
If service A feeds both B and C, and they both feed service D, then D can receive an incoherent view of what A did, because nothing forces B and C to keep their stories straight. But B and C can still both be following their own spec perfectly, so there's no bug in any single service. Now it's not clear whose job it is to fix things.
the problem with "microservices" is the "micro". Why we thought we need so many tiny services is beyond me. How about just a few regular sized services?
At the time “microservices” was coined, “service oriented architecture” had drifted from being an architectural style to being associated with inplementation of the WS-* technical standards, and was frequently used to describe what were essentially monoliths with web services interfaces.
“Microservices” was, IIRC, more about rejecting that and returning to the foundations of SOA than anything else. The original description was each would support a single business domain (sometimes described “business function”, and this may be part of the problem, because in some later descriptions, perhaps through a version of the telephone game, this got shortened to “function” and without understanding the original context...)
I always took it to be a minimum and that "micro" meant "we don't need to wait for a service to have enough features to exist. They can be small." Instead, people see it as a maximum and services should be as small as possible, which ends up being a mess.
They were never meant to be tiny, in the sense of just a few hundred lines of code.
The name was properly chosen poorly and led to many confusions.
Kind of - AFAIK "micro" was never actually throughly defined. In my mind I think of it as mapping to one table (IE, users = user service, balances = balances service) but that might still be a "full service" worth of code if you need anything more than basic CRUD
The original sense was one business domain or business function (which often would include more than one table in a normalized relational db); the broader context was that, given the observation that software architecture tends to reflect software development organization team structure, software development organizations should parallel businesses organizations and that software serving different business functions should be loosely coupled, so that business needs in any area could be addressed with software change with only the unavoidable level of friction from software serving different business functions, which would be directly tied to the business impacts of the change on those connected functions, rather than having unrelated constraints from coupling between unrelated (in business function) software components inhibiting change driven by business needs in a particular area.
"Micro" refers to the economy, not the technology. A service in the macro economy is provided by another company. Think of a SaaS you use. Microservices takes the same model and moves it under the umbrella of a micro economy (i.e. a single company). Like traditional SaaS, each team is responsible for their own product, with communication between teams limited to sharing of documentation. You don't get to call up a developer when you need help.
It's a (human) scaling technique for large organizations. When you have thousands of developers they can't possibly keep in communication with each other. You have to draw a line between them. So, we draw the line the same way we do at the global scale.
Conway's Law, as usual.
This is a fair enough point, but you should also try to keep that tree as small as possible. You should have a damn good reason to make a new service, or break an existing one in two.
People treat the edges on the graph like they're free. Like managing all those external interfaces between services is trivial. It absolutely is not. Each one of those connections represents a contract between services that has be maintained, and that's orders of magnitude more effort then passing data internally.
You have to pull in some kind of new dependency to pass messages between them. Each service's interface had to be documented somewhere. If the interface starts to get complicated you'll probably want a way to generate code to handle serialization/deserialization (which also adds overhead).
In addition to share code, instead of just having a local module (or whatever your language uses) you now have to manage a new package. It either had to be built and published to some repo somewhere, it has to be a git submodule, or you just end up copying and pasting the code everywhere.
Even if it's well architected, each new services adds a significant amount of development overhead.
This actually makes a lot of sense. I have one question though. Why is having 2 microservices depend on a single service a problem?
The explanation given makes sense. If they're operating on the same data, especially if the result goes to the same consumer, are they really different services? On the other hand, if the shared service provides different data to each, is it really one microservice or has it started to become a tad monolithic in that it's one service performing multiple functions?
I like that the author provides both solutions: join (my preferred) or split the share.
I don't understand this. Can you help explain it with a more practical example? Say that N1 (the root service) is a GraphQL API layer or something. And then N2 and N3 are different services feeding different parts of that API—using Linear as my example, say we have a different service for ticket management and one for AI agent management (e.g. Copilot integration). These are clearly different services with different responsibilities / scaling needs / etc.
And then N4 is a shared utility service that's responsible for e.g. performance tracing or logging or something similar. To make the dependency "harder", we could consider that it's a shared service responsible for authentication and authorization. So it's clear why many root services are dependent on it—they need to make individual authorization decisions.
How would you refactor this to remove an undirected dependency loop?
Yeah, a lot of cross-cutting concerns fall into this pattern: logging, authorization, metrics, audit trails, feature-flags, configuration distribution, etc
The only way I can see to avoid this is to have all those cross-cutting concerns handled in the N1 root service before they go into N2/N3, but it requires having N1 handle some things by itself (eg: you can do authorization early), or it requires a lot of additional context to be passed down (eg: passing flags/configuration downstream), or it massively overcomplicates others (eg: having logging be part of N1 forces N2/N3 to respond synchronously).
So yeah, I'm not a fan of the constraint from TFA. It being a DAG is enough.
I think it does indeed make a lot of sense in the particular example given.
But what if we add 2 extra nodes: n5 dependent on n2 alone, and n6 dependent on n3 alone? Should we keep n2 and n3 separate and split n4, or should we merge n2 and n3 and keep n4, or should we keep the topology as it is?
The same sort of problem arises in a class inheritance graph: it would make sense to merge classes n2 and n3 if n4 is the only class inheriting from it, but if you add more nodes, then the simplification might not be possible anymore.
Most components need to depend on an auth service, right? I don’t think that means it’s all necessarily one service (does all of Google Cloud Platform or AWS need to be a single service)?
That's immediately what I thought of. You'll never be able to satisfy this rule when every service has lines pointing to auth.
You'll probably also have lines pointing to your storage service or database even if the data is isolated between them. You could have them all be separate but that's a waste when you can leverage say a big ceph cluster.
The trick I've used is the N1 (gateway) service handles all AuthN and proxies that information to the upstream services to allow them to handle AuthZ. N+ services only accept requests signed by N1 - the original authentication info is removed.
Here's a really simple way to get a cycle.
Service A: publish a notification indicating that some new data is available.
Service B: consume these notifications and call back to service A with queries for the changed data and perhaps surrounding context.
What would you recommend when something like this is desired?
There is no cycle here.
Service B initiates the connection to Service A in order to receive notifications, and Service B initiates the connection to Service A to query for changed data.
Service A never initiates a connection with Service B. If Service B went offline, Service A would never notice.
Requiring that no service is depended on by two services is nonsense.
You absolutely want the same identity service behind all of your services that rely on an identity concept (and no, you can't just say a gateway should be the only thing talking to an identity service - there are real downstream uses cases such as when identity gets managed).
Similarly there's no reason to have multiple image hosting services. It's fine for two different frontends to use the same one. (And don't just say image hosting should be done in the cloud --- that's just a microservice running elsewhere)
Same for audit logging, outbound email or webhooks, acl systems (can you imagine if google docs, sheets, etc all had distinct permissions systems)
Yeah even further, does that mean that SAAS like S3 shouldn't exist because it has multiple users?
I guess one possible solve would be to separate shared services into separate private deployments. Every upstream service gets its own imagine hosting service. Updates can roll out independently. I guess that would solve the blast radius/single source of failure problems but that seems really extreme.
The trick is to have your gateway handle authn, and then proxy authz data upstream so those services can decide how to handle it without needing to make a second call to the identity service.
I agree with you. Its interesting when I look at the examples you provide, that they are all non-domain services, so perhaps that is what codifies a potential rule.
It's about the same for most code all the way down to single threaded function flow.
Yes! This is not unique to microservices.
If you look at this proposal and reject it, i question your experience. My experience is not doing this leads to codebases so intertwined that organizations grind to a halt.
My experience is in the SaaS world, working with orgs from a few dozen to several thousand contributors. When there are a couple dozen teams, a system not designed to separate out concerns will require too much coordinated efforts to develop against.
If a service n4 can't be called by separate services n2 and n3 in different parts of the tree (as shown in counterexample #2), then n4 isn't really a service but just a module of either n2 or n3 that happens to be behind a network interface.
Is there any way to actually enforce this in reality? Eventually some leaf service is going to need to hit an API on an upstream node or even just 2 leaf nodes that need to talk to each other.
IAM roles.
Said less snarky, it should be trivial to define and restrict the dependencies of services (Although there are many ways to do that). If its not trivial, that's a different problem.
I don't mean that. I mean that eventually the business is going to need some feature that requires breaking the acyclic rule.
Ah, you don't mean enforce a novice making a mistake, you mean ensure from a design purity perspective?
I don't think its true that you need requests to flow both ways. For example, if a downstream API needs more context from an upstream one, one solution is to pass that data down as a parameter. You don't need to allow the downstream services to independently loop back to gather more info.
Since you called the problem “trivial,” we can now all depend on you to resolve these problems for us at little cost, correct?
Restricting arbitrary east-west traffic should be table stakes... It should be the default and you opt into services being able to reach each other. So in that sense its already done.
The solution requires AWS since the gp thinks that's the only access control mechanism that matters. So I doubt there is going to be little cost about it.
Rule #2 sounds dumb. If there can't be a single source of truth, for let's say permission checking, that multiple other services relay on, how would you solve that? Replicate it everywhere? Or do you allow for a new business requirement to cause massive refactors to just create a new root in your fancy graph?
Services handle the permissions of their own features. Authentication is handled at the gateway.
Not sure if I agree its really the best way to do things but it can be done.
That implies that every service has a `user -> permissions` table, no? That seems to contradict the idea brought up elsewhere in the thread that microservices should all be the size of one table.
This is exactly the example I thought of and came here to post.
The rule is obviously wrong.
I think just having no cycles is good enough as a rule.
In reality their structure is much more like the Box with Christmas lights I just got from the basement. It would take a knot theory expert half a day to analyze what’s happening inside the box.
This seems completely wrong. In an RPC call you have a trivial loop, for example.
It would make more sense to say that the event tree should not have any cycles, but anyway this seems like a silly point to make.
My main take on microservices at this point is that you only want microservices to isolate failure modes and for independent scaling. Most IO bound logic can live in a single monolith.
It is simpler than that. You only want microservices in the same cases you want services (i.e. SaaS). Meaning, when your team benefits from an independent third-party building and maintaining it. The addition of "micro" to "service" indicates that you are reaching out to a third-party that is paid by the same company instead of paying a separate company.
Why do we use polytree in this context instead of DAG? Because nodes can’t ever come back together?
The author is not saying you should use a polytree but rather that the ideal graph of microservices should also be a polytree.
A polytree has the property that there is exactly one path that each node can be reached. If you think of this as a dependency graph, for each node in the graph you know that none of its dependencies have shared transitive dependencies.
I'll give it one though: if there are no shared transitive dependencies then there cannot be version conflicts between services, where two otherwise functioning services need disparate versions of the same transitive dependency.
Services (or a set of Microservices) should mimic teams at the company. If we have polytree, that should represent departments.
Microservices should have clear owners reflected in the org chart, but the topology of dependencies should definitely not be isomorphic to your org chart.
What's wrong with just imposing a DAG?
Oh that's weird, in the hacker news search index, this link was posted 4 days ago.
Good practical explanation of something I felt but couldn't put a name to.
Isn't it the same wisdom as to avoid cyclic dependencies?
It is not only that. An acyclic graph can be non-planar, which means that as you add more nodes, the number of edges can grow as O(n^2).
A polytree is a planar graph, and the number of edges must grow linearly with the number of edges.
The article is not wrong, but I feel like the polytree restraint is a bit forced, and perhaps not the most important concern.
You really need to consider why you want to use micro services rather than a monolith, and how to achieve those goals.
Here's where I'll get opinionated: the main advantage micro services have over a monolith is the unique failure modes they enable. This might sound weird at first, but bear with me. First of all, there's an uncomfortable fact we need to accept: your web service will fail and fall over and crash. Doesn't matter if you're Google or Microsoft or whatever, you will have failures, eventually. So we have to consider what those failures will look like, and in my book, microservices biggest strength is that, if built correctly, they fail more gracefully than monoliths.
Say you're targeted by a DDOS attack. You can't really keep a sufficiently large DDOS from crashing your API, but you can do damage control. To use an example I've experienced myself, where we foresaw an attack happening (it came fairly regularly, so it was easy to predict) and managed to limit the damage it did to us.
The DDOS targeted our login API. This made sense because most endpoints required a valid token, and without a token the request would be ignored with very little compute wasted on our end. But requests against /login had to hit a database pretty much every time.
We switched to signed JWT for Auth, and every service that exposed an external API had direct access to the public key needed to validate the signatures. This meant that if the Auth service went down, we could still validate tokens. Logged in users were unaffected.
Well, just add predicted, the Auth service got ddosed, and crashed. Even with auto scaling pods, and a service startup time of less than half a second, there was just no way to keep up with the sudden spike. The database ran out of connections, and that was pretty much it for our login service.
So, nobody could login for the duration of the attack, but everyone who was already logged in could keep using our API's as if nothing had happened. Definitely not great, but an acceptable cost, given the circumstances.
Had we used a monolith instead, every single API would've gone down, instead of just the Auth ones.
So, what's the lesson here? Services that expose external API's should be siloed, such that a failure in one, or it's dependencies, does not affect other API's. A polytree can achieve this, but it's not the only way to do it. And for internal services the considerations are different, I'd even go so far as to say simpler. Just be careful to make sure that any internal service than can be brought down by an attack on an external one, doesn't bring other external services down with it.
So rather than a polytree, strive for siloes, or as close to them as you can manage. When you can't make siloes, consider either merging services, or create deliberate weak-points to contain damage
Hi, this is my company blog. Hope you like this week's post.
have you read about Erlang supervision trees? https://adoptingerlang.org/docs/development/supervision_tree...
just imagine how many clients services like auth, notifications and so on has.
Polytrees look good, they don't work on orthogonal services