I was part of the team that proposed this manner of operation (Kubernetes Operator)[1]/controller and it's immensely cool to see a competent team make what seems like an incredible one that really scales the business.
I had always assumed databases would be the silo that had the most benefits for an Operator. There are a few good Postgres Operators, but not many outside of that which actually do the day 2 operations.
thanks rob! It does help that we are backed by object storage, and we try to make architecture choices that keep the system simple and safe to roll out at any time.
This might seem like bait but I assure you it isn't - I reach for MySQL by default over postgres - but why did you choose MySQL for the state database? In terms of HA and synchronous clustering it's a good solution but unless I'm misreading it seems like you only use it for a key value store and for an append only log, neither of which are necessarily the problems you think "MySQL" for.
Given what seems to be a pretty fashionable stack (kubernetes, using etcd directly, cloud first, IaC and custom database technology) I'm interested what a company like that reaches for MySQL. Was the MySQL cluster already deployed for other reasons?
Do you collect metrics from those BYOC clusters? If so, I’m assuming you push them, not pull them? That would seem to follow the same direction as the cluster operator config syncs (i.e. the direction in which connections are made, not in which data flows).
Yep, all pushed based directly from cluster! It helps that most of the observability ecosystem is converging on this? Even Prometheus, while pull-based inside the cluster, can remote-write to external storage, and that's the shape we'd pick even in that world. (we use datadog right now)
I was part of the team that proposed this manner of operation (Kubernetes Operator)[1]/controller and it's immensely cool to see a competent team make what seems like an incredible one that really scales the business.
I had always assumed databases would be the silo that had the most benefits for an Operator. There are a few good Postgres Operators, but not many outside of that which actually do the day 2 operations.
[1]: https://operatorhub.io/what-is-an-operator
thanks rob! It does help that we are backed by object storage, and we try to make architecture choices that keep the system simple and safe to roll out at any time.
This might seem like bait but I assure you it isn't - I reach for MySQL by default over postgres - but why did you choose MySQL for the state database? In terms of HA and synchronous clustering it's a good solution but unless I'm misreading it seems like you only use it for a key value store and for an append only log, neither of which are necessarily the problems you think "MySQL" for.
Given what seems to be a pretty fashionable stack (kubernetes, using etcd directly, cloud first, IaC and custom database technology) I'm interested what a company like that reaches for MySQL. Was the MySQL cluster already deployed for other reasons?
Do you collect metrics from those BYOC clusters? If so, I’m assuming you push them, not pull them? That would seem to follow the same direction as the cluster operator config syncs (i.e. the direction in which connections are made, not in which data flows).
Yep, all pushed based directly from cluster! It helps that most of the observability ecosystem is converging on this? Even Prometheus, while pull-based inside the cluster, can remote-write to external storage, and that's the shape we'd pick even in that world. (we use datadog right now)
Interesting read. A simpler way to manage the data plane.
Curious to understand if you also use OTEL layer to understand what is happening in your BYOC cluster ?
Yes, the database is instrumented with OTel, but Datadog (via local agent) is our primary sink.