I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.
I'm increasingly of the opinion that "basically a data warehouse, but incrementally maintained materialized views under the hood as an implementation detail" (e.g. Materialize, Feldera, anything based on Differential Dataflow / DBSP) is a good point in the design space. I get that they're more complicated than batch, and for many orgs the low latency isn't necessary for pure analytics/dashboards.
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
I’m not sure you have to squint; an mview is just a cached query. The only difference between an mview and a cache is the “incremental” part of the equation.
That is, my only disagreement is the lack of gusto
1. completeness is still a very desirable behavior for many use cases, but it's gonna be tricky to build these use cases on top of the new wave of IVM-like streaming systems, which provides no completeness marker.
2. periodic queries declared in non-streaming declarative form can and should be computed incrementally, but as far as I know, there's still not much mature support for automatic optimizations
the fact that traditional database infrastructure handles many people's needs doesn't really obviate dataflow style architectures. personally I think (3) composition is a real potential win. I would also note that streaming is a really great base on which to build a distributed database that looks more like Postgres on top.
I guess I'm mostly confused about the idea that streaming systems should be trying to supplant sql databases, or that a failure to do so implies that they don't have utility. it looks like the GP has done some really interesting work on automatic parallelism. that isn't pointless just because most people building operational systems still reflexively reach for PG.
That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs.
Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans.
One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.
A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database.
It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies.
I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model.
Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true.
Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe.
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink.
Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.
> People get by with the existing programming languages and models, that seems to be fine.
People like what they like, and they have trouble with anything other than imperative structured programming. It’s been a pain over the last decade plus to get people just to try functional programming, and they already know what functions are!
It’s so hard I think it’s a losing battle. No matter how good a paradigm people will always stick to a) what they know and b) what they’re paid to know.
However, I still think there’s a future for paradigms like dataflow because LLMs use random esoteric languages with great ease, and without complaint. They will use the tool for the job, even if they’re unfamiliar with it at first. “Mainstream” doesn’t really matter anymore.
There is a reason imperative programming is so common. It is more amenable to poorly designed, under specified, iterative development. Most real world software is in that category naturally. If you're meticulously designing and engineering it really well from the start, sure functional languages represent it well without leaving much room for misinterpretation and thus bugs. But no one does that.
Marginal cost of adding a feature has to be proportional to the revenue made by that feature. Then In Java or go you just add a ugly special case to appease the large customer and ignore the small ones' emails. Bugs getting shunted around instead of truly fixed at the root is also totally OK as long as they are not in the major revenue/cost centers of the product. No one has time to replace these piles of hacks and eventually you would have given up most of your languages benefits and your types now mean nothing there is probably a hundred flags making it a union effectively.
Rust is one language though where hacking around goes a long way without breaking too many of the guarantees, although it's not perfect, from my experience at a company where services spanned java go rust and ruby.
> If you're meticulously designing and engineering it really well from the start, sure functional languages represent it well without leaving much room for misinterpretation and thus bugs. But no one does that.
I find functional programming better at iterative development. You don't have to go full on on types but you always have the choice to make it good in the future. And from my experience the final refactoring is just better and let you built way more on top than imperative languages.
The reason imperative programming is so common is because of network effect. Unix won over lisp machines
You paper "A catalog of stream processing optimizations" is great! It seems like a classic in stream processing for me :)
Generally I agree with this comment. Very few niche applications require stream processing and all struggle that had been into research to enable some of the ACID properties for stream processing.
While at it, would you know how kdb+ (mostly used in finance) work? It -- together with q -- seems to achieve very low latency for real time systems.
I only kicked the tires on kdb ages ago, so this may be out of date.
At the time k's speed came from holding time series data in memory as simple dense arrays. The language treats a vector operation like z[] = x[] + y[] as a single operation, and so executes it as one step in the interpreter loop. The actual operation logic is just compiled c code.
So despite being a very simple lisp like language, its array orientation lets it be surprisingly speedy for certain kinds of data crunching.
> That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
.. as long as the processing they do fits in a map().
I still deeply believe that a new class of programs can be unlocked by using reduce() or all_gather() in a massive concurrent stream.
In a stadium, why can't a uuid be embedded as an acoustic chirp emitted by all the smartphones, then triangulation-reduced in a 3D map of all devices, enabling a stadium-wide sound system?
Why don't we have MMORPGs that can deal with 100k users in a single game instance?
Separately from the content of this paper, the LLM prose is unbearable!
These are not proper headers: "The Answer Neither Community Finished Alone", "The Peace Nobody Heard", "The Larger Disappearance". These are clickbait headlines. I don't want to be constantly tantalized--I want to be shown dense, organized information I can rapidly skim for the parts I care about.
That's an obvious example of the Claudese being spoken here, but it emanates from every word. Write your own paper or just publish the prompt and I'll run it in my own LLM.
I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
This is one of the things I wished for in the Beam APIs, and I'm glad they included it in the doc. Some (single) elements are bigger than a single function, and can't be processed one at a time. For example. a very large CSV file. They added support for it later (as Splittable DoFn's) but each implementation had varying support for it.
Streaming is expensive, both in terms of machine and human cost, and the benefit is quite marginal for most applications. However, Google engineers need to get promotion somehow, you know?
TLDR. 15 years ago they avoided SQL and created a language for streaming. Today, the industry decided that SQL materialized views with a “freshness” or time lag promise is the correct mental model.
I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
Agreed agreed.
During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.
[1] I worked on IBM Streams, https://www.ibm.com/docs/en/streams/4.3.0?topic=welcome-intr..., which had its own language, compiler and runtime system. IBM sold this technology in 2023: https://21cs.com/en/resources/articles/2023/10/10/21cs-acqui....
I'm increasingly of the opinion that "basically a data warehouse, but incrementally maintained materialized views under the hood as an implementation detail" (e.g. Materialize, Feldera, anything based on Differential Dataflow / DBSP) is a good point in the design space. I get that they're more complicated than batch, and for many orgs the low latency isn't necessary for pure analytics/dashboards.
I wonder if there's a place for such architecture in the operational path, not just analytics. If you squint, "this microservice reads data from this operational DB and passes it to this other microservice / puts it in a cache / sends it in response to a request" looks sort of like an "ad hoc, informally specified, bug-ridden, slow"[0] implementation of incremental view maintenance (in that a cache, or a service's data model in memory, is a sort of "materialized view" over the source data). I've seen some success in replacing a tangle of imperative-languages-and-network-requests with a declarative, incrementally maintained model in SQL, if the read latency can be kept under control.
I'm curious about the "some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache" part. could you expand on that?
[and yes, I do find these sort of problems interesting! :)]
[0] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
I’m not sure you have to squint; an mview is just a cached query. The only difference between an mview and a cache is the “incremental” part of the equation.
That is, my only disagreement is the lack of gusto
agree agree agree
1. completeness is still a very desirable behavior for many use cases, but it's gonna be tricky to build these use cases on top of the new wave of IVM-like streaming systems, which provides no completeness marker.
2. periodic queries declared in non-streaming declarative form can and should be computed incrementally, but as far as I know, there's still not much mature support for automatic optimizations
the fact that traditional database infrastructure handles many people's needs doesn't really obviate dataflow style architectures. personally I think (3) composition is a real potential win. I would also note that streaming is a really great base on which to build a distributed database that looks more like Postgres on top.
I guess I'm mostly confused about the idea that streaming systems should be trying to supplant sql databases, or that a failure to do so implies that they don't have utility. it looks like the GP has done some really interesting work on automatic parallelism. that isn't pointless just because most people building operational systems still reflexively reach for PG.
That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs.
Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans.
One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.
A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database.
It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies.
I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model. Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true.
Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe.
[0] https://redplanetlabs.com/
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink.
Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.
> People get by with the existing programming languages and models, that seems to be fine.
People like what they like, and they have trouble with anything other than imperative structured programming. It’s been a pain over the last decade plus to get people just to try functional programming, and they already know what functions are!
It’s so hard I think it’s a losing battle. No matter how good a paradigm people will always stick to a) what they know and b) what they’re paid to know.
However, I still think there’s a future for paradigms like dataflow because LLMs use random esoteric languages with great ease, and without complaint. They will use the tool for the job, even if they’re unfamiliar with it at first. “Mainstream” doesn’t really matter anymore.
There is a reason imperative programming is so common. It is more amenable to poorly designed, under specified, iterative development. Most real world software is in that category naturally. If you're meticulously designing and engineering it really well from the start, sure functional languages represent it well without leaving much room for misinterpretation and thus bugs. But no one does that.
Marginal cost of adding a feature has to be proportional to the revenue made by that feature. Then In Java or go you just add a ugly special case to appease the large customer and ignore the small ones' emails. Bugs getting shunted around instead of truly fixed at the root is also totally OK as long as they are not in the major revenue/cost centers of the product. No one has time to replace these piles of hacks and eventually you would have given up most of your languages benefits and your types now mean nothing there is probably a hundred flags making it a union effectively.
Rust is one language though where hacking around goes a long way without breaking too many of the guarantees, although it's not perfect, from my experience at a company where services spanned java go rust and ruby.
> If you're meticulously designing and engineering it really well from the start, sure functional languages represent it well without leaving much room for misinterpretation and thus bugs. But no one does that.
I find functional programming better at iterative development. You don't have to go full on on types but you always have the choice to make it good in the future. And from my experience the final refactoring is just better and let you built way more on top than imperative languages.
The reason imperative programming is so common is because of network effect. Unix won over lisp machines
You paper "A catalog of stream processing optimizations" is great! It seems like a classic in stream processing for me :)
Generally I agree with this comment. Very few niche applications require stream processing and all struggle that had been into research to enable some of the ACID properties for stream processing.
While at it, would you know how kdb+ (mostly used in finance) work? It -- together with q -- seems to achieve very low latency for real time systems.
I only kicked the tires on kdb ages ago, so this may be out of date.
At the time k's speed came from holding time series data in memory as simple dense arrays. The language treats a vector operation like z[] = x[] + y[] as a single operation, and so executes it as one step in the interpreter loop. The actual operation logic is just compiled c code.
So despite being a very simple lisp like language, its array orientation lets it be surprisingly speedy for certain kinds of data crunching.
> That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
.. as long as the processing they do fits in a map().
I still deeply believe that a new class of programs can be unlocked by using reduce() or all_gather() in a massive concurrent stream.
In a stadium, why can't a uuid be embedded as an acoustic chirp emitted by all the smartphones, then triangulation-reduced in a 3D map of all devices, enabling a stadium-wide sound system?
Why don't we have MMORPGs that can deal with 100k users in a single game instance?
Separately from the content of this paper, the LLM prose is unbearable!
These are not proper headers: "The Answer Neither Community Finished Alone", "The Peace Nobody Heard", "The Larger Disappearance". These are clickbait headlines. I don't want to be constantly tantalized--I want to be shown dense, organized information I can rapidly skim for the parts I care about.
That's an obvious example of the Claudese being spoken here, but it emanates from every word. Write your own paper or just publish the prompt and I'll run it in my own LLM.
I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
I've always thought this paper was a bit of an overlooked gem: https://www.vldb.org/conf/2002/S07P01.pdf
It views stream processing as a join between queries and data, where internally everything is just materialized views probing each other.
Interesting that the authors above ended up landing in a similar place conceptually.
> Splitting
This is one of the things I wished for in the Beam APIs, and I'm glad they included it in the doc. Some (single) elements are bigger than a single function, and can't be processed one at a time. For example. a very large CSV file. They added support for it later (as Splittable DoFn's) but each implementation had varying support for it.
Streaming is expensive, both in terms of machine and human cost, and the benefit is quite marginal for most applications. However, Google engineers need to get promotion somehow, you know?
[flagged]
TLDR. 15 years ago they avoided SQL and created a language for streaming. Today, the industry decided that SQL materialized views with a “freshness” or time lag promise is the correct mental model.
[dead]