I think it's crazy that standard SQL has no clean way of handling nested data. It's might not fit elegantly into the relational model, but it's still a common business problem that should be addressed. At the bare minimum something like `array_agg` should be standardized.
Alternative query languages like EdgeQL show what first class support for nested data (and navigations) could look like, while the data model is still relational.
I guess technically it is in the SQL standard, but optional, as S098? I agree that SQL is sorely lacking here and I'm hoping that the OLAP side innovation (presto, bigquery, snowflake, duckdb all seem to do better) help push it forward.
If op is here: you can dramatically improve the robustness and soundness of your benchmark with one simple trick: you can run old versions of ASP.NET (Core) frameworks on newer .NET runtimes with no other changes; i.e. instead of benchmarking ASP.NET 10 + EF 10 on .NET 10 vs ASP.NET 11 + EF 11 on .NET 11, you can bench ASP.NET 10 + EF 10 on .NET 11 vs ASP.NET 11 + EF 11 on .NET 11
(I always upgrade projects by first upgrading the runtime and checking everything then separately (and maybe much later!) upgrading the framework.)
I don't understand the argument why `AsSplitQuery` could be more performant than a single round trip involving a multi join query. People mention data duplication and increased memory usage, but I would assume that `duplication` is just a matter of an extra pointer, not a bit-for-bit duplication of every reference to a single row.
Assume you fetch a single customer entity with their 100 order entities as includes. With single query this will join both tables and produce 100 rows that contain the order data but also each one contains the customer data redundantly. Now Imagine you had two includes there, that will multiply the number of rows again.
AsSingleQuery is as dangerous as this makes it sound. This works surprisingly well if you know that the number of included entities is low, but only then.
You can get much better queries here if you write a Select() and let EF Core translate that into SQL. That will probably do roughly want you are imagining here, usually with subqueries fetching the data from related entities.
Out main screen has at head level over 200 fields, most of them filled in for typical orders. The main table has several child tables, some which have several child tables again. The order line table has several, and especially one of them can typically have 5-7 rows per order line.
Many of our customers routinely dealt with orders that have 10k+ lines. So if you did it all in one go, you've now turned 200 head fields into 10 million that needs to be transmitted to the client and deduplicated there.
We have the root primary key on all child tables, so we fire off a handful of queries to load the complete data set for an order. Very fast as it's all indexed of course.
Databases are incredibly smart when it comes to fetching related data, a single select is indeed better than splitting queries and doing multiple roundtrips.
The problem however is in how results are returned over the wire. Duplicating rows is needless, but seems to be still the standard.
Relational DBs have been around a very long time, and I suspect the processing/transfer time was not the bottleneck, while data access was, so redundant data with the standard relational model wasn't the primary concern.
My experience suggests that they _can_ be good, but this particular pattern they can be remarkably bad at. Source: I keep having to optimise this pattern.
I think compression would reduce the problem not? I think if you swap the wire format to something like jsonb you would need to parse it again anyway and pay the cpu time.
Correct. However, this imho does not need to be a problem when using references to data instead of data duplication when sending results over the wire.
If you join multiple/many tables, you could end up with a large volume of data. And yes, this is bit-for-bit duplication—on the network. The query result is (typically) a single table. This table will get serialized as-is, with all duplicate data.
Couldn't we make references as `byte offsets in the result set` work to handle duplication? Real memory pointers wouldn't work over the network of course, but if the database driver would return results like this, the client could easily stitch these together. My hunch is that even if we implement references on a higher level than raw byte offsets it would still be more performant than just returning R1*R2 bytes for any R1<1:N>R2.
---
EDIT: According to LLM friends you could achieve before wire de-duplication by using FOR JSON AUTO in SQL Server or jsonb_agg in PostgreSQL. Not sure how much overhead that incurs though.
I think jsonb_agg could be a solution. Requires cpu for decoding but I think it is compensated by the reduction of on the wire traffic. Queries are different , especially if you have several one-to-many joins so you have to do it at the framework level. I wonder if it would be feasible a Postgres extension that does this automatically for you
You'd still return a multiplicative amount of rows, even if those rows contained only a reference. `array_agg` in postgres avoids this, but EF does not support using it for collection navigations.
One could envision a "Cartesian product" operation in the wire protocol, but I'm not convinced that's a good approach.
> You'd still return a multiplicative amount of rows, even if those rows contained only a reference
Sure, but a pointer is still a massive win over records, and I think a further cartesian product wire protocol extension would not be worth the hassle.
I don't think pointers (beyond a simple "same as in previous row" marker) will be a huge improvement, since you still get a multiplicative number of rows. And it comes with the cost of keeping all that data in memory. This approach also competes with using cheap compression (e.g. LZ4).
Some kind of "product" operator on the other hand reduces the cost to additive (just like `array_agg`).
looking at the benchmark it doesn't seem to be much of a difference, could someone please explain to me why is this little gain in performance so much important? especially when it did increased the GC work? or I didn't understood the data right? ty in advance
I'll go you one further. We need a standard higher-level DSL which abstracts over the various competing data access libraries in a portable, declarative way, such that 1) database engines can independently optimise themselves to better handle such a language, and 2) programmers can move between different companies and be expected to already know this language. I propose the name SQL for this.
Joking (but not really) aside, EF seems to be the easiest way to shoot yourself in the foot, and write code which you think is transactional, but is not actually transactional, and would be transactional if expressed in pure SQL.
I believe hasura or postgraphile does this for graphql. It's a very niche feature, and kills some of the optimizations if results are further used for filtering, but yes!
One of the challenges for their team I'm assuming is making sure the release notes also get copied to the actual documentation. This was one case where I thought the same thing and only until reading the release notes realized it was already a feature.
I think it's crazy that standard SQL has no clean way of handling nested data. It's might not fit elegantly into the relational model, but it's still a common business problem that should be addressed. At the bare minimum something like `array_agg` should be standardized.
Alternative query languages like EdgeQL show what first class support for nested data (and navigations) could look like, while the data model is still relational.
I guess technically it is in the SQL standard, but optional, as S098? I agree that SQL is sorely lacking here and I'm hoping that the OLAP side innovation (presto, bigquery, snowflake, duckdb all seem to do better) help push it forward.
MULTISET is the SQL native way of doing this but it's only implemented in Oracle, and most people have never heard of it
jOOQ emulates this feature for arbitrary databases and has the best explanatory article on the web about MULTISET imo
https://www.jooq.org/doc/latest/manual/sql-building/column-e...
Is this not what you want? Seems like it's part of the SQL standard.
https://www.postgresql.org/docs/19/ddl-property-graphs.html
That only seems to change how the query is performed, not which kind of data can be returned.
For example how would you use it to return a list of events and, for each event, the list of attendees in that event?
If op is here: you can dramatically improve the robustness and soundness of your benchmark with one simple trick: you can run old versions of ASP.NET (Core) frameworks on newer .NET runtimes with no other changes; i.e. instead of benchmarking ASP.NET 10 + EF 10 on .NET 10 vs ASP.NET 11 + EF 11 on .NET 11, you can bench ASP.NET 10 + EF 10 on .NET 11 vs ASP.NET 11 + EF 11 on .NET 11
(I always upgrade projects by first upgrading the runtime and checking everything then separately (and maybe much later!) upgrading the framework.)
I’ve always solved this with Multiple Active Result Sets and stored procedures.
Collect the data in the stored procedure with temp/in-memory tables and return minimal, non-duplicated, related result sets.
Single round trip, still accumulates results in efficient bulk batches, and allows results to be processed by the client as they stream in.
I don't understand the argument why `AsSplitQuery` could be more performant than a single round trip involving a multi join query. People mention data duplication and increased memory usage, but I would assume that `duplication` is just a matter of an extra pointer, not a bit-for-bit duplication of every reference to a single row.
Please enlighten me.
Assume you fetch a single customer entity with their 100 order entities as includes. With single query this will join both tables and produce 100 rows that contain the order data but also each one contains the customer data redundantly. Now Imagine you had two includes there, that will multiply the number of rows again.
AsSingleQuery is as dangerous as this makes it sound. This works surprisingly well if you know that the number of included entities is low, but only then.
You can get much better queries here if you write a Select() and let EF Core translate that into SQL. That will probably do roughly want you are imagining here, usually with subqueries fetching the data from related entities.
Out main screen has at head level over 200 fields, most of them filled in for typical orders. The main table has several child tables, some which have several child tables again. The order line table has several, and especially one of them can typically have 5-7 rows per order line.
Many of our customers routinely dealt with orders that have 10k+ lines. So if you did it all in one go, you've now turned 200 head fields into 10 million that needs to be transmitted to the client and deduplicated there.
We have the root primary key on all child tables, so we fire off a handful of queries to load the complete data set for an order. Very fast as it's all indexed of course.
Databases are incredibly smart when it comes to fetching related data, a single select is indeed better than splitting queries and doing multiple roundtrips.
The problem however is in how results are returned over the wire. Duplicating rows is needless, but seems to be still the standard.
Relational DBs have been around a very long time, and I suspect the processing/transfer time was not the bottleneck, while data access was, so redundant data with the standard relational model wasn't the primary concern.
My experience suggests that they _can_ be good, but this particular pattern they can be remarkably bad at. Source: I keep having to optimise this pattern.
I think compression would reduce the problem not? I think if you swap the wire format to something like jsonb you would need to parse it again anyway and pay the cpu time.
There is a name for this problem, Cartesian explosion: https://en.wikipedia.org/wiki/Cartesian_explosion
Correct. However, this imho does not need to be a problem when using references to data instead of data duplication when sending results over the wire.
I hope it is implemented with multiple result sets and a single roundtrip
For sure it is
EF Core documentation disagrees though: https://learn.microsoft.com/en-us/ef/core/querying/single-sp...
If you join multiple/many tables, you could end up with a large volume of data. And yes, this is bit-for-bit duplication—on the network. The query result is (typically) a single table. This table will get serialized as-is, with all duplicate data.
Couldn't we make references as `byte offsets in the result set` work to handle duplication? Real memory pointers wouldn't work over the network of course, but if the database driver would return results like this, the client could easily stitch these together. My hunch is that even if we implement references on a higher level than raw byte offsets it would still be more performant than just returning R1*R2 bytes for any R1<1:N>R2.
---
EDIT: According to LLM friends you could achieve before wire de-duplication by using FOR JSON AUTO in SQL Server or jsonb_agg in PostgreSQL. Not sure how much overhead that incurs though.
I think jsonb_agg could be a solution. Requires cpu for decoding but I think it is compensated by the reduction of on the wire traffic. Queries are different , especially if you have several one-to-many joins so you have to do it at the framework level. I wonder if it would be feasible a Postgres extension that does this automatically for you
You'd still return a multiplicative amount of rows, even if those rows contained only a reference. `array_agg` in postgres avoids this, but EF does not support using it for collection navigations.
One could envision a "Cartesian product" operation in the wire protocol, but I'm not convinced that's a good approach.
I don't think pointers (beyond a simple "same as in previous row" marker) will be a huge improvement, since you still get a multiplicative number of rows. And it comes with the cost of keeping all that data in memory. This approach also competes with using cheap compression (e.g. LZ4).
Some kind of "product" operator on the other hand reduces the cost to additive (just like `array_agg`).
> That one is tracked here: https://github.com/npgsql/efcore.pg/issues/2633
That issue is only about supporting `array_agg` as a function on tuples, not as an implementation strategy for `Include`s of collections.
looking at the benchmark it doesn't seem to be much of a difference, could someone please explain to me why is this little gain in performance so much important? especially when it did increased the GC work? or I didn't understood the data right? ty in advance
We need EF Core / DLINQ or equivalents for pretty much all languages.
I'll go you one further. We need a standard higher-level DSL which abstracts over the various competing data access libraries in a portable, declarative way, such that 1) database engines can independently optimise themselves to better handle such a language, and 2) programmers can move between different companies and be expected to already know this language. I propose the name SQL for this.
Joking (but not really) aside, EF seems to be the easiest way to shoot yourself in the foot, and write code which you think is transactional, but is not actually transactional, and would be transactional if expressed in pure SQL.
What EF needs is support for using postgresql's `array_agg` when `Include`ing collections.
I believe hasura or postgraphile does this for graphql. It's a very niche feature, and kills some of the optimizations if results are further used for filtering, but yes!
The real win in EF Core 11 is pruning the reference-nav joins out of split child queries, that's been dead weight since AsSplitQuery existed.
I wish EF Core had first-class support for raw SQL, like Dapper.
It has `context.Database.SqlQuery` and `ExecuteSql`.
It does as of EF Core 8 or 9.
One of the challenges for their team I'm assuming is making sure the release notes also get copied to the actual documentation. This was one case where I thought the same thing and only until reading the release notes realized it was already a feature.