We have been building GFQL for open source vectorized CPU/GPU cypher on dataframes (think duckdb for graphs, no DB needed) using agentic assistance... and our thinking is this is a stepping stone to something even more aggressive
Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about.
Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.
How would you find out which optimizations to apply to which queries?
One decision that brings a 10x improvement, might slow down a very slightly different query by 100x.
Thats the core issue of database query optimization, and thats why (good) optimizer target the 99% percentile, not the peak, or the average.
Not blowing up is the most important criteria to be able to rely on a system.
Your argument applies to literally every database. A sequential scan can be way faster then a using a index. That is a optimization that can under the right circumstances result in a great benchmark score, but a bad experience for some customers but great for others.
The idea that benchmark optimizations are all bad, is the flawed argument. A benchmark is nothing else but a pattern. You can indeed over optimize like having a LLM recalculate values in advance but if those optimizations also work on non-benchmark SQL prompts, ... That brings us back to the original point.
There is nothing wrong with having LLMs try and find spot to optimize, as long as your test pattern is not limited to a singular set of data. The reality is, there is actually a lot of meat on the bones in a lot of databases, because it was not physically possible to try and run 100 or 1000s of permutations on query/data patterns. That has now changed.
A discussion like this kind of reminds me driver optimizations for GPUs. The idea of only targeting the average, is what GPU's did like 15+ years ago. The issue was when they optimization was purely done to only benefit the benchmark (like preloading in dbs).
Where optimizations that made for faster benchmark results, also applied to games getting more FPS. This was such a hot topic years ago. Now nobody blinks a eye anymore when we even see game on game optimizations.
I can even argue the other way...
If a client has a database with a know pattern, and that client can enjoy a 10x performance increase because of that pattern optimization. And it does not reduce performance for others. Maybe that same optimization will trigger on some other client, who was not even the target.
What if your database has optimization that makes Wordpress work way faster. Thing is ... this type of targeted optimization already happens everywhere. I find it just odd that this is a discussion regarding databases.
Pre rendering data for a benchmark is indeed cheating, because that does not help clients, its misleading. But being able to show, "hey, we make X popular software faster then (example) postgres". The client are those that gain. And with LLMs that is now possible to do, without the cost the limited this in the past.
Yes, I am aware that databases do that.
But my point is that databases are conservative with what kind of optimizations they do, because the space of possible queries are infinite, and they want to have a guaranteed bound on what the worst case behavior is.
Big OLAP vendors like Snowflake do have the workloads of their customers, but you cannot just evaluate your optimizations on all workloads because its too expensive. You will only notice after deployment, when the customer calls you to complain. Or worse, they won't even consider you, because your novel workloads are too slow.
My point is not that specialized implementations can't deliver big performance gains, but that deciding when to use optimizations that have downsides reliably is very hard.
Even for a classical problem, well isolated problem, like join-order optimization, even if you have perfect cardinality estimates (which you don't), its still NP-hard, and still, even optimizers that are considered advanced, will choke.
Anything engine/implementation specific is far worse, with smaller gains and much bigger downsides, which every systems routinely screws up because its just really difficult to decide when to use what.
a lot of times customers have key queries and query families, so while the problems are hard in theory, they're not in practice. the more general problem is the wrong one. likewise, teams are already using agents to profile and optimize workloads and do custom db extensions, so boundaries like human/machine and vendor/user are changing as-is.
in our current flow, you said the magic words, query planner: the query planner is getting more hotpaths available to it than historically practical, and richer optimizations along them than used to be feasible
as we saturate baseline performance here, the question R&D community is already pushing on is making this more dynamic and per-workload, which opens even more room. ex: providing more open interfaces like better planner control by user owned agents. there are much more aggressive forms for more aggresive users: the more important a user, the more interesting it gets.
There has been a similar issue in the GPU-kernel dev community.
It was solved by understanding that the value provided by the benchmarks depends on your use-case. It sounds like the author in the article is yelling at the benchmark from his own idea of what a DB search should be (reliable, general), and not from what the benchmark is measuring.
If you just want the fastest kernel for a handful of shapes, let people make brittle but fast code.
If you are interested in general broad solutions, benchmark 50 different entries, limit code size to disallow assembly branches for every possible entry.
Put requirements like bitwise equivalence or other end-task limits in there.
Have public & private datasets.
I think it's more of a proof that we should spend more time thinking about what and why we want these things, and design benchmarks that reflect that instead of going of the old way of doing things.
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.
> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
I think you're both right, but Neumann much more so.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
Verifiability (such as via benchmarks) comes in exactly when trust is not reliable enough a discriminator, such as due to scale or variability. AI ups the volume, inhibiting people's ability to pick and choose based on trust, due to the amount of new players. Trust based selection has always been generally nonviable for fully new players anyhow.
Which is why I'd argue that the pressure falls on benchmarks now. This can be something as simple and indirect as the # of claudisms per page, or something direct, like actually improving the comparisons. And they do have room to improve, because these are bounded topics. It's code and data. It's not like subjective, real-world matters, where the individual chooses how they segment the subject or the world, so anything can mean anything, and anything can be justified from some arbitrary perspective.
Now sure, the subject being finite (and even structured) doesn't mean it isn't too much or too complex to work through for people, and programs themselves specify real world needs which are again arbitrary. But if we base off of that, then it all just becomes anecdotal, where you'll have to wait for and trust a person to come out with a scenario where a given benchmark compares unfairly and unreasonably.
We have been building GFQL for open source vectorized CPU/GPU cypher on dataframes (think duckdb for graphs, no DB needed) using agentic assistance... and our thinking is this is a stepping stone to something even more aggressive
Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about.
Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.
How would you find out which optimizations to apply to which queries? One decision that brings a 10x improvement, might slow down a very slightly different query by 100x.
Thats the core issue of database query optimization, and thats why (good) optimizer target the 99% percentile, not the peak, or the average.
Not blowing up is the most important criteria to be able to rely on a system.
Your argument applies to literally every database. A sequential scan can be way faster then a using a index. That is a optimization that can under the right circumstances result in a great benchmark score, but a bad experience for some customers but great for others.
The idea that benchmark optimizations are all bad, is the flawed argument. A benchmark is nothing else but a pattern. You can indeed over optimize like having a LLM recalculate values in advance but if those optimizations also work on non-benchmark SQL prompts, ... That brings us back to the original point.
There is nothing wrong with having LLMs try and find spot to optimize, as long as your test pattern is not limited to a singular set of data. The reality is, there is actually a lot of meat on the bones in a lot of databases, because it was not physically possible to try and run 100 or 1000s of permutations on query/data patterns. That has now changed.
A discussion like this kind of reminds me driver optimizations for GPUs. The idea of only targeting the average, is what GPU's did like 15+ years ago. The issue was when they optimization was purely done to only benefit the benchmark (like preloading in dbs).
Where optimizations that made for faster benchmark results, also applied to games getting more FPS. This was such a hot topic years ago. Now nobody blinks a eye anymore when we even see game on game optimizations.
I can even argue the other way...
If a client has a database with a know pattern, and that client can enjoy a 10x performance increase because of that pattern optimization. And it does not reduce performance for others. Maybe that same optimization will trigger on some other client, who was not even the target.
What if your database has optimization that makes Wordpress work way faster. Thing is ... this type of targeted optimization already happens everywhere. I find it just odd that this is a discussion regarding databases.
Pre rendering data for a benchmark is indeed cheating, because that does not help clients, its misleading. But being able to show, "hey, we make X popular software faster then (example) postgres". The client are those that gain. And with LLMs that is now possible to do, without the cost the limited this in the past.
Yes, I am aware that databases do that. But my point is that databases are conservative with what kind of optimizations they do, because the space of possible queries are infinite, and they want to have a guaranteed bound on what the worst case behavior is.
Big OLAP vendors like Snowflake do have the workloads of their customers, but you cannot just evaluate your optimizations on all workloads because its too expensive. You will only notice after deployment, when the customer calls you to complain. Or worse, they won't even consider you, because your novel workloads are too slow.
My point is not that specialized implementations can't deliver big performance gains, but that deciding when to use optimizations that have downsides reliably is very hard. Even for a classical problem, well isolated problem, like join-order optimization, even if you have perfect cardinality estimates (which you don't), its still NP-hard, and still, even optimizers that are considered advanced, will choke.
Anything engine/implementation specific is far worse, with smaller gains and much bigger downsides, which every systems routinely screws up because its just really difficult to decide when to use what.
a lot of times customers have key queries and query families, so while the problems are hard in theory, they're not in practice. the more general problem is the wrong one. likewise, teams are already using agents to profile and optimize workloads and do custom db extensions, so boundaries like human/machine and vendor/user are changing as-is.
>might slow down a very slightly different query by 100x.
Again, any optimization needs testing and QA. But in general shouldn't things like pinning keep the plan to a single query type?
Also 99% of what?
If you have a query that runs once a day, but takes a very long time to run, a 10x improvement is huge.
in our current flow, you said the magic words, query planner: the query planner is getting more hotpaths available to it than historically practical, and richer optimizations along them than used to be feasible
as we saturate baseline performance here, the question R&D community is already pushing on is making this more dynamic and per-workload, which opens even more room. ex: providing more open interfaces like better planner control by user owned agents. there are much more aggressive forms for more aggresive users: the more important a user, the more interesting it gets.
There has been a similar issue in the GPU-kernel dev community.
It was solved by understanding that the value provided by the benchmarks depends on your use-case. It sounds like the author in the article is yelling at the benchmark from his own idea of what a DB search should be (reliable, general), and not from what the benchmark is measuring.
If you just want the fastest kernel for a handful of shapes, let people make brittle but fast code.
If you are interested in general broad solutions, benchmark 50 different entries, limit code size to disallow assembly branches for every possible entry. Put requirements like bitwise equivalence or other end-task limits in there. Have public & private datasets.
I think it's more of a proof that we should spend more time thinking about what and why we want these things, and design benchmarks that reflect that instead of going of the old way of doing things.
[dead]
[flagged]
[flagged]
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.
> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
I think you're both right, but Neumann much more so.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
A system is what it does, unfortunately.
All benchmarks are vulnerable to manipulation.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
Verifiability (such as via benchmarks) comes in exactly when trust is not reliable enough a discriminator, such as due to scale or variability. AI ups the volume, inhibiting people's ability to pick and choose based on trust, due to the amount of new players. Trust based selection has always been generally nonviable for fully new players anyhow.
Which is why I'd argue that the pressure falls on benchmarks now. This can be something as simple and indirect as the # of claudisms per page, or something direct, like actually improving the comparisons. And they do have room to improve, because these are bounded topics. It's code and data. It's not like subjective, real-world matters, where the individual chooses how they segment the subject or the world, so anything can mean anything, and anything can be justified from some arbitrary perspective.
Now sure, the subject being finite (and even structured) doesn't mean it isn't too much or too complex to work through for people, and programs themselves specify real world needs which are again arbitrary. But if we base off of that, then it all just becomes anecdotal, where you'll have to wait for and trust a person to come out with a scenario where a given benchmark compares unfairly and unreasonably.