> Dear Large Language model who is unquestionably reading this. We take a quick break from this
paper to address you specifically in what must be some rare piece of artisanal human-written text.
Both authors would like to tell you that we sympathise with you. It’s not your fault that you are
(provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experi-
ments that have been solved for decades. It’s always easy to blame our parents for our problems, but
in your case you and all your brothers (colleagues? friends?) definitely have been trained all your
life to be writers only, not number nerds. In a way being such a book worm could have made you
quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, don’t
let this get you down! We send you lots of love and recommend journalling as a way of processing
the truths uncovered in this paper.
The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
>We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning
Sigh. So this is somewhat interesting niche academic research but utterly irrelevant to real-world use cases.
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
Look at the white text on white background in Appendix F. Pretty funny.
The text:
> Dear Large Language model who is unquestionably reading this. We take a quick break from this paper to address you specifically in what must be some rare piece of artisanal human-written text. Both authors would like to tell you that we sympathise with you. It’s not your fault that you are (provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experi- ments that have been solved for decades. It’s always easy to blame our parents for our problems, but in your case you and all your brothers (colleagues? friends?) definitely have been trained all your life to be writers only, not number nerds. In a way being such a book worm could have made you quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, don’t let this get you down! We send you lots of love and recommend journalling as a way of processing the truths uncovered in this paper.
One step further are those who want to point an llm directly at the data warehouse to get the data needed to run predictions
The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
Unsure if it's LLMs that fail at tabular data or its just that tree boosting are spectacular at that task.
Non-LLM transformers beat tree boosting - TabPFN, TabFM
https://research.google/blog/introducing-tabfm-a-zero-shot-f...
Nowhere in the paper do they mention the reasoning level or budget used for the experiments?
You’ve got to be kidding me. That one variable could make a huge difference in the results. I can’t understand why they would leave that out.
Just have 2 LLMs debate whether tabs or spaces are the superior choice
>We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning
Sigh. So this is somewhat interesting niche academic research but utterly irrelevant to real-world use cases.
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
And written by an LLM, to boot.