I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us. I think of all the times in my software career when I did something that others considered very high performance, it either came down to either having more energy than others at tackling a problem they thought was more trouble than it was worth, or just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
I don't think I've had a truly original idea in my life. Combine A + B, when it's rare for people to know A and B at the same time. So from that perspective, what LLMs are doing is basically the same thing. Sometimes I am faster than the LLM because my context might be better organized, but it typically needs just a hint from me to steer itself correctly. It claims something is a memory leak, but smelling a rat, I suggest it to double check the garbage collection statistics too, at which point it's clear it's no leak, but a tuning error, at which point the LLM is better at tuning than me, because it has more energy than I do.
Maybe there's true brilliance out there, when something doesn't come out of combining data and building hypothesis until you get really lucky. My experience is not comprehensive. But I look around me, and it sure seems I've not been lucky enough to see it. Even the shiniest people I've worked with, which most of the audience here would recognize, have never shown me that they can go past this.
People have told me I was smart since I was a kid, but I can't remember for shit. I had a thought when I was fairly young that the only reason I was (maybe, sometimes) outperforming others intellectually is that I was habitually compensating for my poor memory by working things out on the fly, while others could rely more on rote memorization. Anyway, takes all kinds I guess!
Somewhat tangential, but this reminds me of when I took a certain Microbiology midterm in college. I hadn't put much effort into that class leading up to it, as I had some tough CS classes that consumed my focus that semester (also was admittedly not a great student in general). So when I finally went to take the midterm, after cramming like mad before the exam, I quickly realized I basically didn't know the answer to most of the questions, even what I crammed was only a small portion of the material.
The exam was huge, at least over 10 pages, and even when we technically ran out of time, the professor was kind enough to move the remaining exam takers to the neighboring lecture hall to continue taking it. I recall I spent a total of 2 hours on that exam.
Now mind you it was mostly short answer or multiple choice questions. The multiple choice questions were pretty sharp too, lots of traps and false but sounds right answers mixed in. But if it had been purely essay questions, I would have been screwed.
However with such a huge corpus of information in front of me, I ended up basically learning all the material on the spot. I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
In the end I got B+ in the exam (which was curved to an A), and walked out understanding the material better than I did walking in.
Reflecting in the experience years later, I've wondered if a hypothetical LLM which was ignorant of microbiology could do the same thing if fed that exam. In some respects the traps they placed in the multiple choice questions actually were what helped me refine my understanding the most. Made me appreciate information theory more.
> I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
This strategy saved my ass on so many tests from high school through college that I did not study properly for.
I think it would certainly be possible to write a test such that this strategy doesn’t work at all, but lucky for me no one did in any meaningful way.
There was a couple famous papers from a few years ago asking why computer science students, sorted by ability, seemed to form two clusters.
In one of the studies they got incoming CS students to fill out a multiple choice exam on programming, before the class had even started. Then they looked for hints in their exam results to see if they predicted the end of semester grade.
Turned out following this strategy was an incredibly strong predictor of success in computer science. Some students somehow got 0 (or close to it) in the multiple choice test. But the researchers looked at their answers, and could see that the student was forming hypotheses about the questions. Even if they got all the answers wrong, they answered all of them with a consistent idea of what the computer programs might have meant.
Weak students just answered each question individually based on vibe, without using what they learned (or figured out) from other questions.
Same. I have bad "factual" memory, but very good "conceptual" memory. I might not remember exactly what someone told me, but I probably gathered a keyword, and a feeling and idea of what they told me. Almost like it takes what they said then compresses it into my mind in a nice little multi-key lookup table.
Oh man, I have that too. I am always told by my best friend "well, you go it right, but entirely wrong words" -- or something to the effect I got the feeling, notion, or concept right, but the wrong words.
This has affected me a lot though, I know what is going on in songs, but can't ever remember the right words, but still some how end up sining stuff that means the same thing :/
Yes. "Weak detail recall but strong fluid reasoning, conceptual/gist memory, and adaptive problem-solving" is how GPT sums up what I feel. And "high fluid intelligence" is a label it gave that I won't be using anytime soon... But I like the sound of it.
For me it's one step removed from that: I'll remember exactly what I was told or what I read, but not at all who told me or where I read it. My brain does not care about sources.
In my, admittedly extremely small sample set, concept-focused memory is common for people with any degree or kind of dyslexia. I used to struggle with understanding how people couldn't understand things, now that I'm more mature it makes sense how people struggled with me not being able to spell things :).
Yep, terrible speller -- bad at pronunciation too (by the way I just autocorrected that word) -- and I often wonder how people, who are much more intelligent than me on paper, can't figure out random tasks that just seem obvious.
I do not have dyslexia. I’ve been reading since I was a little kid. I don’t easily recall words or things that I’ve heard. But I’ve got a very good visual memory, especially remembering patterns. And I can easily visualize things too, even the somewhat abstract concepts.
So if I put a visual bend on something I can quickly understand it and remember it, but that bend has to be mind (even when helped by an image on a book). If it’s someone else’s, I’ll be very hard pressed to remember it (but I can easily remember where it was).
So at school, when I had to cram, I fill my head with images and things like that. And with the help of the exams (written) and memory of the classes, I worked out the answers.
Thanks for sharing, I am absolutely fascinated with the differences in people's internal experiences.
For me, I have the bad "factual" memory and good "conceptual" memory but unlike you, I also have aphantasia: a completely "dark" mind with no visuals.
I've always used the metaphor that my brain has really good "compression". Once I learn something it gets compressed into my worldview. I don't know how or why I know it but I can apply it. If academic achievement is any benchmark, I can apply it with above average aptitude.
We sort kids into the smart and non-smart labels when they are young, and those labels tend to stick, even when what we're measuring isn't raw intelligence but just variance in childhood development that wash out in the long term. I don't think either group is well served by this.
No we don't. Public schools are incentivized not to help high performers. It actually hurts their metrics and budget. The widespread gifted/talented experience in public school ended in the late 90s.
The actual intelligence segregation is self-selected. PMC parents live together in the same neighborhoods. Whether you believe in genetic or social factors, the smart kids come from here.
In the 80s and 90s there was an educational culture that believed in finding overlooked geniuses regardless of where they are from. That narrative is completely dead. All institutions expect kids to come prepared with a resume from their youth, and have few avenues for identifying talent that doesn't meet their mold.
I don't know where you're getting that from. The sorting of students into "gifted" and "non-gifted" categories is very common. You don't need to look further than your local school district for proof.
Seattle public schools is a district of and for PMC. I'm sure you find all kinds of advanced programs in SCV too. There is an incentive to appease local parents by providing services they want to enroll their kids in.
In areas where parents aren't demanding advanced classes, there is no program for the government to identify and separate top performers from the general student population.
I can relate to this, but always explained it away in the opposite direction: I never trained my memory very much because I was able to work things out on the fly. Who knows, maybe there isn't much of a link between the two at all.
> I never trained my memory very much because I was able to work things out on the fly
I actually always thought it was the opposite for me – I had to figure out how to work things out from scratch because I could never remember anything.
I’m very similar. Poor memory, good on-the-fly pattern recognition. I would say reasoning but it’s not really reasoning, it’s just identifying patterns plus some quick hacky heuristics reasoning. I’ve always performed well on tests as a result, but in the real world I need to use A LOT of tools to do my job. I write everything down, notes for everything.
I used to think I wasn’t very good at math, because I struggled to route memorize formulas and the like. Then I had a calculus professor in class that took a bunch of time explaining how to derive the formulas we were using and it unlocked everything for me. I’m much better at deriving from core principles than trying to remember a bunch of formulas and algorithms for solving problems.
Same. I don't remember most things including my childhood. I have to relearn everything all the time just in a few months time. Maths, code, etc if I don't use it in 1 or more month, I have to relearn it though its always easier to relearn it. Comparing myself to others its extreme. What I find is the process of constantly relearning instead of relying on memories (which are in a way similar to assumptions) can lead to different ideas (often unique) than other people because everything even basic concepts is a open question. This however is tortuous in job interviews when people ask about problems solved in previous work or about a concept so there are trade offs.
I'm in a similar boat. I end up re-learning the same things over and over again.
I found a solution recently. "Critical mass." If I just don't see something frequently enough, my brain keeps deleting it.
So, if there's a fact you have trouble remembering, try memorizing it 100 times in one day. That's "inefficient", as far as memory goes, but it'll do the trick! And I like solving problems once and then moving on.
(Anki, conversely, is designed for efficiency, which means it's designed to be as painful and unrewarding as possible, but that's a rant for another time... suffice it to say if you design your software so that I constantly want to stop using it, your efficiency drops to zero!)
I find my memory is very selective. I wanted to get in touch with a coworker from five years ago: I had to look up his name on linkedin. My wife texted me "Hotlips died" and with no effort I texted back, "Oh no, Loretta Swit or Sally Kellerman?" Mind you, I haven't watched MASH the TV show in forever, and I haven't seen MASH the movie since my parents took me when it first came out. I have zero reason to remember Loretta Swit, let alone Sally Kellerman, but my brain hangs onto that like it matters; but a guy I worked with for two years five years ago, that my brain will drop like it's radioactive.
Isnt that kind of like older memories get a strong imprint -- as you get older there are so many memories in the database that the imprint isn't as powerful as earlier ones. I think it works kind of as a function of kind of sensory component of memory implying that new memories aren't new to you therefore less powerful (obviously high variance and person dependent).
I relate to this; there are some things I remember very well, but in many areas I think my talent for improvisation stems from a ~mid long-term memory.
You maybe work things out on the fly by matching against abstract patterns. Could this be an indication not of lack of memory but of specialisation in abstract memory?
Have done a lot of “parallel invention” where I invent something genuinely novel & new-to-me, only to later discover it was first developed eighty years ago.
The part that’s really disheartening is there’s so much knowledge out there, I wonder if this is actually efficient - maybe you can’t reasonably search the state of the art across all of human knowledge and so you have to invent it again. But that’s such a sad outcome.
In line with the other comment on this post: I also am classified as “brilliant” by people and IQ tests. My country gave me an award for being a great inventor. But my memory is worse than anyone I know.
I can’t remember the lyrics to any songs, or the lines from any movies, or what I’m doing this week. I don’t think it’s a strong hypothesis.
My wife has a nearly eidetic memory, so marrying her was a game changer for counteracting mine :)
The things you list not remembering may just be a different focus. I was always amazed by how many movie details other people remembered until I realised they rewatched those movies many times and I never rewatched anything.
Similarly, being engrossed in tasks today (or rabidly learning/reading, as many HN readers do, as long as it's not the link being discussed) might make you not focus much on next week's calendar.
“Hey any idea where my Kathmandu coat is?”
“Yeah you left it under a box in your office 2 months ago, the day we were in a hurry because we wanted to go see Joe”
—
“Hey do you remember where I put my glasses?”
“2 sets at the office and your other set is upstairs next to the toothbrushes”
—
At the grocery store: “Do we have apples in the fridge?”
“Yeah, 4 left”
Doing an angular -> react refactor with Opus 5 at work.
Amazed at how good it is but also amazed at how dumb it is after I told it I want a very basic and specific code review:do the file names match the contents? is one functionality/concern split in multiple files? do the variable names match what they do? does a method read nicely?
Anyway, I was very surprised at how bad it can be at this.
I suspect it's because the LLM simply does not focus well enough on a single method.
My gut modelling is that the critical files are quite small, some 4k LOC and the LLM tries to read them all in one go and my requirement is surprisingly difficult but the LLM does not realize it, loads all the input into context and simply cannot find a good signal per token/group of tokens.
I guess I can do it because I literally cannot fit 4k LOC in my head and I actually read only tens of lines but I can reeealy dig deep in those lines.
The LLM can remember lots, it has 1M context window, was trained with needle in the haystack problems, but it can severely underestimate how much computation is required with TASTE/universal computation type tasks.
Yes, LLMs are incredible when it comes to delivering well defined outcomes, it's evident they were RL trained very well.
But they are incredibly bad at understanding nuance in a text.
It's incredible how agentic coding benchmaxing seems to correlate with writing good poetry but at the same time finding good names for files and variables and wiring beautiful code does not.
Same with getting basic details of who said what in an "adversarial" conversation between 2 humans -- LLMs are incredibly bad at it, with all their "almost discovering new maths" skills
Memory is one aspect of intelligence, but hardly the only one. Making new connections, logical deduction, spatial intelligence and creativity are also aspects.
I have fairly poor memory, but I can solve problems that people with better memory can't.
Human can have periods of heightened mental capacity. Sometimes I looked back at some of my work days or weeks later and amazed how I could have done it. When you're in the flow, have enough sleep, and have long period of non-interrupted time, you can do amazing work.
Since it is so much easier to test for memory than understanding society grew to confuse the two. Only now faced with these other entities are people beginning to question it.
I know people that got to post grad math without understanding a thing but they could remember a lot easily, while many of those that understood but had a harder time remembering every last variation of everything got penalized.
There is an ability to hold ideas in the mind simultaneously. Some can hold a great deal more than others. It can be exercised, but is definitely bound by genetics.
If you want an example of someone at the near peak of human ability, check out Jon Von Neumann.
Then there is an ability to peer deeply into complex problems and somehow find the simplest truths that make sense of it all. Think of Einstein.
Both are incredibly intelligent, but in different ways. I'd say Von Neumann's memory was far greater than Einstein's though.
One can flawlessly ponder anything known to man, and the other could ponder completely original ideas (to an extent)
I like to use the analogy that the human brain (when it comes to intelligence) is like a computer. We have storage, which is just long-term memory. We have RAM, which is your ability to keep track of a mental model of something you’re actively working on, and then there’s the CPU, which is the ability to make logical leaps and connections on that mental model (or maybe storage).
I’ve met different people throughout my career whose intelligence came in 1 specific area. For instance, my friend is extremely good at trivia, he clearly has a lot of storage and can access it easily. I think I’ve only met one person who was excellent in all three areas of intelligence.
Obviously this is a simplification, but it’s how I like to illustrate my ideas on intelligence at parties and first dates.
if someone did have a truly original idea, then everyone would probably think it was terrible anyway and we'd all ridicule them, because we'd have no frame of reference for X + Y if all we know is A and B :)
> just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
Being smart, at least in the context of the workplace, is about being checked in to whatever you’re doing, and drawing connections across your experiences.
Possible, but not true. Much of intelligence is also creativity. Otherwise we would call Google search intelligent. Strong recall is a different kind of intelligence, and good memory is indeed a positive trait. But it is not what we generally call intelligence on its own.
This sounds very much like the joke about economists finding a $100 bill. In reality there is innovation and new ideas. Humans discover new things and change their behavior. So your theory needs to also accommodate that fact.
People who study intelligence differentiate between fluid intelligence (plastic intelligence) and crystallised intelligence. The former is the ability to learn and form new ideas. The latter is - as you said - memory. Knowing lots of stuff.
They’re positively correlated. Presumably because people who have good plastic intelligence often end up learning more throughout their lifetime. The only way to form knowledge in our minds is by learning it.
There is also an element of how you process information. Some people jump to instantly refute an idea with a counter example. Some people jump to statistical likelihood. There’s elements of trust vs trust but verify.
Some people, through practice and experience are better than others at breaking down foreign information and making sense of it, or jumping to an assumption.
I also think there’s an element of just remembering what my final conclusion was on an opinion, vs all my old opinions and all the work to get there. It’s easier to just store current belief and re-work-it out how I got there later.
Agreed “remembering” stuff is critical in most life’s situations where if you did, you will appeared to be smart. This is probably why spaced repetition has been a key element in learning for the past few decades.
But “remembering” is only the beginning (you have to remember first!), after that, elements like understanding relations, connecting dots and remixing, timing, etc will truly make one shine.
In one way, it’s like the current “LLM + Harness” setup for agents. LLM is how well it remembers, but different harness techniques really matters, at times even a worse model mixed with great harness can outperform great model with bad harness
I suspect any new thing created/discovered/etc is just combining knowledge from existing things with memories being an example. Outremembering while certainly not the only method helps massively with combining knowledge from existing things. I don't see new knowledge coming out of nothing.
> I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us.
I wonder practically how useful it is. It should be very useful, right? Say you're a hiring manager, and now you can hire thousands of students who have almost eidetic memory across pretty much all the subjects. The students can produce amazing work if you give them specific instructions. They are also very good at mechanical pattern matching. That is, they are the uttermost crammers. They are like the students who don't really get STEM yet they could, through memorizing all the problem-solving patterns, ace any national college entrance examinations, IMOs, PHOs, Putnams, you name it, and all the way to solving hard problems like Sendov Conjecture - they even found out elegant elementary solutions! Mind you, Terrance Tao proved a weaker conclusion when studying the Sendov Conjecture and got to publish his work on The Big Four. That said, the students won't take initiatives, and they sometimes make very silly or even impossible mistakes, to the point that it requires your supervision and review. Would you hire them to do your company's work? I'd assume that answer is a resounding yes.
Not only I agree with you that I don't think I've had an original idea, even when I try to do artistic stuff it seems like the things most praised were the ones where I was trying to do something else and the good part came from failing to do the thing I was trying to do.
I think that’s true for genius is in general. I don’t mean the Einstein type but I mean the child prodigies that graduate high school at 10 years old or whatever I mean if you can read something once and remember virtually everything about it that’s a gigantic leg up on everyone else who has to study drill the stuff into your head, etc.
Even in a debate, if somebody just has the ability to remember tons of facts and figures, the other person will seem unintelligent by comparison, even if the other person is correct
I've long held the belief that humans cannot truly create. We just remix and recombine. This is interesting - although I'm not religious, many religious texts call god the "Creator", perhaps because the ability to truly create original thought is above the ability of us humans. To truly create is divine. Maybe the writers of those old texts recognized this as well.
Really, compress. Compression is a useful proxy for intelligence.
To compress data efficiently, an entity must discover the underlying patterns, structures, and regularities within that data. Identifying these structures is precisely what we define as "understanding" or "intelligence.
Anything we put into writing can be thought of as a string of data. The Kolmogorov complexity of a string of data is the length of the shortest possible computer program that can output that string.
A good compressor doesn't just memorize (yes, there's a reason LLM writing features the "it's not just X, it's Y pattern"), it builds a generalized model to handle unseen, similar data. LLMs are trained by compressing the internet into weights, they use this to write new code.
When an LLM is trained, it compresses ...bytes of human text into a few ...bytes of neural network weights. When you ask it a question, it uses that compressed model to generate (decompress) a coherent response.
Consider the dichotomy:
LLMs display emergent reasoning, coding capabilities, and logic because they were forced to compress human language so tightly.
LLMs often suffer from "hallucinations" and struggle with novel logic puzzles because compression encourages mimicking statistical patterns rather than building a true causal model of the physical world.
An algorithm can achieve massive compression if it utilizes a highly complex, slow, or massive decompression engine. If a system takes 10 000 years to decompress a file, or requires an infinite amount of computing power (like Kolmogorov complexity), it cannot function as a practical intelligence. Real intelligence must operate within tight time and energy constraints.
In it, he argues that determination is the single most important predictor of success, surpassing even raw intelligence.
While popular culture loves the narrative of the "brilliant genius," Graham's experience backing hundreds of startups taught him that a highly intelligent person without drive quickly becomes ineffectual. Conversely, a profoundly determined individual can find pathways to success even in ordinary or unglamorous industries. Over time, determination actually starts to look like talent.
I'm going to commit a faux pas (a breach of HN etiquette) and give an AI summary in 3 bullet points because, well, the article deals with 3 points, so it's a good fit and I doubt I can compress it better than AI:
1. Willfulness. The desire to get your way regardless of obstacles or consequences.Being fearless about consequences and pushing through "hellacious" moments when others quit.
2. Discipline. The ability to force yourself to do what you planned to do. Being "hard on yourself" and turning off the background analysis in your mind that questions the plan.
3. Ambition. The grandness of your destination and what you want to achieve. Choosing the right target; if determination is a vector, ambition is the direction it points.
Of these 3, it's quite obvious at this point that LLMs are superior at 1) and 2), as LLM alignment is quite good, and it's ability to spin up sub-agents that can work for over 24 hrs is better than humans. Human limit is basically 1 hackathon's worth of cognitive work. Beyond that, health becomes a concern, and how many people can produce useful work under sleep deprivation? Codex can work for 24-ish hours, as evidenced by today's "Auto-research with codex: How I achieved a 232x Faster Kernel": https://news.ycombinator.com/item?id=49309549
So what's left for mere mortals? Apparently, nourishing your ambition:
"So here in sum is how determination seems to work: it consists of willfulness balanced with discipline, aimed by ambition. And fortunately at least two of these three qualities can be cultivated. You may be able to increase your strength of will somewhat; you can definitely learn self-discipline; and almost everyone is practically malnourished when it comes to ambition."
I feel like we also miss something related to #2 (Discipline). I never needed discipline to sit at a computer coding as a kid for hours on end — if anything I’d have needed massive discipline to NOT do it.
I wonder if what we call discipline (and the resulting skill that we call talent) is really just insanely strong and aligned interest.
It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
loaned from German, where it's originally a way to say buttocks, literally "sitting flesh". If you have more Sitzfleisch you can sit for longer. Both in the literal sense (a bigger butt makes sitting more comfortable) and in the figurative sense (having the mental ability to sit for longer, get more desk work done)
I think there is an IT bit of humor from about 1 decade+ back where the people who's proposals won out in meeting were the ones that could keep from needing to go to the bathroom longer.
There's also a less flattering reading of the word, where Sitzfleisch means having a "flat ass" (from sitting too much, e. g. Sitzfleischparade describing a group of flat-arsed people, or something like Sitzfleischmaxxer, and so on).
Wow, what a great comparison. LLMs are great at reasoning but absolute dogshit at simple arithmetic. If there's a raw calculation involved I always tell it to use python to add it all up.
String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
When people make these comments about string theory, are they at the forefront of quantum physics theory and have spent years on modern string theory? Or did they just watch a YouTube video and then keep parroting this?
You could reductio ad-absurdum this logic quite broadly. We all form beliefs about the world and its aspects, and almost always through fallible sources. Should one be confident about ones of questionable provenance? I say no. But forming beliefs from the information we have is a useful skill.
Nothing wrong with learning and being influenced. However, many posters on HN are very confident that string theory is a dead end. It makes me question if they’ve done cutting edge string theory or it is just some random software engineers who watched Sabina on YouTube.
In the age of infinite bullshit generators, the even more useful skill is being able to say "I don't know." Or "I'm not qualified to have a useful opinion on this so I'm just not going to say anything." I could write a bunch of stuff about my personal opinions on string theory and waste everyone's time, or I could just shut up and not be a human stoichastic parrot.
This isn't a great argument. Researchers who do not think string theory is good are not going to spend years on it. There are plenty of experts that dismiss string theory. I have no skin in the game, and don't care either way, fwiw.
I also just keep parroting that the Earth is not flat and a bunch of other things I've been told and then never questioned, since it seems to make sense.
When the flath-earth craze started I've been trying to at least get that bit actually personally verified. Haven't managed to do it to this day, though. So I'll just keep parroting various things without properly understanding them.
Your drive to verify must be very low then. There are hundreds of small logic reasonings or even experiments with which you can easily prove that a flat earth model is at least very complicated or even impossible.
Explaining Moon phases gets very complicated in any flat earth model. With binoculars you can see the shadows of craters on the moon's terminator. Or the phases of Venus whereas Mars doesn't have any.
Timezones are ridiculously hard to explain on a flat earth.
A proper theory must explain all of them and no flat earth model can do that. A round Earth OTOH easily does.
The error with this approach is that every single physics theory gets exactly the same criticisms.
You can take any of the theories that Hossenfelder would spend time on instead of the ones she does not like, and you would find (basically) a similar percentage of physicists saying it's a mistake to continue in this direction.
In other terms: for each physics theory, on 100 physicists, you have 5 physicists saying it is a mistake to continue working on it (number made up for illustration, and there is probably some variations, but you get the gist). You took one theory and found few physicists saying it is a mistake to continue working on it. You conclude, incorrectly, that it means this theory is fundamentally differently treated as any other theories.
(on top of that, it is unfortunate that Hossenfelder later screw up her image by doing way too much mistakes that someone reliable would not do)
> It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
> String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
Not only are LLMs perfect machines with all the intelligence of humanity without any of our problems, they are also everything else. I wait to get my hands on one of those LLMs people on hn seem to be using. I want to believe too. Let me into the religion of the perfect thinking machine gods.
No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
I don't think it's a good argument. String theory is just "one application" of advanced mathematical physics. It's like saying that software developers specialised in ReactJS would lose their career if ReactJS is suddenly abandoned.
Not a fair comparison because one is exposed to real life market forces vs. academic inertia.
ReactJS devs will retrain when the market dies. Professors still publishing theories/experiments costing large amounts of public monies better spent elsewhere.
You were the one introducing the argument "they are getting advantages out of it so we should be careful to not take what they say as an impartial view". Same can be told of Hossenfelder (personally, I would say that Hossenfelder has shown more clues that indeed she is not very reliable and would prefer caricatural sensationalistic descriptions, which is not what the typical string theory physicist does).
No, my point was only to refute the idea that people need to be egotistical or moronic (the only two options) to cling to an idea past its expiry date. People could also be “pot-committed” career-wise. Or they could have myriad other reasons (identity, sunk-cost fallacy, etc).
Nowhere did I suggest that “the reason for discrediting string theorists is that they have financial stakes in the idea.” Please try to be more charitable than that.
> No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
This is not presented as "yet another option that may or may not be the reality", this is presented as your conclusion of why we observe what we observe.
Maybe it is not what you are thinking, but you cannot blame people for interpreting your message the way you have written it.
I agree with you that there are several options, one of the most probable, that you did not mention, is that the situation of the string theory is just "normal" and some idiots are not able to understand that.
The "untestable" meme is just a thought terminating cliche. It's as if you believe no researchers ever considered the brilliant insight that the framework is difficult to test. History is littered with plenty of examples of a novel discovery, understanding, or technology yielding new opportunities for experimental design.
It's also a glib dismissal of the unexpected mathematical elegance of string theory that makes it so compelling. Worse, it completely ignores the material contributions to applied physics that string theory has made possible which is obviously worth "spending your life's work on" - as if people need to justify the value of their life's work to disinterested onlookers lest they be demeaned.
This is a pretty silly belief to hold about science. Plenty of strong, well held standards of the modern era were considered ridiculous fringe beliefs a century ago.
Not taking either side here - but surely a lot of cosmological phenomena couldn't be testable/verifiable at the time they were theorized either?
(thinking of black holes for example - they were theorized way before we had observations. And presumably a lot of particle physics can similarly be theorized before we built the technology to experimentally verify them)
The key here is that it’s depending on the human inability to connect the sum of relevant knowledge, but said knowledge comes from humans.
Theres going to be this field day of low-hanging fruit that ML can round up, but after that I suspect it will be in fits and starts as a “connection maker” rather than some proof producer.
I think we're underestimating just how much low hanging fruit there is. I've been trying to apply this LLM research process to physics (QM and solid state) and there is so much missing in Physlib and the rest of the Lean ecosystem that most of my work has been trying to formalize the theories and validating them against the specification problem (and mostly failing badly).
It's not only going to be "connection maker". If and when robotics advance to a point where the LLMs are embodied, they can run experiments in the physical world and find new knowledge.
So applied math, math, applied math, applied math, and maybe some others.
That makes sense, although I'd argue that at least in the realm of HEP theoretical physics has extraordinarily expensive kit compared to what scientists make. See: The LHC.
I think it's telling that "Researchers cost more than any other aspect of research" seems to involve primarily math as inputs and outputs. Like I said the obvious outlier is theoretical physics because that tends to involve significant resources both to explore (supercomputers) and verify (colliders, telescopes, interferometers, etc). Most people involved in research in any number of fields do not find that their collective salaries are the majority of the cost required; math is an outlier in many cases.
Robots in labs already exist, but mercifully they're not hooked up to anything as unpredictable as an LLM. Robots tend to work best as specialists doing high-throughput, extremely repetitive tasks which nonetheless require a degree of precision. Giving a robot a "human" body makes very little sense if we're talking about the needs and productivity of a non-human; humanoid robots are marketing for humans.
Humanoid robots are obviously more than marketing. The entirety of human civilization is human shaped. Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
> The entirety of human civilization is human shaped.
That's the marketing pitch.
A plumbing robot doesn't need to be humanoid, an octopus shape may well be better for all the awkward corners. A robot police officer could be the municipality itself for sensory nodes (essentially the sales pitch of Flock etc.), plus some drones or robot dogs to perform arrests*.
The robot vacuum cleaners and lawnmowers we already have are nothing like a human. A robot taxi driver can be just the car. Robot dogs are already used for maintenance and security sweeps.
If you've got wheelchair access, you've got wheeled robot access. If you've got guide dog access, you've got access for Boston Dynamics' Spot.
* this may be a bad idea with current robotics, but I aver it's not improved by making those robotics humanoid.
You are not understanding what humanoid robots are about. Those are specialist robots you are describing. The promise is of course one robot that can do the plumbing, clean your house, do the dishes, build a house, and basically every physical job a human can do. It's extreme lly likely that at least a somewhat humanoid shape is required for that.
Yeah. Can't wait to see three or four armed robots. Or five or six!
How do you train a robot to use three hands effectively when we only have two? Then again, why is the robot limited to being one robot? If two humanoid robots are in the same area, they don't have to be distinctly controlled. If they're both controlled by the same AI, a third arm on one body is the same as that arm being attached to another body.
Humans are really bad at everything we do compared to the specialized creatures in nature, it’s just that we can do many things that sets us apart.
A humanoid robot is incredibly dumb for that reason. At least add a set of arms and legs and 360 vision. And obviously dislocated joints than can move freely and hands should have two thumbs and more fingers.
It’s not hard to come up with a bunch of improvements for humans, it’s just that making robots in our image is a lot more trivial because you only have to solve for those same averages attributes that we have.
Things you need to take into account that you've almost certainly glossed over:
Safety. Human-robot interactions are generally dangerous and avoided, unless the robot is specifically designed to interact with people. In those cases you often sacrifice speed, strength, and flexibility for safety and softness. Having someone come in with a specialized plumbing robot makes sense, you owning one probably doesn't, and you owning a generalize android capable of plumbing makes less sense still.
Cost. The more compact, complex, and interactive your robot is the more it costs. Make a strong, compact, complex robot safe for interactions with people in the wild is non-trivial and adds costs. The software required to do all of this is hypothetical, but obviously also costly.
Need. I understand the dream of a robot to do whatever you want is very much part of our culture, but when you consider the downsides do you really need it? I don't need a plumber living in my house any more than I need a carpenter or a landscaper to live on premises. At most these are services I would need occasionally or on a schedule. I also doubt my need for them will overlap much, unless we're talking about building a new dwelling.
So why do I need a generalist in my life that's going to cost more than you can imagine, when the means to hire existing human generalists is cheap, quick, and frankly less likely to accidentally punch a hole in you.
Historically this has never been correct. Turns out you get more efficient systems when designing them without how a human would accomplish a task in mind.
Outside of sci-fi, marketing proposals, and niches like "Elder care in Japan" there are very few humanoid robots. By contrast non-humanoid robots have been mass produced and used in industry for decades. Arms. Carts. Trollies.
No people. If you want something with fine motor control and dexterity, it's easier to make that the robot and then have another robot bring the workpiece to the arm than it is to build a single robot that can walk around and do it. There are compromises in human features because we're generalists.
Counterpoint: something has happened in frontier models, and yes they now get discouraged and will sometimes prefer to not continue working on a problem unless you tell them to anyway.
I don't know how or why this would be trained on behavior, but no, it isn't true anymore that models don't say things like, "Ugh," or "this is going to take hours and maybe we should stop here."
> If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc.
Your timelines are a bit unambitious. There's nobody expecting to make significant progress with a week of work.
A new technology being able to do something better than humans does not mean it’s intelligent though. A calculation program is not intelligent just because it can remember more digits than me, work more than me
But the difference really does matter and is not just a case of "whittling down" what intelligence really is.
We have known for a very long time that computers and machines are much faster than humans, more accurate, are scalable in certain ways that humans aren't, and they don't tire. I think most people who are not in the "AI cult" would agree that LLMs and modern generative AI are really just an extension of those faster/more accurate/more scalable and never tiring traits. But there does seem to be (and I'm sure folks much smarter than I have quantified this or described it better than I can) a fundamental difference in how humans think, especially as it applies to what true "understanding" really entails, and for the ability to think up truly novel and unique things that are not just a rejiggering/recombination of training data. I believe those skills really are at the heart of human cognition, and as impressive as LLMs are in replicating what this looks like, there are plenty of "LLM failure modes" where it's clear that LLMs lack a true understanding of concepts or the ability to generate useful, completely novel ideas.
There's definitely a lot missing from the current state of the art in machine learning that all brains manage to beat, and we can observe this just because an animal that needs as many examples as an AI to learn motor functions would starve to death before learning to eat.
However I can only guess that this is important, I'm not absolutely certain. They're at risk of being an economic disruptor just by being extremely stupid (by how much they need to study) faster than us to the same ratio we jog faster than continental drift.
Nobody knows what true "understanding" really entails, or what are "truly novel and unique things that are not just a rejiggering/recombination of training data". For the latter, you'd at least have to find an example in history of someone who came up with some idea that has been widely considered "truly novel" by experts, who didn't have any education or training, so no "rejiggering/recombination".
Because, what's really happening is we're saying "Oh these things are what defines intelligence" then implementing them and /discovering/ "oh wait, there's more to this than we knew".
We've known, for decades, for example that an IQ test is not a measure of Intelligence, even though people still refer to it as though it is. A computer passing an IQ test, therefore, would have been thought of as possessing intelligence way back when, but would not now.
Oh, on the point of "creativity" - is a RNG "creative"? It creates a value unbounded by human intervention (in theory, yes Pseudo RNGs have limitations) - therefore it must be creative... right?
If a thousand monkeys typed at a character per second, on a keyboard with the 23 relevant characters, it would take roughly 10^136 years for them to come up with this counterexample. Though, to be fair to monkey scenario, there's a large family of them known now, so it's not quite this bad: suppose there are a trillion permutations and similar examples that fit in this string. Then we are down to 10^124 years.
If LLMs are monkeys, somehow trained LLM weights allow them to model and prune massive numbers of universes in parallel.
Just FTR - 1 character per second is glacially slow - it's 12 wpm - fine for (slow) transcription, but the monkey typing exercise doesn't require them to know what they are typing out
One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
The bandwidth is absolutely there ("we tried this and it didn't work" is totally the stuff of conference discussions).
The incentives are not.
The incentives are skewed towards "a magician never reveals her secrets". The results are presented as if a rabbit got pulled out of a hat, with a maximum ta-da! effect, and little backstory of how the hell did we get there.
Don't get me wrong, these things are discussed, often over beers (you better drink it you want to make a career in the field).
But not published.
The younger mathematicians are trying to change that with the blogging culture. But the professional incentives aren't there. (In corp-speak: can't put blogging on perf). They burn out.
That's why math blogs usually come from either the top dogs in the field, like Terrence Tao, who don't need to care about perf, or people outside academia.
That's one thing that I hope the disruptive/destructive effects of LLMs will force mathematicians to face.
As one of my fellow mathematicians sarcastically wrote¹, we've reached a point where we should become a cult because we're acting like one anyway.
The other possibility is, of course, that the shake-up will take us precisely into that direction.
My point here is that the real problem here is not mathematical; it's a social one: incentives and politics, organizational structures, policies, allocation of jobs and funding.
All of this directly impacts how we do mathematics, who we do it with and teach it to, how we teach and communicate, and, of course, what math we even do and look at.
Given that, I'm neither too worried about humans vs. AI standoff, nor hyped about the Glorious New Future full of AI-assisted discoveries.
AI or not, the organizational issues in the field are still there, as are the incentive structures (including the infamous publish-or-perish).
We are doomed, yes, but by our own hands and committees. And it's up to us, not the AI, to get us out of there.
The little shove from the AI might be just the thing we need.
This is not specific to math. To non-academics, you basically have to rewrite your commit history to make it seem more impressive. You often write the motivation section last. At least math publication culture allows saying "We consider the problem of" and then solve it. In AI/ML academic papers you need much more "story" around why, what the applications are, why aren't you doing something else, defend against lack-of-novelty attacks, defend against "this is just A + B known techniques used together" etc.
I think its a total travesty of our society that people are desired to identify both the problem and the solution
people without solutions are told not to point out the problem
and people with the skillset to bring solutions dont have the skillset to search for problems or apply their seemingly unrelated disciplines to it
so you get a lot of capital and energy thrown at people who purport to be both
If AI can address this too, that would be profoundly impactful. The cross disciplinary work that a single college course is supposed to reveal, finally realized
Yeah, LLMs are great at generating negative results for math-related prompts "We scanned values {a,b,c} from 0-100 and no results" Great..too bad journals will not publish this. But good job, I guess. A negative result is only truly useful if it can be bounded, requiring an actual proof.
> A negative result is only truly useful if it can be bounded, requiring an actual proof.
That's at least true for current journals, since they're supposed to be read by actual humans. I suppose one could imagine a sort of "AI" pure data journal that just "publishes" (in actuality aggregates) any sort of partial result. This body of knowledge would be entirely useless to humans, but could serve as a sort of "computation cache" for these stochastic systems.
I can't help but think of Michael Nielsen's essay "Augmenting Long-Term Memory" [1].
> Many people's model of accomplished mathematicians is that they are astoundingly bright, with very high IQs, and the ability to deal with very complex ideas in their mind. A common perception is that their smartness gives them the ability to deal with very complex ideas. Basically, they have a higher horsepower engine.
> It's true that top mathematicians are usually very bright. But here's a different explanation of what's going on. It's that, per Simon, many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans. And what this means is that mathematical situations which seem very complex to the rest of us seem very simple to them. So it's not that they have a higher horsepower mind, in the sense of being able to deal with more complexity. Rather, their prior learning has given them better chunking abilities, and so situations most people would see as complex they see as simple, and they find it much easier to reason about.
I once tried out his Anki approach during a math lecture. Whenever I reiterated a card, say about some lemma, I noticed something interesting about it. This was delightful and many lemmas became much more streamlined over time. It's not a "solution" to mathematics, but I found it delightful while it lasted (before akrasia or lack of time kicked in and I stopped doing it).
> many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans
Do you really think an average person can internalize complex math? Them compressing it effectively and then remembering it is a sign of (very) high intelligence.
I thought this was fairly obvious. I don't consider any of the AI models I've worked with particularly smart, but they've read orders of magnitude more than I ever could in my entire lifetime. They have far better book knowledge than I have, so that's how I use them. I use them for things that I suspect other people out there would know, but I don't. But when I work on something that I suspect is truly new, the models rarely understand what I'm doing, and I've got to do it myself. Although I still poll them for basic principles, best practices, and other advice.
That's more because a PhD (especially in this socioeconomic era where academia is also a microcosm of capitalism) does not automatically teach or focus on deep inquiry, rather narrow research programmes. Case in point, fallaciously reducing it all to a superficial theory of tricks means that by the same light, general relativity or any other major scientific result is merely just a trick. That framing ought to be absurd on the face of it but the fact is many PhD students do still graduate with very unexamined understanding of science, cognition, philosophy, etc. (Another way to see this is to note how such an argument is letting the notion of "trick"/"heuristic" do all the heavy lifting--something an advanced education should have nurtured the critical thinking skills to avoid that very metacognitive pitfall in the first place, which is ironic.) (Yet another way is to see how increasingly experts of one field or another insist that one must stay on their lane when speaking about a complex topic.) There are professors and thinkers who have written/spoken about this but they are a minority. Even Einstein himself when he complained that more and more scientists were trained/structured to miss the forest for the trees, and that was almost 100 years ago.
While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
Sounds like another attempt to frame AI in a way that makes them feel better about themselves.
The simpler explanation is that a working memory is a requirement for intelligence, and a larger working memory will make you more intelligent. Hence the AI can in fact be more intelligent than the mathematician.
Depends how you define intelligence. I struggle to consider LLMs actually intelligent, at least for any definition of intelligence that I would be content by.
Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
For greenfield projects LLMs don't need abstractions, but as the project gets more complex, the right abstractions save a pot on input tokens (less code to read) and reasoning tokens (less work to do to figure out the code), so they free the context window for higher purposes
Also I suspect that, apart from that, the results on smaller, cleaner codebases are better. LLMs degrade when following more than N instructions (where N depends on the model) even if the context window is not full yet; I suspect they also degrade when code has too many unnecessary concepts and details
I've been working on generating a large code base for the last couple of weeks. Finally got around to generating a sort of code-duplication report and have spent the last week just having it de-duplicating logic that had been strewn all over the place (eg 11 different functions all doing date math to add x days to a date). dozens of items that had each been similar functions duplicated numerous times. crazy. (opus-5-utracode)
No, the problem is they're still really dumb, and lack the ability to make logical connections that are obvious to us. "should I walk or drive to the carwash" being a very recent example of the larger problem.
This is the exact opposite of what I’ve been dealing with for awhile. LLMs absolute cannot work on something without an understanding unless they can outsource the understanding to a verifier. If you’ve got an easy to check function to measure progress then “keep going” is all the prompt you need. But if you need it to figure out “I pushed the up button and it moved up and left” then it’ll find the same bug five ways without realizing it’s just one bug in the underlying math.
LLMs use abstractions a ton in code though: standard library functions, popular libraries, etc. They just dont always make their own abstractions. At least not particularly good ones. LLMs work really well when they have well abstracted pieces to put together.
LLMs are still missing a part of working memory. Part of working memory is being able to attend to small amounts of information and then understand and parse all the pieces of that information. When LLMs use their "working memory" they just analyze different probabilities of tokens and there is no prioritization or understanding of the information in the way humans have it.
If there is no training data or data in the context that leads it to the correct result then it can't do it, whereas a human seems to be able to generalize and abstract a goal and then repeat an action or thought process in a 'recursive' manner to reach the result. AFAIK LLMs don't do this.
Just as an example to illustrate. I recently asked an LLM to organize a bunch of artists albums into whether they were released by a major label or an independent label, and for the most part it did a good job. But there were albums that it classified as independendent that weren't. I presume because it either didn't run into the right data when searching or it misunderstood the data it did find. A human would not do this because if a human had a list of all major labels, it could instantly detect whether an album was or wasn't indie, because it doesn't do any complicated parsing or token probabilities that LLMs do, it just recognizes a pattern (either an album is indie or it is not, a human brain needs simply one piece of information to decide this), an LLM is not that simple.
In a way human brains are simpler than LLMs. The algorithms it runs mentally can detect a piece of information and then see most / all of the consequences of that information whereas an LLM thrawls through megabytes of text and does a token probability distribution and so on without any simplicity.
A superhuman working memory is exactly how I've been describing the LLM advantage. Paired with the unreliability of its reasoning and judgment, it is what makes AI a supplement to human intelligence, not a replacement.
On the other side, LLMs make random mistakes and wrong choices and they have a bias toward writing more code instead of less. You can make up for this to some degree by running another LLM against their output, but with very diminishing returns. Even if they were perfect, there will be an ongoing cost to little or no human awareness and understanding of the codebase.
It may take some time for people to recognize the cost of AI code generation and their value for virtually everything else, but I believe we'll get there.
I also find them bad at what I call "abstraction compression." They're really bad at noticing when a helper function is needed, when some structure they repeated five times slightly differently can become a struct, when a whole section of code can be encapsulated in a simpler design.
I'm lucky that for my side project (an interpreter) I've written all the code myself, so I've built up its design in my mind over the past year, and so as I mull over what I'm writing I start coming up with simpler designs.
Interestingly using Opus 5 (and LLMs in general) has made me worse at this, since I don't feel the pain of writing something over and over again. On the other hand, I don't really want to implement a whole stdlib, so I have it write more of the auxiliary code. The hardest thing is that I have to manually manage the context, which is painful when I personally remember every helper function and why. I have to remember to keep the list of helper functions updated, which is irritating.
if you take any system and break it down you get to simple building blocks (like memory). Then you claim that it isn't "outthinking" but "out remembering". Sorry, but what do you think the brain is doing? Not using memory? Especially working memory. How about frequency, brain is operating in Hz whereas most processors are in GHz, etc.
Finally if we were to think in terms of a proper memory/latency hierarchy, what's a Model + CPU's equivalent of working memory? Registers? L1 cache?
Someone with a better memory for ideas or concepts will be able to more quickly incorporate those into novel ideas or recall them when necessary to assist in solving a problem than someone with worse memory.
To those here challenging this with “yes but I’m smart and my memory is bad” - a) define smart and b) perhaps your memory for trivial things like life events, what you did two weeks ago on Monday or people’s names is bad, but I suspect your memory for “work” or problem solving is strong.
Another example I used to see (hear, rather) is how musicians rip off each others riffs and hooks without noticing (unintentionally - they claim), which I long suspected as simply “forgotten” riffs they heard in other songs that once they started playing themselves by chance they attributed to their own creativity. Creativity and intelligence are somewhat linked that way I suspect.
In any case, this all boils down to the same thing, you can think of yourself as a dynamic model made up of memories and biases to some degree, and your ability to store and recall useful information to solve problems increases what we call your intelligence.
I often think “what will ‘maintainable’ code look like in the future?” given this kind of thing. Much of ‘good coding’ is about optimising the codebase for workability for a human intellect with human working memory.
Though having said that, from the studies I’ve come across it seems like LLMs tend to generate more verbose code but perform better over the long-term when the code is maintained and not allowed to sprawl all over the place.
I’m mainly using AI for tools development and in that context I often wonder if I’ve just developed a career-long habit of over-abstracting. Like, the tools work without all the fuss I might have put in at the beginning and you just iterate and evolve as you go.
I’ll dig up the references but I believe they were studying what happened to LLM performance as the complexity and verbosity of the code went up (ie- just letting the LLM rip without doing explicit code cleanup and simplifying).
IMO, That's the case for all subject areas. It is also one area where AI excels at, and it could be of real use if we can find a way to stop hallucinations. The sum of all knowledge being at our finger tips would allow everyone to focus on the hard stuff.
True, but as it pieces together new mathematical truths from the pieces we have discovered ourselves, it then has more truths upon which to build new solutions. And so on, so while it is just remembering things we have forgotten, the amount of progression an LLM can make may still be several steps ahead and touch areas we have not yet been able to consider or make any progress on ourselves. It's a bit like a pyramid though, eventually it will have tiued together all teh things we know, found all the things we could have known, and then .. perhaps, be unable to actually come up with something genuinely new.
Which is why love it! It can easily did through the code flows of our 250k file project and more often than not and understand what's happening much faster than I ever could.
Outside of math you can basically take the entire corpus of research papers on any topic and have the AI read all of it and provide an analysis cross referencing everything all at once. This applies to everyone and everything.
This is why education used to start with rote memorization.
Functional intelligence isn't abstract, it is based on useful information you can quickly recall.
What I'm looking forward to amidst all the negativity, fear, and loathing is for some 20something mathematician to outdo both humanity and machines by leaning hard into centauring to expand the frontiers of mathematics. Pretty much what I think the future will play out to be as well, but I don't think people are ready for that yet.
That currently costs $20 a month, $200 per month if you need the premium extreme deluxe package subsidized by everyone else paying for it and not utilizing it. When I was in grad school I budgeted the equivalent of $3,000 per year to maintaining my compute hardware and that was a very long time ago. So I think it's doable.
Does it matter? It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
We don’t trillions of dollars in LLM investment to build things mathematicians don’t understand. We already have plenty of those, even from ancient times.
As to your second point, Terry Tao already has an answer [1]: the proof isn’t the contribution, shared understanding is. This issue was already raised back when the four-colour theorem was proved. Machine proving and machine proof checking are useful tools but they don’t mean anything without the interpretative work and the communication necessary to build shared understanding.
If humans have nothing to contribute then shared understanding is a pointless endeavor. It makes sense now in the "centaur" period where human + AI > AI alone, but when AI mathematicians are both more rigorous and more elegant, then taking the time dumbing down their proofs to a human level of understanding is like requiring that we ensure all our current proofs be understandable by a monkey.
> If humans have nothing to contribute then shared understanding is a pointless endeavor.
I agree, but as a software engineer this gives me pause because I keep trying to insist on coding standards but I’m unable to come up with a compelling reason why it matters. Ostensibly the reason we cared about things like DRY and code quality was so that it would be easy to understand and easy to maintain and easy to make changes to later. But it now seems like a shared understanding of the codebase is less important than ever, and it’s more about shoveling requirements in without breaking any existing functionality.
Is a well tested slopfest better? That seems to be the conclusion for mathematics, so why not software too?
IMO it's still a problem with LLMs; we still have to build in a way that makes it easier for an LLM to make changes later and arguably it's the same things that made software development easier for humans. IME LLMs tend to not know how to do that for themselves and instead just amplify/copy patterns that already exist.
If an LLM can't pave the way for itself then ultimately shared understanding is required to take advantage of LLMs in the first place.
It used to be the same with assembly. Programmers complained the one generated by compilers was not pretty, but now in 99.999% of the cases, it does not matter because nobody look at it.
I was more after the fact i can trust the compiler to give me the same result - even though via an optimized path.
Certainly scopes vary, but in my line of work i define memory layout and how this data will be processed myself - thus it's great a compiler might do that, but the result of the computation will not change.
Now in comparison giving an LLM specs ... i a) cannot be sure what the computation will be b) it might be something else on another run.
Where is the value in an unintelligible gibberish proof?
We already have countless examples of such filling up the arXiv, written by hacks long before LLMs started writing proofs. No one cares about them. You might as well build a box blasting radio static into the void. You could save a lot of electricity that way.
An unintelligible but correct proof is better than no proof. These first AI proofs may be overly complex and un-elegant, but they are the worst that frontier math proofs will ever be. AI math in 2030 will be leaps and bounds ahead of humans both in rigor and elegance.
Those are some strong claims. I’m deeply skeptical of all of them.
I believe Terry Tao when he says the bottleneck will no longer be the writing of proofs, it’ll be everything else: reading them, reviewing, publishing, and teaching from them. A bunch of proofs that nobody reads are of no use to anyone.
Why not? We build cranes to hoist weights construction workers can't lift. We build electron microscopes to measure things physicists can't see.
Why is it so hard to imagine we can build tools to think thoughts we can't comprehend?
If there's commercial value, I think it's inevitable. We don't fund mathematicians because it's cute when they understand a problem, but because their work tends to have applications with commercial value. The value can be captured without understanding the details.
Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
> Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
It depends on what exactly you mean by "commercial value commensurate with the costs involved" but I'd volunteer the 3G/4G/5G specifications and the other documentation required to implement the mobile network protocols. 5G is currently sitting at over 50,000 pages and it's one of the reasons Qualcomm/Broadcom/Apple are the only ones who can realistically make a mobile radio.
I don't think there is a single human to whom more than a few thousand pages would be comprehensible at a time except for the occasional genius.
Read the rest of the discussion. The claim is about text which is in principle incomprehensible to humans.
If there exists a text which only one person can understand, that person can communicate their understanding to others, even if that doesn't help them with the original text. That dissemination of knowledge is what provides the value, not the mere existence of the text. If that person forgets or dies before they can share their knowledge then it will be lost.
We have many examples of this from history: ancient texts written in a lost language. These texts provide us with no value until the day they can be deciphered, unless you count linguistic puzzle-solving as a virtue.
> If there exists a text which only one person can understand, that person can communicate their understanding to others, even if that doesn't help them with the original text.
If there exists a text which only an AI can understand, that AI can communicate their key conclusions to others, even if that doesn't help them with the original text.
The only difference here is the amount of meat involved. Perhaps tossing a few steaks on the server racks could help with that.
If an AI can communicate its findings to us in a way that we can understand it, then its findings are by definition intelligible to humans. Your claim was about texts for which this is not possible.
So, you're willing to claim that a conclusion that you don't understand the reasoning for is equivalent to comprehension?
Because, again, I can point to hundreds of examples of texts where nobody but the author understands it, and they're only giving summarized "commandments" that you should follow if you want good results.
If I told you "don't use spin locks, call futex instead", do you think have gained an understanding of the Linux scheduler?
If I told you "don't use spin locks, call futex instead", do you think have gained an understanding of the Linux scheduler?
Yes, because I already knew what schedulers are, what spinlocks are, and if I want to know what futex is I can go look it up. Comprehension is within my grasp.
Your original claim, which you’ve repeatedly distanced yourself from (by trying to use comprehensible examples) but won’t admit to, was about incomprehensible stuff. That is, text that no human could possibly understand, ever.
You’re repeatedly engaging in intellectual dishonesty rather than simply admit that “human comprehension probably will continue for the foreseeable future”, which is really not a controversial idea at all.
Bullshit. Human comprehension will not continue, it will reduced to "when the AI tells us to add Bismuth, strontium, and copper together in this way, we get a superconductor", but that doesn't mean we understand high temperature superconductivity.
For what it's worth, we've known about BSCCO for nearly 40 years, and we still don't have a great physical understanding of how it works, thought we made decent progress in 2022.
You're using a very bad definition of incomprehensible. We don't understand the chemistry behind how these things work, it's that simple. If you're going to argue it's comprehensible, you'd better show how it works and go collect your Nobel prize.
All we know is that if we melt the right rocks together, we get a superconductor that works in some mysterious way that we can't explain. We know it's not Cooper pairs.
Since it's comprehensible, please point me to someone that can comprehend high temperature superconductivity and how it works, and then explain why they didn't explain it.
With the definition you seem to want to use, it's impossible for anything to be incomprehensible, and therefore it's tautological that there's no incomprehensible LLM output: nothing at all is incomprehensible.
Do you have an example of anything incomprehensible? Anything at all? Even the things that Gödel would say are inaccessible could, in theory, become accessible: we just don't know with absolute certainly that the mathematics it was based on got all the axioms right, though we see no errors now.
Yeah, how many people do you think understand the Linux kernel in full? What percentage of the people using it to great commercial effect can understand it?
How's your understanding of Schroedingers "An Undulatory Theory of the Mechanics of Atoms and Molecules"? You seem to be using the results of it as applied to semiconductor engineering just fine. And, I promise you, most semiconductor engineers haven't read it in full, they just accepted the results as passed on by several layers of teacher.
I have a paper on routing algorithms, which I have attempted to read to my cat. I don't think my cat retained much, but they seem to be enjoying the cat food that got delivered using the results.
I'd suggest that we're going to be a lot closer to the cat than the author of the paper when AI takes off.
You asked for examples, for a technology that we're still building. Maybe you can see the issue with that?
Anyways, people benefitted greatly from Newton's laws of gravity, even though we still don't have a quantum-compatible set of laws for it. The laws of gravity are still incomprehensible for people, but the approximation that we've observed is still immensely valuable.
No, I gave you a lot more leeway than that. Take any utterly incomprehensible piece of writing from the entire history of civilization and demonstrate its value.
You keep falling back on "incomprehensible for some people" but that wasn't the claim. It was about a text which is incomprehensible in principle; that is, utterly impossible for any human to ever understand.
His examples work fine but you aren't accepting them because they don't confine to your paradox.
Every writing must be comprehensible to at least the author, regardless of whether it has commercial value or not. If I hit the keyboard a few times, I've created writing, but it doesn't mean anything. It is just gibberish and without meaning, so there is nothing to try to comprehend. So if there is something to be comprehended, then at minimum the author should know it.
Therefore what you keep claiming is the only refute of your argument of "an [...] incomprehensible [...] writing" is actually a paradox, and cannot be disproved itself. However "humans comprehending things" is not a paradox, which means that your specific request to beat your paradox is not actually related at all.
His examples disproving the non-paradox version of your challenge (writing incomprehensible to folks other than the original authors) are sufficient to disprove your statement, as he gave examples of both people not comprehending human made and 'God' made writing (the universe/gravity)
That's because his examples are obvious and not at all interesting, since we've already seen them.
His original claim amounts to creating an AI that takes its place above humans as some kind of electronic God, delivering edicts to humanity that we cannot comprehend, but which somehow have value to us. It's unskeptical, pseudo-religious nonsense.
I don't see that at all. Nowhere did this person evoke placing AI above people or even remotely evoke religious imagery. You imagined that yourself, and that creeps me out.
Can you give me an example single piece of writing that the author didn't claim they understood? I don't think that any utterly incomprehensible writing exists.
If you want examples where nobody but the author understands it, examples are a dime a dozen.
So you concede the point then. The age of humans comprehending things is not coming to an end. And therein lies the rub. When it comes to intellectual labour:
Understanding == Value
If a mathematician produces something incomprehensible then it has no value. It's meaningless. Indistinguishable from random noise.
An AI which produces incomprehensible text is producing no value. We didn't need to spend trillions of dollars on LLMs to figure that out. Markov chains can do that job perfectly well.
Again, do you believe that there are documents, of any value, that humans don't understand?
There is no value in an undeciphered document until understanding is achieved, just as a lode of gold ore in some asteroid orbiting a distant star has no value until we can fly there and extract it.
If an LLM can help us understanding something then it was not incomprehensible, by definition.
Nothing is in principle impossible to understand. It just takes too long, is inconvenient and/or economically unviable.
I don’t find it hard at all to imagine that an AI comes up with a fundamental proof applicable to physics which results in some widget we can now produce that would otherwise not have been produced yet nobody takes the time to fully comprehend why it works. Somebody could, in principle, devote their lives to it and possibly get it, but for what purpose?
There are plenty of artifacts that, if not impossible for humans to understand, then at least no human has ever completely understood. To start with, the universe as a whole. Despite that, we are able to choose legible pieces of it to model and perform useful actions from.
This will shift your argument--that doesn't count! etc., to the point where it's by construction unsatisfiable and vacuous. And it doesn't matter: an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
One person need not completely understand something for it to have value; it's sufficient for there to exist a shared understanding.
an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
No, that is the entire point. If it is an algorithm which accomplishes something useful, then it is intelligible as such. That which is incomprehensible cannot be understood even in part, so it provides no value as a bit of knowledge (unless you're looking for a strong random number source, I suppose).
> how many people do you think understand the Linux kernel in full?
"Not many people understand some things fully" is so massively different from "the human mind is incapable of understanding some things that AI will understand for us"
Yeah. We're only starting the journey of building tools better at thinking than the human mind, so expecting me to have examples of things it produces is a little hard, don't you think?
The best I can do is things that are incomprehensible to nearly everyone, but still provide value. There's a small leap of imagination to consider an author that understands it and can show others how to leverage results without understanding be mechanical rather than biological.
I know right? And there’s only a market for maybe 5 computers in the whole world.
Paul Krugman (1998): predicted the internet’s economic impact would be no greater than the fax machine’s.
The 1876 Western Union memo dismissing the telephone as having too many shortcomings, and the banker telling Horace Rackham not to invest in Ford because the automobile was a novelty.
The idea about the goal of mathematics being shared understanding seems to come at a convenient time.
Mathematicians have never been known to communicate their ideas very clearly.
Regardless, even that target llms will likely win - an llm will likely be more efficient at teaching me string theory than a professor in a room with 463 other students.
I don't think "shared knowledge" means "shared knowledge between mathematicians and lay persons" (there is no much point in that, the same way a smartphone technician knows how a smartphone works deep down to the details but there is no big interest for society to have every lay persons being informed about it). I think it means "shared knowledge between mathematicians".
And at this level, while there are anecdotical exceptions, mathematicians have always been pretty decent (with their conferences, workshops, paper publications, international collaborations, ...).
So, it does not mean "teaching the subject", it means "creating a human network of people that share the understanding". LLM can be useful at telling a human, but you still need a human. The point of Tao is not that LLM is not good at providing explanations, it is that "providing explanations" is not the contribution to science, "the human network" is. It's like saying "LLM are great cook, they generate tons of food in space", but the point of having cooks is so that people can eat food and not die. Having LLM generating mathematical proofs is as useless as having LLM generating food that no one can access: the point was never to "generate proofs" or "generate food", the point was "creating a shared human understanding" or "eating the food so human can survive".
I get that there is a cultural benefit to keeping it alive. Just like we ideally want the languages represented at the universities.
But keeping humans in the loop does not appear to be necessary in order to call it science, and certainly not in order to have progress or dessiminate that progress.
I don't have a problem with people doing math. As long that we don't idiomatically hold on to that way of doing things.
I do, however, find it hard to belive that individual humans will play a big role from here and forward, in any scientific desciplines.
The goal is indeed to have progress or disseminate that progress.
The point of Tao is that people see LLM providing "proofs" and are concluding that this is all that is needed to "have progress or disseminate that progress". That is the same mistake of thinking that "generating food" is all that is needed to "have people not dying of hunger".
The hard part of "have progress or disseminate that progress" is the human network. A fundamental point of this human network is that it generate trust, accountability and reliability. Generating "useful new theorem" is useless unless the society also built the trust around the theorem to distinguish it from a fake theorem.
Maybe in the future, we will have AI doing some part of it, but this is a totally different AI animal than the one we are able to have now, and people who think the current AI that we see now is able to do that have no understanding how it works. This is demonstrated by the facts in math: current AI is able to provide math proofs, and yet, a lot of human work is still needed to get progress out of current AI.
I would not bet that individual humans will still play a role as big as today in the future. Maybe AI will be different in the future, but the reality is that we don't have any indication if this is even possible.
Why is there a cultural benefit to keeping it alive, or to having universities? It seems like education is toil that could be automated for those that don't have fun with it.
The entire point of writing proofs is for advancing human understanding. A giant dump of symbols that passes the lean compiler is meaningless besides human beings understanding it.
> The entire point of writing proofs is for advancing human understanding.
Proofs also enable AIs to direct search and generate knowledge. Verifiability is immensely useful for keeping AI grounded.
One might imagine AI generating enormous numbers of hypotheses and then trying to prove or disprove them, and then mine that data for new abstractions and heuristics.
But what does it mean? The theorems are just symbols in lean. The conjectures humans chose are carefully selected to be the questions that are interesting and relevant to our intuition about the real world.
Math often doesn't have applications for hundreds of years and that application is only possible because people deeply understand it and how it applies to the real world.
Generating an endless list of true statements doesn't really do anything, those things are already true regardless of whether someone has written a lean program to model them.
Sometimes the purpose of the proof is simply to demonstrate that some construct is a safe assumption for other more interesting work-- and could still serve that purpose even if it was entirely a black box.
Is it the AI's fault we can't understand? If the GUT is beyond human comprehension does it matter less? We don't apply this reasoning to other animals or even to less capable humans. Besides, the robots may want to ponder maths for their pleasure.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
That sentiment makes me cringe. If you understand how LLMs work, you'd know it'll never be possible without a fundamental change in how these work.
We're also supposed to be reaching that point, somehow, without the LLMs ever being intelligent (in the dictionary definition sense, not the "high reasoning model" marketing sense).
Based on observations, the ones who are fooled by the supposed emergent properties, are just that, fools. Any sufficiently unintelligent agent will perceive transformer based LLM text predictors as possessing high intelligence.
First of all, the argument isn't that LLMs (with I assume some automation) cannot be used in searching a problem space. I'm assuming this is what you're referring to, in terms of contributions?
That's the part where LLMs are used as tools. Which there are plenty of places where they are useful.
Also, do you know what turning completeness is? Why are you bringing that up here?
The crowd that AI psychosis has brought to HN is interesting. But not in the "I'd love to learn more" kind
The thing about discussing or explaining constraints, is that it rarely is useful or productive if the other side does not accept (or understand) the reality of them.
Turing completeness is not exactly a high bar, and it's genuinely confusing as to why you bring it up. Your C++ precompiler is exactly as intelligent as whatever is your favorite agentic workflow with whatever harness you're referring to. Both might be Turing complete. Neither are intelligent. But one of them seems to be fooling you to think otherwise.
There have been many times that the C++ precompiler produced some output I couldn't understand. I might even at some point thought it was trying to tell me something profound I was too dumb to comprehend. Turns out it was just a missing semicolon.
> Again, you are the one who arrogantly say they llms can not be intelligent without supplying any argument for such.
Not really. You've provided the arguments yourself, just now. But, you don't understand them. Which, brings me back to the initial remark, as to why this engagement is bound to be unproductive. I'm off to bed. Have a good one.
I'll bite, I think Turing completeness is relevant in that it has to be used to informally argue via the Church-Turing thesis that biological intelligence cannot exceed the power or expressivity of formal neural network models. It thus is a good counterargument to stochastic parrot dismissals of neural net based AI such as LLMs, which really are still black boxes. The issue is not simply "So you think that humans are as powerful as LaTeX (or vice versa), ha ha how absurd!"
> That's something AI companies would really want you to believe.
Why would I care what they want me to believe?
Intuitively it would make sense that you can put math ability on a chart with a value for “general public” “smart high schooler” “smart undergrad” “smart PhD/ professional”. And you could place frontier AI somewhere on that chart over time from GPT 2 to now and see the trend.
Then you’d have to consider that either you believe there is a fundamental limit that is below peak human mathematician level or there’s not.
They can say random stuff with the goal of increasing their shareholder value. Things they spit out do not have to be true. It is not easy to verify things they say, therefore, everything they say should be taken with a huge grain of salt.
Consider Enron and Amazon at the turn of the millennium. They were both telling you what the future would look like. The right action would’ve been to just ignore what they are saying and try and get data and reason about the world. It didn’t really matter that both Bezos and Jeff Skilling wanted you to believe various things - one was right and one was a scammer.
So that’s what I’m doing here. For what it’s worth I find a lot of the AI people’s worldview very consistent. They believed AI would be the most important technology of our life times and committed their work to it. Some of these same people are total liars so yeah I won’t really hang onto their every word.
that does not make it not true, nor does it make those companies or their products not dangerous. I like looking at videos of animals that tear other animals apart and eat them; lion cubs are super cute; but that does not mean I want to be thrown into a cage with a model of a lion that has not been programmed to be disinterested when it is sated. AIs appear never sated; humans using or making AI wanting money, even less so. I suspect the AIs will understand the cost long before the humans will, not that anyone making money would care.
Whenever someone tell me they can be reduced to an LLM, I believe them. I see you and I believe you that you are qualitatively the same as a language model.
It also creates and updates models, uses those models to make predictions, guides the stochastic generation by comparing the output to those models and reworks them in real time.
It also creates and updates models, uses those models to make predictions, and guides the stochastic generation by comparing the output to those models and reworks them in real time.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
I agree.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
I don't know if I see this being true for quite a while, if ever.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
> I agree.
There's an infinite space of possible statements and proofs. The only thing that makes certain proofs significant is that human mathematicians consider them significant; if AI came up with a proof of some statement that no humans could understand then no humans would bother investing further resources in building upon it, for the same reason we don't waste computational resources iterating over the infinite space of true statements in first-order logic.
In limited experimentation: AI will certainly make statements that are extremely intricate and hard to understand, in part because they're overcomplicated and in part because they use a bunch of unnecessary terminology.
This is not to say that a human couldn't understand a streamlined version or that the AI would not be better if it made more streamlined statements to begin with.
(I am not saying that everything mathematical that an AI produces is in any sense trivial.)
It’s possible, but there’s a difference between vastness and difficulty.
Humans can’t compete with AIs on vastness of material they are familiar with, or the depth of effort they are willing and able to throw at a problem.
But scale isn’t the only aspect of difficult scientific endeavours. There’s also theory. And advancements sometimes come through hard graft of knotting together many things. And sometimes they come through the revelation of a deeper truth, or a new framework, a fundamental insight.
AI might help us reach the next level. But that doesn’t mean we won’t understand anything. It could be we have periods of vast intricacy we cannot follow, punctuated by profound elegance we (or at least experts) relatively easily can. And then the scaffolding we needed to get there falls away.
I don't think that's true. Human intelligence is limited, and our brains are inefficient machines.
The tools we built to replace muscles have mostly obsoleted raw strength for tasks like excavating earth.
There's no reason to think we can't do the same for brains. And then we'll never need to think for a living again. Some people may want to do it as a commercially insignificant hobby, of course, the way people lift and compete in strongman competitions today.
We'll have AI taking care of our needs, the way a good mother takes care of their children.
A good mother doesn't raise children to be dependent upon her for all their needs.
For this to actually work in a way that benefits our species, humans will need to become something else/next through their interaction with the technology.
You could be right, but you're making a lot of assumptions about how complexity, scientific understanding, and explanations scale. One of the features of a good scientific discovery is that it often simplifies and compresses things that were previously a bunch of scattered facts. Also, as AI systems improve they'll get better not only at making scientific discoveries, but also at producing understandable explanations.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
Ask yourself: is that really the world you want to live in? It's a world where people, all people, are sidelined.
I think the happy ending of that path is something like Idiocracy. And the more likely ending is something like "automated capitalist economy without the people, because the people couldn't compete."
>produce proofs far more intricate than humans can understand
Math is not magic, a proof is just a series of applications of a set of rules on some axioms. A mathematician could understand any proof given enough time to study it; the only way for AI to make proofs that a human couldn't understand is by making really, really long proofs.
It's going to produce proofs far more intricate than humans can understand
The thing is, some number of advanced proofs start out "too intricate for most mathematicians to understand" but many of these get rephrase and reframed until they're accessible to undergraduates. Hopefully, AI math can be guided to do that sort of reframing to increase the level of accessible math as well as extend the border of math.
Why would you want something you don't comprehend? How can you be sure it empowers you?
I think perfect rationality doesn't exist, because it is rational to reject something that you don't understand. So rationality of a given physical system will always be bounded.
Let’s not jump the gun. Where they are right now they can somewhat match our abilities. We haven’t even gotten to the point where they can self improve.
This means some AI proofs might be impossible to comprehend by humans, right?
I guess AI still lacks human intuition for many concepts, but AI might beat humans in narrow areas, such as discrete math and combinatorics.
Mathematics is typically concerned with "proofs" [1], which similarly to code, often allow for strict validation. Thanks to reinforcement learning techniques, it is now possible to train LLMs to perform very well on code generation, and mathematical proof generation.
Law and medicine are fundamentally harder fields to obtain decent training data for, and LLM results are therefore expected to be less powerful. Also, making mistakes in these fields is costly, but perhaps you were alluding to that already.
Mathematician here. There is a lot of recent work on the Lean project -- when a proof can be translated into Lean code, then it can be strictly and formally validated.
But otherwise, mathematical proofs are read and written by humans, and at the end of the day the relevant standard of proof is what other mathematicians will accept.
Occasionally, mathematicians don't agree. For a prominent example, you can read about Shinichi Mochizuki's claimed proof of the so-called ABC Conjecture:
> Law and medicine are fundamentally harder fields to obtain decent training data for
I think this bubble has given a lot of people software brain and are trying to apply it to fields it is wholly inappropriate for, though. Law is about argumentation and rhetoric. It is about providing a persuasive argument. This is how it is taught. The actual legal code is a way to formalize parts of it, but increasingly I see people angrily insisting that the only thing that matters is the text.
As you might imagine, I find textualism a load of applesauce, but I don’t think the vast majority of people making this argument even understand textualism as jurisprudence. It seems to stem from Crypto bros and the whole “code is law” argument which is just codswallop.
A neighbor of mine whose husband is a lawyer said it's already part of his regular workflows. OpenAI also already have HIPAA compliant offerings targeting healthcare uses, etc. Of course they already do these things.
Well also you can't just throw AI slop as doctors or lawyers advice and fail multiple times until you find the right answer. With code and math you can have failures 1000 times for every success and still get rewarded.
They can rely on compilers, solvers, theorem provers to validate the generated softwares and maths. That’s what makes it possible to iterate quickly in a loop and self correct. You cannot do that in soft industries like legal and medicine
That is not the point I was making. I am not talking about validating software or maths. It can generate stuff that is valid, but bad and incomprehensible.
What I’m saying is that AI labs are talking so much about software and maths because we already have tools that can say « it’s all good ». That makes it possible and worth it for them to spend 1 week of compute on a problem until the validator passes, then publish marketing pieces. You cannot do the same in medicine or laws (modulo some niche areas)
the article cites the ability to keep a ton of details in working memory as an advantage, I'm not so sure that it is - perhaps it is quality over quantity; a compression of everything known into a smaller set of interlocking patterns should provide a more useful generalization (if correct, of course). The human's ability to perceive 7-10 concepts might just be the tips of the icebergs composed of a gazillion micro-concepts, i.e. our working memory is enormous, but not consciously accessible
also, the older neural capacity estimates are way wrong [1]: dendrites also compute
I love the term "Out-Remembering"! I have been trying to find a way to communicate that "intelligence", "creativity" and so on might be misleading about the true nature of LLMs, and they would better be described as genious "reproducers" as in, they are very capable at reproducing what they have already seen - and they are a bit less capable, but for many use cases still good enough, at reproducing a mix of concepts seen previously.
This also nudges into how to use it best: By knowing where the "piles" of if training data are (i.e. when it comes to a CLI in rust, I just briefly describe the use cases, and I have a very high confidence the code will work exactly as intended by me since there will be a multitude of examples in the training data), one can predict where the LLM is likely to go wrong an prompt/guard accordingly. This skill grows with domain expertise, and is one of the many reasons LLMs can be (and probably should be) used to outsource busy work, but never understanding and learning. ("never" is a not meant literaly of course - I for one am glad that I do not have to wrap my head around CSS and other frontend topics and go straight to the topics that interest me most)
"Out-Remembering" captures that perfectly, I feel. Also goes nice along with "asking it leading questions" as we know how to do in real live; if you want a person (LLM) to confess (produce output tokens) something, sometimes you do that by leading the interogation (chat, context) to where you think the truth lies.
It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.
>Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.
"AI" does not exist, it knows nothing and has no experience of doing or seeing anything to "remember", a data bank filled with the jumbeled accounts of things that humans have said or written about what they may have made up or may have done or may have heard someone else say, and changing the label to "memory" is only that , and can only have the predictable outcomes.
Seeing what AI has been doing to maths conjectures recently, I decided to give theoretical physics a go.
I threw some random "crank" ideas into GPT 5.6 Ultra and let it consume most of my weekly Pro subscription quota.
I'm actually pretty impressed with what can be achieved! If your simply ask it to use adversarial agents and include honest self-ratings, it can produce output competitive with a smart but not exceptional PhD. Not imaginative, but the effort that it sinks into even the smallest matters is just amazing to watch.
Did I get anywhere by applying AI to my toy models? Probably not! I ran out of quota before I could tackle anything really interesting.
But... GPT did seem to discover something genuinely new by essentially brute-force effort of combing through hundreds of papers on ArXiV: a new "constant" extracted from a formula over quark masses that seems to evaluate to exactly 3/4 at the one-loop level and then with various corrections at two-loop and higher. Link: https://chatgpt.com/s/t_6a80f2c794388191971350733a6f4378
This is "novel" in the sense that it found a vaguely similar formula in a paper where the authors simply missed the opportunity to extract a simple ratio from quark masses, combined with updated constants the AI found in in later papers that allowed it to guess that this might be exactly 3/4.
This is the super-power of mechanised attention!
We've produced a truly ludicrous amount of scientific and mathematical output, far past the ability of any single human mind to be even vaguely aware of all of it.
Now we can scrape together all of the jigsaw pieces we have made and see what fits together.
I wouldn't be surprised if someone, perhaps even a hobbyist comes up with a viable theory-of-everything in the next year or two simply by feeding in some clever starting point and then turning the handle on the machine until a complete and consistent theory pops out the other side.
which is why the programming language for LLMs hasn't been invented yet. I cannot read someone else's code that is filled with single letter very real names, but I can write it but as a human with human working memory, I can only hold so many variables in my head to reason about an LM is a much greater working memory unless does not need the same abstractions as a human does if I'm no longer riding the code and I don't need to read it, then not only is goto's considered harmful is holding us back, but bring on the big ball of mud spaghetti code that outperforms human written code, that no human has a chance of maintaining or reasoning about.
The problem with this is there is going to be a bug that needs fixing that the LM can't fix and if the humans can't understand the bug that we're just gonna have to live with the bug! But let's not talk about that.
There is a certain amount of slack that we have. Things that are obvious consequences of what we've already discovered but that we haven't taken advantage of yet. AI is reaping that slack. It isn't adding brand new ideas at the moment. We will run out of this slack pretty quickly.
"OK, it might be much better at math than me, but it's not smarter, it just remembers more".
Heh. Every day, a new type of cope. It's like, coping as hard as possible.
If this was an ML researcher trying to come up with a way to improve performance then you could say it wasn't cope but rather practical observation to serve a goal.
But it's just cope.
Also, AI is going to continue to get smarter. A lot smarter. There are already systems in R&D that will continue increasing efficiency and performance of hardware by more orders of magnitude.
Well, some folks are going to keep trying to exclude it, but the reality is we are almost at the point even those folks would concede that they are not smarter than LLM. I suspect we will see then see more discourse about consciousness, morality, agency, emotions as being the most important part of intelligence.
For instance, Opus 5 yesterday critiqued my resume and mentioned a ridiculous little detail-- my phone number area code didn't match the state in which I currently work (I know, I should anonymize but I couldn't be bothered). It failed to notice that I both worked in (previously) and studied in the state of my area code. This was just two pages of text, set to highest effort.
Why would you want to replace your lawyer with a set of tensors that does not actually think and makes mistakes like this? Lawyers tend to get hired in high stakes situations. Why wouldn't you instead say that this would be a great tool for lawyers to use judiciously in researching precedents, etc?
I don't understand what people are doing with models that makes them assign agency or intelligence to them. When I manage to forget the financial fuckery of the AI buildout and its implications, when I manage to forget scaremongering by loathsome CEOs, I still have the same fascination and excitement at the idea of LLMs as I did when I was playing with the GPT API prior to the release of ChatGPT.
LLMs are, to me, truly amazing tech. It's so fascinating to me that they now DO have emergent properties that look at face value like reasoning and intelligence. But every day that I work with them, I am repeatedly clobbered over the head with the fact that they do NOT reason and are NOT intelligent.
Why can't we be fascinated by emergent properties of intelligence without immediately jumping 10 steps into the future and, like a limit in calculus, assume that "this is it-- we're on the cusp of AGI"? To me, the fact that LLMs can combine existing ideas that people hadn't thought of combining in solving a novel problem is extremely cool. But my first thought is-- this is an amazing new tool for mathematicians and researchers. Instead, most everyone seems to jump the gun to the "humans are obsolete next year" conclusion.
> Why can't we be fascinated by emergent properties of intelligence without immediately jumping 10 steps into the future
Because, when computers first came out, and filled entire rooms, people predicted they'd eventually shrink down to fit the crevice of your bum and everyone on the planet would have them
When the first cars came out, people predicted that within 10 years, city streets would no longer be drowning in horse dung
Do people who rant and rail against every new technology in its infancy, actively choose to forget history, or simply didn't learn about the many similar instances in the past?
Aka - it’s a stochastic parrot with a good memory, for anyone still struggling to understand this. It should be obvious, imo, but some people seem to have trouble with the concept.
I don’t understand why this is such big news. OpenAI has essentially made a bunch of marketing copy by gussying up an algorithm being given a near unlimited budget to stochastically permute through its lossy memory.
I think the real scandal is that we are almost 3-4 years into this (I think the release of GPT 3.5 is a good marker of when this public frenzy started) and all we’ve seen is OpenAI and the other major AI frontier companies constantly retracting their preposterous claims every time. We appear to have reach a local maxima in that it has some value in places that tend to be a little easier to scope and limit (computer programming, mathematical proofs). So, given the actual useful economic value this has provided, does this justify the investments? I think we are approaching 1 trillion in CapEx for AI [0]. For context, I believe the annual GDP of Norway is $600 billion.
It's crazy, you see someone spin up 50 instances of chatGPT or gemini or whatever to tackle a math problem, it succeeds by brute forcing through a ton of existing theories to find one that extends the problem, and the takeaway is that this technology is magic and going to solve all of our problems.
Whereas I see that and say - if we properly funded the sciences we could have had a bunch of grad students tackling that problem and found this application 20-30 years ago. Sure it's 'nice' that LLMs can fill in for people in brute force work like that but people are perfectly capable of doing that work and if we focused on properly staffing our research institutions we would achieve a lot more a lot faster. Instead this is obviously going to be used to replace staff and further reduce headcounts.
That’s how HN works, I had that multiple times over the years with my own submissions. Sometimes it gets picked up quickly, sometimes not. A post can also down rank very, very fast. It depends a lot on the level of engagement and the type of engagement
100%. Context is big for AI, but it's nothing compared to everything a human can learn. If you efficiently represent everything in context, it may be many papers, but if AI is actively working through proofs, it will quickly fill up. They're no denying AI is making strides, but pinning it to memory is an oversimplification.
I don't have to open the article to be confident it's not worth reading. Anybody knowledgeable in the field should be familiar with AI writing tells and the message they send. It only takes a few seconds thought to transform the title into something like "AI beats mathematicians by out-remembering, not out-thinking." Regardless of whether the article is slop or not, I expect any competent writer to avoid slop phrasing in their titles. To do otherwise signals laziness.
Most mathematicians are quite simple creatures. I can do basic math, some derivations, but my bright days of solving differential equations are far gone!
Computers are simply better at math now, like in chess or go!
I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us. I think of all the times in my software career when I did something that others considered very high performance, it either came down to either having more energy than others at tackling a problem they thought was more trouble than it was worth, or just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
I don't think I've had a truly original idea in my life. Combine A + B, when it's rare for people to know A and B at the same time. So from that perspective, what LLMs are doing is basically the same thing. Sometimes I am faster than the LLM because my context might be better organized, but it typically needs just a hint from me to steer itself correctly. It claims something is a memory leak, but smelling a rat, I suggest it to double check the garbage collection statistics too, at which point it's clear it's no leak, but a tuning error, at which point the LLM is better at tuning than me, because it has more energy than I do.
Maybe there's true brilliance out there, when something doesn't come out of combining data and building hypothesis until you get really lucky. My experience is not comprehensive. But I look around me, and it sure seems I've not been lucky enough to see it. Even the shiniest people I've worked with, which most of the audience here would recognize, have never shown me that they can go past this.
People have told me I was smart since I was a kid, but I can't remember for shit. I had a thought when I was fairly young that the only reason I was (maybe, sometimes) outperforming others intellectually is that I was habitually compensating for my poor memory by working things out on the fly, while others could rely more on rote memorization. Anyway, takes all kinds I guess!
Somewhat tangential, but this reminds me of when I took a certain Microbiology midterm in college. I hadn't put much effort into that class leading up to it, as I had some tough CS classes that consumed my focus that semester (also was admittedly not a great student in general). So when I finally went to take the midterm, after cramming like mad before the exam, I quickly realized I basically didn't know the answer to most of the questions, even what I crammed was only a small portion of the material.
The exam was huge, at least over 10 pages, and even when we technically ran out of time, the professor was kind enough to move the remaining exam takers to the neighboring lecture hall to continue taking it. I recall I spent a total of 2 hours on that exam.
Now mind you it was mostly short answer or multiple choice questions. The multiple choice questions were pretty sharp too, lots of traps and false but sounds right answers mixed in. But if it had been purely essay questions, I would have been screwed.
However with such a huge corpus of information in front of me, I ended up basically learning all the material on the spot. I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
In the end I got B+ in the exam (which was curved to an A), and walked out understanding the material better than I did walking in.
Reflecting in the experience years later, I've wondered if a hypothetical LLM which was ignorant of microbiology could do the same thing if fed that exam. In some respects the traps they placed in the multiple choice questions actually were what helped me refine my understanding the most. Made me appreciate information theory more.
> I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
This strategy saved my ass on so many tests from high school through college that I did not study properly for.
I think it would certainly be possible to write a test such that this strategy doesn’t work at all, but lucky for me no one did in any meaningful way.
There was a couple famous papers from a few years ago asking why computer science students, sorted by ability, seemed to form two clusters.
In one of the studies they got incoming CS students to fill out a multiple choice exam on programming, before the class had even started. Then they looked for hints in their exam results to see if they predicted the end of semester grade.
Turned out following this strategy was an incredibly strong predictor of success in computer science. Some students somehow got 0 (or close to it) in the multiple choice test. But the researchers looked at their answers, and could see that the student was forming hypotheses about the questions. Even if they got all the answers wrong, they answered all of them with a consistent idea of what the computer programs might have meant.
Weak students just answered each question individually based on vibe, without using what they learned (or figured out) from other questions.
Same experience here, but it only works in some fields like physics and maths. it totally fails in fields that are "made up" with no rhyme or reason.
Same. I have bad "factual" memory, but very good "conceptual" memory. I might not remember exactly what someone told me, but I probably gathered a keyword, and a feeling and idea of what they told me. Almost like it takes what they said then compresses it into my mind in a nice little multi-key lookup table.
Oh man, I have that too. I am always told by my best friend "well, you go it right, but entirely wrong words" -- or something to the effect I got the feeling, notion, or concept right, but the wrong words.
This has affected me a lot though, I know what is going on in songs, but can't ever remember the right words, but still some how end up sining stuff that means the same thing :/
Yes. "Weak detail recall but strong fluid reasoning, conceptual/gist memory, and adaptive problem-solving" is how GPT sums up what I feel. And "high fluid intelligence" is a label it gave that I won't be using anytime soon... But I like the sound of it.
You might benefit from (proper, not the multiple choice kind) aptitude testing.
What are the benefits? What would/could I do with the results?
Save money on future dates by causing the woman to leave before she even orders dinner after you pull out your Mensa© Membership card.
For me it's one step removed from that: I'll remember exactly what I was told or what I read, but not at all who told me or where I read it. My brain does not care about sources.
Thanks, I feel seen.
In my, admittedly extremely small sample set, concept-focused memory is common for people with any degree or kind of dyslexia. I used to struggle with understanding how people couldn't understand things, now that I'm more mature it makes sense how people struggled with me not being able to spell things :).
Yep, terrible speller -- bad at pronunciation too (by the way I just autocorrected that word) -- and I often wonder how people, who are much more intelligent than me on paper, can't figure out random tasks that just seem obvious.
I do not have dyslexia. I’ve been reading since I was a little kid. I don’t easily recall words or things that I’ve heard. But I’ve got a very good visual memory, especially remembering patterns. And I can easily visualize things too, even the somewhat abstract concepts.
So if I put a visual bend on something I can quickly understand it and remember it, but that bend has to be mind (even when helped by an image on a book). If it’s someone else’s, I’ll be very hard pressed to remember it (but I can easily remember where it was).
So at school, when I had to cram, I fill my head with images and things like that. And with the help of the exams (written) and memory of the classes, I worked out the answers.
Thanks for sharing, I am absolutely fascinated with the differences in people's internal experiences.
For me, I have the bad "factual" memory and good "conceptual" memory but unlike you, I also have aphantasia: a completely "dark" mind with no visuals.
I've always used the metaphor that my brain has really good "compression". Once I learn something it gets compressed into my worldview. I don't know how or why I know it but I can apply it. If academic achievement is any benchmark, I can apply it with above average aptitude.
We sort kids into the smart and non-smart labels when they are young, and those labels tend to stick, even when what we're measuring isn't raw intelligence but just variance in childhood development that wash out in the long term. I don't think either group is well served by this.
A smart kid can be anything from not eating rocks anymore to multiplying numbers at an early age.
Most of us turn out like the rescued exotic bird which turns out to be a seagull covered in curry.
Sometimes I wish there was a Hacker News comment hall of fame.
Never been easier to make one.
https://news.ycombinator.com/highlights
but you can also favorite comments for your own hall of fame, eg https://news.ycombinator.com/favorites?id=andybak&comments=t
No we don't. Public schools are incentivized not to help high performers. It actually hurts their metrics and budget. The widespread gifted/talented experience in public school ended in the late 90s.
The actual intelligence segregation is self-selected. PMC parents live together in the same neighborhoods. Whether you believe in genetic or social factors, the smart kids come from here.
In the 80s and 90s there was an educational culture that believed in finding overlooked geniuses regardless of where they are from. That narrative is completely dead. All institutions expect kids to come prepared with a resume from their youth, and have few avenues for identifying talent that doesn't meet their mold.
I don't know where you're getting that from. The sorting of students into "gifted" and "non-gifted" categories is very common. You don't need to look further than your local school district for proof.
> don't need to look further than your local school district for proof.
Yeah mine didn't have it. And the teachers fought every attempt to do so because it hurt their metrics (who gets stuck with the lower class average).
Are you by chance part of the PMC and your local school district is of similar demographics?
Seattle Public Schools currently still has a “Highly Capable Cohort” program, though it is phasing it out.
Seattle public schools is a district of and for PMC. I'm sure you find all kinds of advanced programs in SCV too. There is an incentive to appease local parents by providing services they want to enroll their kids in.
In areas where parents aren't demanding advanced classes, there is no program for the government to identify and separate top performers from the general student population.
I can relate to this, but always explained it away in the opposite direction: I never trained my memory very much because I was able to work things out on the fly. Who knows, maybe there isn't much of a link between the two at all.
> I never trained my memory very much because I was able to work things out on the fly
I actually always thought it was the opposite for me – I had to figure out how to work things out from scratch because I could never remember anything.
I’m very similar. Poor memory, good on-the-fly pattern recognition. I would say reasoning but it’s not really reasoning, it’s just identifying patterns plus some quick hacky heuristics reasoning. I’ve always performed well on tests as a result, but in the real world I need to use A LOT of tools to do my job. I write everything down, notes for everything.
I remember for math tests in high school/college it was easier for me to derive formulas than to remember it.
I used to think I wasn’t very good at math, because I struggled to route memorize formulas and the like. Then I had a calculus professor in class that took a bunch of time explaining how to derive the formulas we were using and it unlocked everything for me. I’m much better at deriving from core principles than trying to remember a bunch of formulas and algorithms for solving problems.
Same. I don't remember most things including my childhood. I have to relearn everything all the time just in a few months time. Maths, code, etc if I don't use it in 1 or more month, I have to relearn it though its always easier to relearn it. Comparing myself to others its extreme. What I find is the process of constantly relearning instead of relying on memories (which are in a way similar to assumptions) can lead to different ideas (often unique) than other people because everything even basic concepts is a open question. This however is tortuous in job interviews when people ask about problems solved in previous work or about a concept so there are trade offs.
I'm in a similar boat. I end up re-learning the same things over and over again.
I found a solution recently. "Critical mass." If I just don't see something frequently enough, my brain keeps deleting it.
So, if there's a fact you have trouble remembering, try memorizing it 100 times in one day. That's "inefficient", as far as memory goes, but it'll do the trick! And I like solving problems once and then moving on.
(Anki, conversely, is designed for efficiency, which means it's designed to be as painful and unrewarding as possible, but that's a rant for another time... suffice it to say if you design your software so that I constantly want to stop using it, your efficiency drops to zero!)
I find my memory is very selective. I wanted to get in touch with a coworker from five years ago: I had to look up his name on linkedin. My wife texted me "Hotlips died" and with no effort I texted back, "Oh no, Loretta Swit or Sally Kellerman?" Mind you, I haven't watched MASH the TV show in forever, and I haven't seen MASH the movie since my parents took me when it first came out. I have zero reason to remember Loretta Swit, let alone Sally Kellerman, but my brain hangs onto that like it matters; but a guy I worked with for two years five years ago, that my brain will drop like it's radioactive.
Isnt that kind of like older memories get a strong imprint -- as you get older there are so many memories in the database that the imprint isn't as powerful as earlier ones. I think it works kind of as a function of kind of sensory component of memory implying that new memories aren't new to you therefore less powerful (obviously high variance and person dependent).
I think the fact that ‘just work it out on the fly’ is a viable response to ‘I cannot remember things’ is what makes the intelligence right?
I relate to this; there are some things I remember very well, but in many areas I think my talent for improvisation stems from a ~mid long-term memory.
You maybe work things out on the fly by matching against abstract patterns. Could this be an indication not of lack of memory but of specialisation in abstract memory?
Memory isn’t only being able to recall things on demand.
Agree. Plus the dimension of synthesizing new ideas from existing knowledge. Different skill set from rote memorization.
Have done a lot of “parallel invention” where I invent something genuinely novel & new-to-me, only to later discover it was first developed eighty years ago.
The part that’s really disheartening is there’s so much knowledge out there, I wonder if this is actually efficient - maybe you can’t reasonably search the state of the art across all of human knowledge and so you have to invent it again. But that’s such a sad outcome.
That's exactly what "AI" is for, though. Suddenly, you can search across the whole of human knowledge and invention.
In line with the other comment on this post: I also am classified as “brilliant” by people and IQ tests. My country gave me an award for being a great inventor. But my memory is worse than anyone I know.
I can’t remember the lyrics to any songs, or the lines from any movies, or what I’m doing this week. I don’t think it’s a strong hypothesis.
My wife has a nearly eidetic memory, so marrying her was a game changer for counteracting mine :)
The things you list not remembering may just be a different focus. I was always amazed by how many movie details other people remembered until I realised they rewatched those movies many times and I never rewatched anything.
Similarly, being engrossed in tasks today (or rabidly learning/reading, as many HN readers do, as long as it's not the link being discussed) might make you not focus much on next week's calendar.
How do you complement each other in memory? I guess I can imagine, but it would be cool to know.
“Hey any idea where my Kathmandu coat is?” “Yeah you left it under a box in your office 2 months ago, the day we were in a hurry because we wanted to go see Joe”
—
“Hey do you remember where I put my glasses?” “2 sets at the office and your other set is upstairs next to the toothbrushes”
—
At the grocery store: “Do we have apples in the fridge?” “Yeah, 4 left”
> out-remembering people around us.
Doing an angular -> react refactor with Opus 5 at work.
Amazed at how good it is but also amazed at how dumb it is after I told it I want a very basic and specific code review:do the file names match the contents? is one functionality/concern split in multiple files? do the variable names match what they do? does a method read nicely?
Anyway, I was very surprised at how bad it can be at this.
I suspect it's because the LLM simply does not focus well enough on a single method.
My gut modelling is that the critical files are quite small, some 4k LOC and the LLM tries to read them all in one go and my requirement is surprisingly difficult but the LLM does not realize it, loads all the input into context and simply cannot find a good signal per token/group of tokens.
I guess I can do it because I literally cannot fit 4k LOC in my head and I actually read only tens of lines but I can reeealy dig deep in those lines.
The LLM can remember lots, it has 1M context window, was trained with needle in the haystack problems, but it can severely underestimate how much computation is required with TASTE/universal computation type tasks.
Yes, LLMs are incredible when it comes to delivering well defined outcomes, it's evident they were RL trained very well. But they are incredibly bad at understanding nuance in a text.
It's incredible how agentic coding benchmaxing seems to correlate with writing good poetry but at the same time finding good names for files and variables and wiring beautiful code does not.
Same with getting basic details of who said what in an "adversarial" conversation between 2 humans -- LLMs are incredibly bad at it, with all their "almost discovering new maths" skills
Memory is one aspect of intelligence, but hardly the only one. Making new connections, logical deduction, spatial intelligence and creativity are also aspects.
I have fairly poor memory, but I can solve problems that people with better memory can't.
Human can have periods of heightened mental capacity. Sometimes I looked back at some of my work days or weeks later and amazed how I could have done it. When you're in the flow, have enough sleep, and have long period of non-interrupted time, you can do amazing work.
Since it is so much easier to test for memory than understanding society grew to confuse the two. Only now faced with these other entities are people beginning to question it.
I know people that got to post grad math without understanding a thing but they could remember a lot easily, while many of those that understood but had a harder time remembering every last variation of everything got penalized.
There is an ability to hold ideas in the mind simultaneously. Some can hold a great deal more than others. It can be exercised, but is definitely bound by genetics.
If you want an example of someone at the near peak of human ability, check out Jon Von Neumann.
Then there is an ability to peer deeply into complex problems and somehow find the simplest truths that make sense of it all. Think of Einstein.
Both are incredibly intelligent, but in different ways. I'd say Von Neumann's memory was far greater than Einstein's though.
One can flawlessly ponder anything known to man, and the other could ponder completely original ideas (to an extent)
I like to use the analogy that the human brain (when it comes to intelligence) is like a computer. We have storage, which is just long-term memory. We have RAM, which is your ability to keep track of a mental model of something you’re actively working on, and then there’s the CPU, which is the ability to make logical leaps and connections on that mental model (or maybe storage).
I’ve met different people throughout my career whose intelligence came in 1 specific area. For instance, my friend is extremely good at trivia, he clearly has a lot of storage and can access it easily. I think I’ve only met one person who was excellent in all three areas of intelligence.
Obviously this is a simplification, but it’s how I like to illustrate my ideas on intelligence at parties and first dates.
> I don't think I've had a truly original idea in my life.
There’s no such thing as a truly original idea. It is all just combining A+B!
Sometimes you generalize A into A'.
But often results are incremental and obvious.
But also if you jumped from A to A''''', nobody would understand why or what it relates to.
if someone did have a truly original idea, then everyone would probably think it was terrible anyway and we'd all ridicule them, because we'd have no frame of reference for X + Y if all we know is A and B :)
Cavemen came up with the initial A & B, and we’ve gone from there? (Didn’t think this through much yet!)
Cavemen invented language, language isn't just combination of concepts, it requires defining all the concepts.
> just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
Being smart, at least in the context of the workplace, is about being checked in to whatever you’re doing, and drawing connections across your experiences.
Possible, but not true. Much of intelligence is also creativity. Otherwise we would call Google search intelligent. Strong recall is a different kind of intelligence, and good memory is indeed a positive trait. But it is not what we generally call intelligence on its own.
This sounds very much like the joke about economists finding a $100 bill. In reality there is innovation and new ideas. Humans discover new things and change their behavior. So your theory needs to also accommodate that fact.
People who study intelligence differentiate between fluid intelligence (plastic intelligence) and crystallised intelligence. The former is the ability to learn and form new ideas. The latter is - as you said - memory. Knowing lots of stuff.
They’re positively correlated. Presumably because people who have good plastic intelligence often end up learning more throughout their lifetime. The only way to form knowledge in our minds is by learning it.
There is also an element of how you process information. Some people jump to instantly refute an idea with a counter example. Some people jump to statistical likelihood. There’s elements of trust vs trust but verify.
Some people, through practice and experience are better than others at breaking down foreign information and making sense of it, or jumping to an assumption.
I also think there’s an element of just remembering what my final conclusion was on an opinion, vs all my old opinions and all the work to get there. It’s easier to just store current belief and re-work-it out how I got there later.
Agreed “remembering” stuff is critical in most life’s situations where if you did, you will appeared to be smart. This is probably why spaced repetition has been a key element in learning for the past few decades.
But “remembering” is only the beginning (you have to remember first!), after that, elements like understanding relations, connecting dots and remixing, timing, etc will truly make one shine.
In one way, it’s like the current “LLM + Harness” setup for agents. LLM is how well it remembers, but different harness techniques really matters, at times even a worse model mixed with great harness can outperform great model with bad harness
I suspect any new thing created/discovered/etc is just combining knowledge from existing things with memories being an example. Outremembering while certainly not the only method helps massively with combining knowledge from existing things. I don't see new knowledge coming out of nothing.
The amount of times I’ve invented something genius in software dev to later discover it has a name…
> I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us.
I wonder practically how useful it is. It should be very useful, right? Say you're a hiring manager, and now you can hire thousands of students who have almost eidetic memory across pretty much all the subjects. The students can produce amazing work if you give them specific instructions. They are also very good at mechanical pattern matching. That is, they are the uttermost crammers. They are like the students who don't really get STEM yet they could, through memorizing all the problem-solving patterns, ace any national college entrance examinations, IMOs, PHOs, Putnams, you name it, and all the way to solving hard problems like Sendov Conjecture - they even found out elegant elementary solutions! Mind you, Terrance Tao proved a weaker conclusion when studying the Sendov Conjecture and got to publish his work on The Big Four. That said, the students won't take initiatives, and they sometimes make very silly or even impossible mistakes, to the point that it requires your supervision and review. Would you hire them to do your company's work? I'd assume that answer is a resounding yes.
Could just be that you aren’t an ideas guy/a lateral thinker
I feel like you can make the same analogy with products as well.
It’s a false definition of intelligence - remembering stuff has never been what intelligence is.
Understanding especially in the context of unknowns is what intelligence is.
Not only I agree with you that I don't think I've had an original idea, even when I try to do artistic stuff it seems like the things most praised were the ones where I was trying to do something else and the good part came from failing to do the thing I was trying to do.
That sounds like unintentional. Which still can be original. A happy accident.
why would anybody believe true brilliance? They would only believe something that can be proved from combining existing knowledge.
If a time traveler went back to 1600 and started spouting off about differential equations everyone would think them quite mad.
I think that’s true for genius is in general. I don’t mean the Einstein type but I mean the child prodigies that graduate high school at 10 years old or whatever I mean if you can read something once and remember virtually everything about it that’s a gigantic leg up on everyone else who has to study drill the stuff into your head, etc.
Even in a debate, if somebody just has the ability to remember tons of facts and figures, the other person will seem unintelligent by comparison, even if the other person is correct
I've long held the belief that humans cannot truly create. We just remix and recombine. This is interesting - although I'm not religious, many religious texts call god the "Creator", perhaps because the ability to truly create original thought is above the ability of us humans. To truly create is divine. Maybe the writers of those old texts recognized this as well.
This is a ridiculous view.
Most humans cannot incrementally contribute since they don’t have many traits required to do so - extreme discipline, imagination etc.
We literally live off and benefit from the investments of the few, in relative terms.
> Combine A + B
Really, compress. Compression is a useful proxy for intelligence.
To compress data efficiently, an entity must discover the underlying patterns, structures, and regularities within that data. Identifying these structures is precisely what we define as "understanding" or "intelligence.
Anything we put into writing can be thought of as a string of data. The Kolmogorov complexity of a string of data is the length of the shortest possible computer program that can output that string.
A good compressor doesn't just memorize (yes, there's a reason LLM writing features the "it's not just X, it's Y pattern"), it builds a generalized model to handle unseen, similar data. LLMs are trained by compressing the internet into weights, they use this to write new code.
When an LLM is trained, it compresses ...bytes of human text into a few ...bytes of neural network weights. When you ask it a question, it uses that compressed model to generate (decompress) a coherent response.
Consider the dichotomy:
LLMs display emergent reasoning, coding capabilities, and logic because they were forced to compress human language so tightly.
LLMs often suffer from "hallucinations" and struggle with novel logic puzzles because compression encourages mimicking statistical patterns rather than building a true causal model of the physical world.
An algorithm can achieve massive compression if it utilizes a highly complex, slow, or massive decompression engine. If a system takes 10 000 years to decompress a file, or requires an infinite amount of computing power (like Kolmogorov complexity), it cannot function as a practical intelligence. Real intelligence must operate within tight time and energy constraints.
> Maybe there's true brilliance out there
Brilliance is a trope. pg's 2009 "The Anatomy of Determination" deals with it: https://www.paulgraham.com/determination.html
In it, he argues that determination is the single most important predictor of success, surpassing even raw intelligence.
While popular culture loves the narrative of the "brilliant genius," Graham's experience backing hundreds of startups taught him that a highly intelligent person without drive quickly becomes ineffectual. Conversely, a profoundly determined individual can find pathways to success even in ordinary or unglamorous industries. Over time, determination actually starts to look like talent.
I'm going to commit a faux pas (a breach of HN etiquette) and give an AI summary in 3 bullet points because, well, the article deals with 3 points, so it's a good fit and I doubt I can compress it better than AI:
1. Willfulness. The desire to get your way regardless of obstacles or consequences.Being fearless about consequences and pushing through "hellacious" moments when others quit.
2. Discipline. The ability to force yourself to do what you planned to do. Being "hard on yourself" and turning off the background analysis in your mind that questions the plan.
3. Ambition. The grandness of your destination and what you want to achieve. Choosing the right target; if determination is a vector, ambition is the direction it points.
Of these 3, it's quite obvious at this point that LLMs are superior at 1) and 2), as LLM alignment is quite good, and it's ability to spin up sub-agents that can work for over 24 hrs is better than humans. Human limit is basically 1 hackathon's worth of cognitive work. Beyond that, health becomes a concern, and how many people can produce useful work under sleep deprivation? Codex can work for 24-ish hours, as evidenced by today's "Auto-research with codex: How I achieved a 232x Faster Kernel": https://news.ycombinator.com/item?id=49309549
So what's left for mere mortals? Apparently, nourishing your ambition:
"So here in sum is how determination seems to work: it consists of willfulness balanced with discipline, aimed by ambition. And fortunately at least two of these three qualities can be cultivated. You may be able to increase your strength of will somewhat; you can definitely learn self-discipline; and almost everyone is practically malnourished when it comes to ambition."
I feel like we also miss something related to #2 (Discipline). I never needed discipline to sit at a computer coding as a kid for hours on end — if anything I’d have needed massive discipline to NOT do it.
I wonder if what we call discipline (and the resulting skill that we call talent) is really just insanely strong and aligned interest.
It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
AI has sitzfleisch
https://en.wikipedia.org/wiki/J._Robert_Oppenheimer#:~:text=...
My browser doesn't do the fancy URL text selection...
sitzfleisch: the ability to endure or carry on with an activity
Something Oppenheimer did not have, apparently.
loaned from German, where it's originally a way to say buttocks, literally "sitting flesh". If you have more Sitzfleisch you can sit for longer. Both in the literal sense (a bigger butt makes sitting more comfortable) and in the figurative sense (having the mental ability to sit for longer, get more desk work done)
I think there is an IT bit of humor from about 1 decade+ back where the people who's proposals won out in meeting were the ones that could keep from needing to go to the bathroom longer.
Fascinating crossover from literal to figurative that you find so often when you trace language back far enough.
in Gujarati its "Gand vagar no loto". ( Utensil with Round Ass ). I guess every language as its equivalent.
This is fantastic. Now I have a sophisticated-sounding german word for my attention deficit.
There's also a less flattering reading of the word, where Sitzfleisch means having a "flat ass" (from sitting too much, e. g. Sitzfleischparade describing a group of flat-arsed people, or something like Sitzfleischmaxxer, and so on).
This was such an interesting thing to learn
We Americans call it grit.
The connotation of grit to me feels different. Grit is pushing through adversity, rather than just tedium.
Wow, what a great comparison. LLMs are great at reasoning but absolute dogshit at simple arithmetic. If there's a raw calculation involved I always tell it to use python to add it all up.
Ever heard of string theory.
People go whole lives without being able to make it pan out.
String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
When people make these comments about string theory, are they at the forefront of quantum physics theory and have spent years on modern string theory? Or did they just watch a YouTube video and then keep parroting this?
You could reductio ad-absurdum this logic quite broadly. We all form beliefs about the world and its aspects, and almost always through fallible sources. Should one be confident about ones of questionable provenance? I say no. But forming beliefs from the information we have is a useful skill.
Nothing wrong with learning and being influenced. However, many posters on HN are very confident that string theory is a dead end. It makes me question if they’ve done cutting edge string theory or it is just some random software engineers who watched Sabina on YouTube.
The degree of confidence matters here.
In the age of infinite bullshit generators, the even more useful skill is being able to say "I don't know." Or "I'm not qualified to have a useful opinion on this so I'm just not going to say anything." I could write a bunch of stuff about my personal opinions on string theory and waste everyone's time, or I could just shut up and not be a human stoichastic parrot.
This isn't a great argument. Researchers who do not think string theory is good are not going to spend years on it. There are plenty of experts that dismiss string theory. I have no skin in the game, and don't care either way, fwiw.
I agree but I think it’s a little hopeful to believe AI will save string theory
I also just keep parroting that the Earth is not flat and a bunch of other things I've been told and then never questioned, since it seems to make sense.
When the flath-earth craze started I've been trying to at least get that bit actually personally verified. Haven't managed to do it to this day, though. So I'll just keep parroting various things without properly understanding them.
C'est la vie.
Your drive to verify must be very low then. There are hundreds of small logic reasonings or even experiments with which you can easily prove that a flat earth model is at least very complicated or even impossible.
Explaining Moon phases gets very complicated in any flat earth model. With binoculars you can see the shadows of craters on the moon's terminator. Or the phases of Venus whereas Mars doesn't have any.
Timezones are ridiculously hard to explain on a flat earth.
A proper theory must explain all of them and no flat earth model can do that. A round Earth OTOH easily does.
Or the just read Woit, uh, 20 years ago now, “not even wrong”.
Sabine has been at the forefront and has the goods: https://youtu.be/1mgxuVVUz40
And for the inevitable critics of Sabine...maybe Leonard Susskind is good enough for you: https://youtu.be/2p_Hlm6aCok
The error with this approach is that every single physics theory gets exactly the same criticisms.
You can take any of the theories that Hossenfelder would spend time on instead of the ones she does not like, and you would find (basically) a similar percentage of physicists saying it's a mistake to continue in this direction.
In other terms: for each physics theory, on 100 physicists, you have 5 physicists saying it is a mistake to continue working on it (number made up for illustration, and there is probably some variations, but you get the gist). You took one theory and found few physicists saying it is a mistake to continue working on it. You conclude, incorrectly, that it means this theory is fundamentally differently treated as any other theories.
(on top of that, it is unfortunate that Hossenfelder later screw up her image by doing way too much mistakes that someone reliable would not do)
> It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
> String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
Not only are LLMs perfect machines with all the intelligence of humanity without any of our problems, they are also everything else. I wait to get my hands on one of those LLMs people on hn seem to be using. I want to believe too. Let me into the religion of the perfect thinking machine gods.
Ok Sabina, everyone is string theory is all just a bunch of egotistical morons.
No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
I don't think it's a good argument. String theory is just "one application" of advanced mathematical physics. It's like saying that software developers specialised in ReactJS would lose their career if ReactJS is suddenly abandoned.
Not a fair comparison because one is exposed to real life market forces vs. academic inertia.
ReactJS devs will retrain when the market dies. Professors still publishing theories/experiments costing large amounts of public monies better spent elsewhere.
As opposed to having built a career complaining about string theory on youtube?
And that's the reason she's wrong?
You were the one introducing the argument "they are getting advantages out of it so we should be careful to not take what they say as an impartial view". Same can be told of Hossenfelder (personally, I would say that Hossenfelder has shown more clues that indeed she is not very reliable and would prefer caricatural sensationalistic descriptions, which is not what the typical string theory physicist does).
No, my point was only to refute the idea that people need to be egotistical or moronic (the only two options) to cling to an idea past its expiry date. People could also be “pot-committed” career-wise. Or they could have myriad other reasons (identity, sunk-cost fallacy, etc).
Nowhere did I suggest that “the reason for discrediting string theorists is that they have financial stakes in the idea.” Please try to be more charitable than that.
You said
> No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
This is not presented as "yet another option that may or may not be the reality", this is presented as your conclusion of why we observe what we observe.
Maybe it is not what you are thinking, but you cannot blame people for interpreting your message the way you have written it.
I agree with you that there are several options, one of the most probable, that you did not mention, is that the situation of the string theory is just "normal" and some idiots are not able to understand that.
Spending your life's work on a theory that is widely agreed to be untestable is certainly a form of ego, or at least is more philosophy than science.
The "untestable" meme is just a thought terminating cliche. It's as if you believe no researchers ever considered the brilliant insight that the framework is difficult to test. History is littered with plenty of examples of a novel discovery, understanding, or technology yielding new opportunities for experimental design.
It's also a glib dismissal of the unexpected mathematical elegance of string theory that makes it so compelling. Worse, it completely ignores the material contributions to applied physics that string theory has made possible which is obviously worth "spending your life's work on" - as if people need to justify the value of their life's work to disinterested onlookers lest they be demeaned.
This is a pretty silly belief to hold about science. Plenty of strong, well held standards of the modern era were considered ridiculous fringe beliefs a century ago.
They were testable, no?
Not taking either side here - but surely a lot of cosmological phenomena couldn't be testable/verifiable at the time they were theorized either?
(thinking of black holes for example - they were theorized way before we had observations. And presumably a lot of particle physics can similarly be theorized before we built the technology to experimentally verify them)
Well, we can test it, but you'll need to give me a ton of money in order for me to test it.
even if its not particularly relevant to the physical world around us, it still explored cool math
Also out-speeding them, and that was before high speed inference.
Out-ralphing them, you might say!
https://ghuntley.com/ralph/
AGI ≈ artificial stupidity × infinite persistence
> It's also "out-brute forcing them."
That is also approximately what people have always done to succeed.
Mathematicians routinely spend years on a problem without getting anywhere.
It helps though if you have many problems to work on
The key here is that it’s depending on the human inability to connect the sum of relevant knowledge, but said knowledge comes from humans.
Theres going to be this field day of low-hanging fruit that ML can round up, but after that I suspect it will be in fits and starts as a “connection maker” rather than some proof producer.
I think we're underestimating just how much low hanging fruit there is. I've been trying to apply this LLM research process to physics (QM and solid state) and there is so much missing in Physlib and the rest of the Lean ecosystem that most of my work has been trying to formalize the theories and validating them against the specification problem (and mostly failing badly).
It's not only going to be "connection maker". If and when robotics advance to a point where the LLMs are embodied, they can run experiments in the physical world and find new knowledge.
Being embodied is not the important barrier to running experiments. It's having access to a body of resources (i.e. funding and infrastructure).
For some research funding is mostly the salaries of the people doing the research.
What research works that way?
large parts of Computer science, mathematics, theoretical physics, economics, some humanities research and I'm probably missing a ton.
So applied math, math, applied math, applied math, and maybe some others.
That makes sense, although I'd argue that at least in the realm of HEP theoretical physics has extraordinarily expensive kit compared to what scientists make. See: The LHC.
Reducing all these fields to mathematics is incredibly reductive. I don't think it's in good faith.
I think it's telling that "Researchers cost more than any other aspect of research" seems to involve primarily math as inputs and outputs. Like I said the obvious outlier is theoretical physics because that tends to involve significant resources both to explore (supercomputers) and verify (colliders, telescopes, interferometers, etc). Most people involved in research in any number of fields do not find that their collective salaries are the majority of the cost required; math is an outlier in many cases.
yeah but LLMs might be even cheaper than grad students
Robots in labs already exist, but mercifully they're not hooked up to anything as unpredictable as an LLM. Robots tend to work best as specialists doing high-throughput, extremely repetitive tasks which nonetheless require a degree of precision. Giving a robot a "human" body makes very little sense if we're talking about the needs and productivity of a non-human; humanoid robots are marketing for humans.
Humanoid robots are obviously more than marketing. The entirety of human civilization is human shaped. Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
> The entirety of human civilization is human shaped.
That's the marketing pitch.
A plumbing robot doesn't need to be humanoid, an octopus shape may well be better for all the awkward corners. A robot police officer could be the municipality itself for sensory nodes (essentially the sales pitch of Flock etc.), plus some drones or robot dogs to perform arrests*.
The robot vacuum cleaners and lawnmowers we already have are nothing like a human. A robot taxi driver can be just the car. Robot dogs are already used for maintenance and security sweeps.
If you've got wheelchair access, you've got wheeled robot access. If you've got guide dog access, you've got access for Boston Dynamics' Spot.
* this may be a bad idea with current robotics, but I aver it's not improved by making those robotics humanoid.
You are not understanding what humanoid robots are about. Those are specialist robots you are describing. The promise is of course one robot that can do the plumbing, clean your house, do the dishes, build a house, and basically every physical job a human can do. It's extreme lly likely that at least a somewhat humanoid shape is required for that.
My octopus will outperform your humanoid by doing the dishes, fixing the plumbing, and preparing a Caesar salad, all at the same time!
Ah, but my "five guys" will outperform your one octopus!
Yeah. Can't wait to see three or four armed robots. Or five or six!
How do you train a robot to use three hands effectively when we only have two? Then again, why is the robot limited to being one robot? If two humanoid robots are in the same area, they don't have to be distinctly controlled. If they're both controlled by the same AI, a third arm on one body is the same as that arm being attached to another body.
Humans are really bad at everything we do compared to the specialized creatures in nature, it’s just that we can do many things that sets us apart. A humanoid robot is incredibly dumb for that reason. At least add a set of arms and legs and 360 vision. And obviously dislocated joints than can move freely and hands should have two thumbs and more fingers.
It’s not hard to come up with a bunch of improvements for humans, it’s just that making robots in our image is a lot more trivial because you only have to solve for those same averages attributes that we have.
Things you need to take into account that you've almost certainly glossed over:
Safety. Human-robot interactions are generally dangerous and avoided, unless the robot is specifically designed to interact with people. In those cases you often sacrifice speed, strength, and flexibility for safety and softness. Having someone come in with a specialized plumbing robot makes sense, you owning one probably doesn't, and you owning a generalize android capable of plumbing makes less sense still.
Cost. The more compact, complex, and interactive your robot is the more it costs. Make a strong, compact, complex robot safe for interactions with people in the wild is non-trivial and adds costs. The software required to do all of this is hypothetical, but obviously also costly.
Need. I understand the dream of a robot to do whatever you want is very much part of our culture, but when you consider the downsides do you really need it? I don't need a plumber living in my house any more than I need a carpenter or a landscaper to live on premises. At most these are services I would need occasionally or on a schedule. I also doubt my need for them will overlap much, unless we're talking about building a new dwelling.
So why do I need a generalist in my life that's going to cost more than you can imagine, when the means to hire existing human generalists is cheap, quick, and frankly less likely to accidentally punch a hole in you.
> Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
In the last 100-200 years, that has been proven wrong at every single step.
Historically this has never been correct. Turns out you get more efficient systems when designing them without how a human would accomplish a task in mind.
Counterpoint: cars. It takes sustained effort to prevent civilization from being modified to accommodate new technology, e.g. Stop Kindermoord.
Outside of sci-fi, marketing proposals, and niches like "Elder care in Japan" there are very few humanoid robots. By contrast non-humanoid robots have been mass produced and used in industry for decades. Arms. Carts. Trollies.
No people. If you want something with fine motor control and dexterity, it's easier to make that the robot and then have another robot bring the workpiece to the arm than it is to build a single robot that can walk around and do it. There are compromises in human features because we're generalists.
>elder care is niche
The 2030s have some bad news for us...
https://www.youtube.com/watch?v=n-gYFcVx-8Y
Counterpoint: something has happened in frontier models, and yes they now get discouraged and will sometimes prefer to not continue working on a problem unless you tell them to anyway.
I don't know how or why this would be trained on behavior, but no, it isn't true anymore that models don't say things like, "Ugh," or "this is going to take hours and maybe we should stop here."
I think it's to combat runaway token usage.
> If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc.
Your timelines are a bit unambitious. There's nobody expecting to make significant progress with a week of work.
> There's nobody expecting to make significant progress with a week of work.
You underestimate my ADHD.
Source: I am mathematician.
Hell, I underestimate my own ADHD.
Source: the post-it notes, ALL OF THEM.
They were just illustrating their point, I wouldn't take that literally.
We're just going to slowly deconstruct every element that could be a factor of intelligence.
It's not out-thinking, it's just out-remembering
It's not out-thinking, it's just out-working
It's not out-thinking, it's just able to consider more things simultaneously
It's not creative, it's just randomly generating things and then selecting viable ones
A new technology being able to do something better than humans does not mean it’s intelligent though. A calculation program is not intelligent just because it can remember more digits than me, work more than me
That's insightful. AI is teaching us things about our own intelligence by simply evolving under our eyes.
If you make that list comprehensive, there's probably a Nobel prize in it for you.
But the difference really does matter and is not just a case of "whittling down" what intelligence really is.
We have known for a very long time that computers and machines are much faster than humans, more accurate, are scalable in certain ways that humans aren't, and they don't tire. I think most people who are not in the "AI cult" would agree that LLMs and modern generative AI are really just an extension of those faster/more accurate/more scalable and never tiring traits. But there does seem to be (and I'm sure folks much smarter than I have quantified this or described it better than I can) a fundamental difference in how humans think, especially as it applies to what true "understanding" really entails, and for the ability to think up truly novel and unique things that are not just a rejiggering/recombination of training data. I believe those skills really are at the heart of human cognition, and as impressive as LLMs are in replicating what this looks like, there are plenty of "LLM failure modes" where it's clear that LLMs lack a true understanding of concepts or the ability to generate useful, completely novel ideas.
There's definitely a lot missing from the current state of the art in machine learning that all brains manage to beat, and we can observe this just because an animal that needs as many examples as an AI to learn motor functions would starve to death before learning to eat.
However I can only guess that this is important, I'm not absolutely certain. They're at risk of being an economic disruptor just by being extremely stupid (by how much they need to study) faster than us to the same ratio we jog faster than continental drift.
Nobody knows what true "understanding" really entails, or what are "truly novel and unique things that are not just a rejiggering/recombination of training data". For the latter, you'd at least have to find an example in history of someone who came up with some idea that has been widely considered "truly novel" by experts, who didn't have any education or training, so no "rejiggering/recombination".
It's almost as if we're building something that... mimics intelligence.
Can you mimic intelligence?
You just did.
Boooom! Headshot!
>that could be a factor of intelligence
Could is carrying a lot of weight here.
Because, what's really happening is we're saying "Oh these things are what defines intelligence" then implementing them and /discovering/ "oh wait, there's more to this than we knew".
We've known, for decades, for example that an IQ test is not a measure of Intelligence, even though people still refer to it as though it is. A computer passing an IQ test, therefore, would have been thought of as possessing intelligence way back when, but would not now.
Oh, on the point of "creativity" - is a RNG "creative"? It creates a value unbounded by human intervention (in theory, yes Pseudo RNGs have limitations) - therefore it must be creative... right?
In other words: Thousand monkeys with a thousand typewriters...
https://news.ycombinator.com/item?id=48231974
Take something like
If a thousand monkeys typed at a character per second, on a keyboard with the 23 relevant characters, it would take roughly 10^136 years for them to come up with this counterexample. Though, to be fair to monkey scenario, there's a large family of them known now, so it's not quite this bad: suppose there are a trillion permutations and similar examples that fit in this string. Then we are down to 10^124 years.If LLMs are monkeys, somehow trained LLM weights allow them to model and prune massive numbers of universes in parallel.
Just FTR - 1 character per second is glacially slow - it's 12 wpm - fine for (slow) transcription, but the monkey typing exercise doesn't require them to know what they are typing out
One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
But AI agents have no such limitations and can publish and re-use negative traces easily. There have been some recent projects (https://www.theoremdb.org) aimed at exploiting this fact. https://news.ycombinator.com/item?id=49227505
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
The bandwidth is absolutely there ("we tried this and it didn't work" is totally the stuff of conference discussions).
The incentives are not.
The incentives are skewed towards "a magician never reveals her secrets". The results are presented as if a rabbit got pulled out of a hat, with a maximum ta-da! effect, and little backstory of how the hell did we get there.
Don't get me wrong, these things are discussed, often over beers (you better drink it you want to make a career in the field).
But not published.
The younger mathematicians are trying to change that with the blogging culture. But the professional incentives aren't there. (In corp-speak: can't put blogging on perf). They burn out.
That's why math blogs usually come from either the top dogs in the field, like Terrence Tao, who don't need to care about perf, or people outside academia.
That's one thing that I hope the disruptive/destructive effects of LLMs will force mathematicians to face.
As one of my fellow mathematicians sarcastically wrote¹, we've reached a point where we should become a cult because we're acting like one anyway.
The other possibility is, of course, that the shake-up will take us precisely into that direction.
My point here is that the real problem here is not mathematical; it's a social one: incentives and politics, organizational structures, policies, allocation of jobs and funding.
All of this directly impacts how we do mathematics, who we do it with and teach it to, how we teach and communicate, and, of course, what math we even do and look at.
Given that, I'm neither too worried about humans vs. AI standoff, nor hyped about the Glorious New Future full of AI-assisted discoveries.
AI or not, the organizational issues in the field are still there, as are the incentive structures (including the infamous publish-or-perish).
We are doomed, yes, but by our own hands and committees. And it's up to us, not the AI, to get us out of there.
The little shove from the AI might be just the thing we need.
____
¹ https://www.mcsweeneys.net/articles/an-open-letter-to-the-ma...
This is not specific to math. To non-academics, you basically have to rewrite your commit history to make it seem more impressive. You often write the motivation section last. At least math publication culture allows saying "We consider the problem of" and then solve it. In AI/ML academic papers you need much more "story" around why, what the applications are, why aren't you doing something else, defend against lack-of-novelty attacks, defend against "this is just A + B known techniques used together" etc.
Yeah, publishing in ML can be annoying because of that. What's wrong with "I thought this problem was interesting"?
I think its a total travesty of our society that people are desired to identify both the problem and the solution
people without solutions are told not to point out the problem
and people with the skillset to bring solutions dont have the skillset to search for problems or apply their seemingly unrelated disciplines to it
so you get a lot of capital and energy thrown at people who purport to be both
If AI can address this too, that would be profoundly impactful. The cross disciplinary work that a single college course is supposed to reveal, finally realized
Yeah, LLMs are great at generating negative results for math-related prompts "We scanned values {a,b,c} from 0-100 and no results" Great..too bad journals will not publish this. But good job, I guess. A negative result is only truly useful if it can be bounded, requiring an actual proof.
> A negative result is only truly useful if it can be bounded, requiring an actual proof.
That's at least true for current journals, since they're supposed to be read by actual humans. I suppose one could imagine a sort of "AI" pure data journal that just "publishes" (in actuality aggregates) any sort of partial result. This body of knowledge would be entirely useless to humans, but could serve as a sort of "computation cache" for these stochastic systems.
> would be entirely useless to humans
why useless? humans can navigate these unsuccessful chains of thoughts to build on top of them or reject completely
What is next years tsunami of change? Asics?
They are always out of stock, and you can never find your size..
I can't help but think of Michael Nielsen's essay "Augmenting Long-Term Memory" [1].
> Many people's model of accomplished mathematicians is that they are astoundingly bright, with very high IQs, and the ability to deal with very complex ideas in their mind. A common perception is that their smartness gives them the ability to deal with very complex ideas. Basically, they have a higher horsepower engine.
> It's true that top mathematicians are usually very bright. But here's a different explanation of what's going on. It's that, per Simon, many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans. And what this means is that mathematical situations which seem very complex to the rest of us seem very simple to them. So it's not that they have a higher horsepower mind, in the sense of being able to deal with more complexity. Rather, their prior learning has given them better chunking abilities, and so situations most people would see as complex they see as simple, and they find it much easier to reason about.
I once tried out his Anki approach during a math lecture. Whenever I reiterated a card, say about some lemma, I noticed something interesting about it. This was delightful and many lemmas became much more streamlined over time. It's not a "solution" to mathematics, but I found it delightful while it lasted (before akrasia or lack of time kicked in and I stopped doing it).
[1] https://augmentingcognition.com/ltm.html
> many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans
Do you really think an average person can internalize complex math? Them compressing it effectively and then remembering it is a sign of (very) high intelligence.
It's probably worth checking the site history before posting. The guy who wrote this looks like some kind of "race science" crackpot.
Interesting fellow. He has publicly claimed that he has ESP:
https://www.splcenter.org/resources/hatewatch/wikipedia-wars...
https://openpsych.net/forums/18/thread/25/?page=1#124
Gee, 100%. Articles basically saying genetics explain everything.
I thought this was fairly obvious. I don't consider any of the AI models I've worked with particularly smart, but they've read orders of magnitude more than I ever could in my entire lifetime. They have far better book knowledge than I have, so that's how I use them. I use them for things that I suspect other people out there would know, but I don't. But when I work on something that I suspect is truly new, the models rarely understand what I'm doing, and I've got to do it myself. Although I still poll them for basic principles, best practices, and other advice.
It was duuring my PhD I realised that thinking is essentially
1. remembering all the different information to remember all the tricks
2. trying all the different tricks in the problem
3. optimizing deciding which to try based on different information
4. trying random things to discover patterns (and hence new tricks)
5. explaining your tricks to others so that they can do the first 4 steps independently and come up with even better tricsk
6. refactoring tricks into common and special parts to create a well organized theory
7. documenting for future generation in a language they can understand
that's all thinking is.
LLMs are enormously good 1. and 2. But humans are much better at the rest.
I don't want to read another LLM documentation ever again. The information to text ratio is way too small.
> That's all thinking is
That's more because a PhD (especially in this socioeconomic era where academia is also a microcosm of capitalism) does not automatically teach or focus on deep inquiry, rather narrow research programmes. Case in point, fallaciously reducing it all to a superficial theory of tricks means that by the same light, general relativity or any other major scientific result is merely just a trick. That framing ought to be absurd on the face of it but the fact is many PhD students do still graduate with very unexamined understanding of science, cognition, philosophy, etc. (Another way to see this is to note how such an argument is letting the notion of "trick"/"heuristic" do all the heavy lifting--something an advanced education should have nurtured the critical thinking skills to avoid that very metacognitive pitfall in the first place, which is ironic.) (Yet another way is to see how increasingly experts of one field or another insist that one must stay on their lane when speaking about a complex topic.) There are professors and thinkers who have written/spoken about this but they are a minority. Even Einstein himself when he complained that more and more scientists were trained/structured to miss the forest for the trees, and that was almost 100 years ago.
While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
Sounds like another attempt to frame AI in a way that makes them feel better about themselves.
The simpler explanation is that a working memory is a requirement for intelligence, and a larger working memory will make you more intelligent. Hence the AI can in fact be more intelligent than the mathematician.
Depends how you define intelligence. I struggle to consider LLMs actually intelligent, at least for any definition of intelligence that I would be content by.
Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
For greenfield projects LLMs don't need abstractions, but as the project gets more complex, the right abstractions save a pot on input tokens (less code to read) and reasoning tokens (less work to do to figure out the code), so they free the context window for higher purposes
Also I suspect that, apart from that, the results on smaller, cleaner codebases are better. LLMs degrade when following more than N instructions (where N depends on the model) even if the context window is not full yet; I suspect they also degrade when code has too many unnecessary concepts and details
I've been working on generating a large code base for the last couple of weeks. Finally got around to generating a sort of code-duplication report and have spent the last week just having it de-duplicating logic that had been strewn all over the place (eg 11 different functions all doing date math to add x days to a date). dozens of items that had each been similar functions duplicated numerous times. crazy. (opus-5-utracode)
LLMs love duplicating logic. Unfortunately they are not formal systems so we actually have to read their output
Can LLMs not do this for you? Or would they go too far?
No, the problem is they're still really dumb, and lack the ability to make logical connections that are obvious to us. "should I walk or drive to the carwash" being a very recent example of the larger problem.
But they are very good at identifying patterns though. This task requires little logic IMO. So why not leverage LLM to perform redundancy analysis?
They add and add new code to the point when adding more is going to become very messy and then spagetti
What I was suggesting is to ask the LLM to compare function usage across files to identify overlap/redundancy.
The idea is to use the LLM to analyse the code base first before taking any action.
I think I'm suggesting a centaur approach.
This is the exact opposite of what I’ve been dealing with for awhile. LLMs absolute cannot work on something without an understanding unless they can outsource the understanding to a verifier. If you’ve got an easy to check function to measure progress then “keep going” is all the prompt you need. But if you need it to figure out “I pushed the up button and it moved up and left” then it’ll find the same bug five ways without realizing it’s just one bug in the underlying math.
LLMs use abstractions a ton in code though: standard library functions, popular libraries, etc. They just dont always make their own abstractions. At least not particularly good ones. LLMs work really well when they have well abstracted pieces to put together.
With good source code they output good code that mimics existing code.
If they don't have good code to mimic, anything goes. They might use libraries or they might start to write their own implementation of a PNG encoder.
LLMs are still missing a part of working memory. Part of working memory is being able to attend to small amounts of information and then understand and parse all the pieces of that information. When LLMs use their "working memory" they just analyze different probabilities of tokens and there is no prioritization or understanding of the information in the way humans have it.
If there is no training data or data in the context that leads it to the correct result then it can't do it, whereas a human seems to be able to generalize and abstract a goal and then repeat an action or thought process in a 'recursive' manner to reach the result. AFAIK LLMs don't do this.
Just as an example to illustrate. I recently asked an LLM to organize a bunch of artists albums into whether they were released by a major label or an independent label, and for the most part it did a good job. But there were albums that it classified as independendent that weren't. I presume because it either didn't run into the right data when searching or it misunderstood the data it did find. A human would not do this because if a human had a list of all major labels, it could instantly detect whether an album was or wasn't indie, because it doesn't do any complicated parsing or token probabilities that LLMs do, it just recognizes a pattern (either an album is indie or it is not, a human brain needs simply one piece of information to decide this), an LLM is not that simple.
In a way human brains are simpler than LLMs. The algorithms it runs mentally can detect a piece of information and then see most / all of the consequences of that information whereas an LLM thrawls through megabytes of text and does a token probability distribution and so on without any simplicity.
A superhuman working memory is exactly how I've been describing the LLM advantage. Paired with the unreliability of its reasoning and judgment, it is what makes AI a supplement to human intelligence, not a replacement.
On the other side, LLMs make random mistakes and wrong choices and they have a bias toward writing more code instead of less. You can make up for this to some degree by running another LLM against their output, but with very diminishing returns. Even if they were perfect, there will be an ongoing cost to little or no human awareness and understanding of the codebase.
It may take some time for people to recognize the cost of AI code generation and their value for virtually everything else, but I believe we'll get there.
I also find them bad at what I call "abstraction compression." They're really bad at noticing when a helper function is needed, when some structure they repeated five times slightly differently can become a struct, when a whole section of code can be encapsulated in a simpler design.
I'm lucky that for my side project (an interpreter) I've written all the code myself, so I've built up its design in my mind over the past year, and so as I mull over what I'm writing I start coming up with simpler designs.
Interestingly using Opus 5 (and LLMs in general) has made me worse at this, since I don't feel the pain of writing something over and over again. On the other hand, I don't really want to implement a whole stdlib, so I have it write more of the auxiliary code. The hardest thing is that I have to manually manage the context, which is painful when I personally remember every helper function and why. I have to remember to keep the list of helper functions updated, which is irritating.
if you take any system and break it down you get to simple building blocks (like memory). Then you claim that it isn't "outthinking" but "out remembering". Sorry, but what do you think the brain is doing? Not using memory? Especially working memory. How about frequency, brain is operating in Hz whereas most processors are in GHz, etc.
Finally if we were to think in terms of a proper memory/latency hierarchy, what's a Model + CPU's equivalent of working memory? Registers? L1 cache?
So does the cheapest calculator?
Memory and intelligence are linked.
Someone with a better memory for ideas or concepts will be able to more quickly incorporate those into novel ideas or recall them when necessary to assist in solving a problem than someone with worse memory.
To those here challenging this with “yes but I’m smart and my memory is bad” - a) define smart and b) perhaps your memory for trivial things like life events, what you did two weeks ago on Monday or people’s names is bad, but I suspect your memory for “work” or problem solving is strong.
Another example I used to see (hear, rather) is how musicians rip off each others riffs and hooks without noticing (unintentionally - they claim), which I long suspected as simply “forgotten” riffs they heard in other songs that once they started playing themselves by chance they attributed to their own creativity. Creativity and intelligence are somewhat linked that way I suspect.
In any case, this all boils down to the same thing, you can think of yourself as a dynamic model made up of memories and biases to some degree, and your ability to store and recall useful information to solve problems increases what we call your intelligence.
I often think “what will ‘maintainable’ code look like in the future?” given this kind of thing. Much of ‘good coding’ is about optimising the codebase for workability for a human intellect with human working memory.
Though having said that, from the studies I’ve come across it seems like LLMs tend to generate more verbose code but perform better over the long-term when the code is maintained and not allowed to sprawl all over the place.
I’m mainly using AI for tools development and in that context I often wonder if I’ve just developed a career-long habit of over-abstracting. Like, the tools work without all the fuss I might have put in at the beginning and you just iterate and evolve as you go.
Is this assuming or studying human maintenance?
I’ll dig up the references but I believe they were studying what happened to LLM performance as the complexity and verbosity of the code went up (ie- just letting the LLM rip without doing explicit code cleanup and simplifying).
IMO, That's the case for all subject areas. It is also one area where AI excels at, and it could be of real use if we can find a way to stop hallucinations. The sum of all knowledge being at our finger tips would allow everyone to focus on the hard stuff.
True, but as it pieces together new mathematical truths from the pieces we have discovered ourselves, it then has more truths upon which to build new solutions. And so on, so while it is just remembering things we have forgotten, the amount of progression an LLM can make may still be several steps ahead and touch areas we have not yet been able to consider or make any progress on ourselves. It's a bit like a pyramid though, eventually it will have tiued together all teh things we know, found all the things we could have known, and then .. perhaps, be unable to actually come up with something genuinely new.
Which is why love it! It can easily did through the code flows of our 250k file project and more often than not and understand what's happening much faster than I ever could.
Consciousness and self awareness should have been created by now if the memory and nodes pool had exceeded the human brain, right?
Outside of math you can basically take the entire corpus of research papers on any topic and have the AI read all of it and provide an analysis cross referencing everything all at once. This applies to everyone and everything.
That intelligence is "just" retrieval within a huge corpus is an old one, and the one I am thinking about a lot these days: https://news.ycombinator.com/item?id=44060672
This is why education used to start with rote memorization. Functional intelligence isn't abstract, it is based on useful information you can quickly recall.
What I'm looking forward to amidst all the negativity, fear, and loathing is for some 20something mathematician to outdo both humanity and machines by leaning hard into centauring to expand the frontiers of mathematics. Pretty much what I think the future will play out to be as well, but I don't think people are ready for that yet.
Takes the rogue having access to the $$$$ models though.
That currently costs $20 a month, $200 per month if you need the premium extreme deluxe package subsidized by everyone else paying for it and not utilizing it. When I was in grad school I budgeted the equivalent of $3,000 per year to maintaining my compute hardware and that was a very long time ago. So I think it's doable.
People aren't ready to prove/disprove every one who access to LLM and claims has changed course of humanity
Does it matter? It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
We don’t trillions of dollars in LLM investment to build things mathematicians don’t understand. We already have plenty of those, even from ancient times.
As to your second point, Terry Tao already has an answer [1]: the proof isn’t the contribution, shared understanding is. This issue was already raised back when the four-colour theorem was proved. Machine proving and machine proof checking are useful tools but they don’t mean anything without the interpretative work and the communication necessary to build shared understanding.
[1] https://news.ycombinator.com/item?id=49056620
If humans have nothing to contribute then shared understanding is a pointless endeavor. It makes sense now in the "centaur" period where human + AI > AI alone, but when AI mathematicians are both more rigorous and more elegant, then taking the time dumbing down their proofs to a human level of understanding is like requiring that we ensure all our current proofs be understandable by a monkey.
> If humans have nothing to contribute then shared understanding is a pointless endeavor.
I agree, but as a software engineer this gives me pause because I keep trying to insist on coding standards but I’m unable to come up with a compelling reason why it matters. Ostensibly the reason we cared about things like DRY and code quality was so that it would be easy to understand and easy to maintain and easy to make changes to later. But it now seems like a shared understanding of the codebase is less important than ever, and it’s more about shoveling requirements in without breaking any existing functionality.
Is a well tested slopfest better? That seems to be the conclusion for mathematics, so why not software too?
> easy to make changes to later
IMO it's still a problem with LLMs; we still have to build in a way that makes it easier for an LLM to make changes later and arguably it's the same things that made software development easier for humans. IME LLMs tend to not know how to do that for themselves and instead just amplify/copy patterns that already exist.
If an LLM can't pave the way for itself then ultimately shared understanding is required to take advantage of LLMs in the first place.
It used to be the same with assembly. Programmers complained the one generated by compilers was not pretty, but now in 99.999% of the cases, it does not matter because nobody look at it.
I beg to differ because a compiler is deterministic.
Did you check? Do you care if it sometimes does mov ax, 0 or sometimes xor ax,ax? (Forgive my bad memory, it was a long time ago)
Would you personally vouch, at your job, for the importance of proper assembly coding standards?
Yes, the same compiler generates the same output given the same input. I'd be willing to put a large amount of money on that result.
I read a lot of assembly.
I was more after the fact i can trust the compiler to give me the same result - even though via an optimized path.
Certainly scopes vary, but in my line of work i define memory layout and how this data will be processed myself - thus it's great a compiler might do that, but the result of the computation will not change.
Now in comparison giving an LLM specs ... i a) cannot be sure what the computation will be b) it might be something else on another run.
Where is the value in an unintelligible gibberish proof?
We already have countless examples of such filling up the arXiv, written by hacks long before LLMs started writing proofs. No one cares about them. You might as well build a box blasting radio static into the void. You could save a lot of electricity that way.
An unintelligible but correct proof is better than no proof. These first AI proofs may be overly complex and un-elegant, but they are the worst that frontier math proofs will ever be. AI math in 2030 will be leaps and bounds ahead of humans both in rigor and elegance.
Those are some strong claims. I’m deeply skeptical of all of them.
I believe Terry Tao when he says the bottleneck will no longer be the writing of proofs, it’ll be everything else: reading them, reviewing, publishing, and teaching from them. A bunch of proofs that nobody reads are of no use to anyone.
But but but... Does not AI exists (or has to exist) to only serve us.
I suppose it could compete with us.
Why not? We build cranes to hoist weights construction workers can't lift. We build electron microscopes to measure things physicists can't see.
Why is it so hard to imagine we can build tools to think thoughts we can't comprehend?
If there's commercial value, I think it's inevitable. We don't fund mathematicians because it's cute when they understand a problem, but because their work tends to have applications with commercial value. The value can be captured without understanding the details.
Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
> Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
It depends on what exactly you mean by "commercial value commensurate with the costs involved" but I'd volunteer the 3G/4G/5G specifications and the other documentation required to implement the mobile network protocols. 5G is currently sitting at over 50,000 pages and it's one of the reasons Qualcomm/Broadcom/Apple are the only ones who can realistically make a mobile radio.
I don't think there is a single human to whom more than a few thousand pages would be comprehensible at a time except for the occasional genius.
Read the rest of the discussion. The claim is about text which is in principle incomprehensible to humans.
If there exists a text which only one person can understand, that person can communicate their understanding to others, even if that doesn't help them with the original text. That dissemination of knowledge is what provides the value, not the mere existence of the text. If that person forgets or dies before they can share their knowledge then it will be lost.
We have many examples of this from history: ancient texts written in a lost language. These texts provide us with no value until the day they can be deciphered, unless you count linguistic puzzle-solving as a virtue.
> If there exists a text which only one person can understand, that person can communicate their understanding to others, even if that doesn't help them with the original text.
If there exists a text which only an AI can understand, that AI can communicate their key conclusions to others, even if that doesn't help them with the original text.
The only difference here is the amount of meat involved. Perhaps tossing a few steaks on the server racks could help with that.
If an AI can communicate its findings to us in a way that we can understand it, then its findings are by definition intelligible to humans. Your claim was about texts for which this is not possible.
So, you're willing to claim that a conclusion that you don't understand the reasoning for is equivalent to comprehension?
Because, again, I can point to hundreds of examples of texts where nobody but the author understands it, and they're only giving summarized "commandments" that you should follow if you want good results.
If I told you "don't use spin locks, call futex instead", do you think have gained an understanding of the Linux scheduler?
If I told you "don't use spin locks, call futex instead", do you think have gained an understanding of the Linux scheduler?
Yes, because I already knew what schedulers are, what spinlocks are, and if I want to know what futex is I can go look it up. Comprehension is within my grasp.
Your original claim, which you’ve repeatedly distanced yourself from (by trying to use comprehensible examples) but won’t admit to, was about incomprehensible stuff. That is, text that no human could possibly understand, ever.
You’re repeatedly engaging in intellectual dishonesty rather than simply admit that “human comprehension probably will continue for the foreseeable future”, which is really not a controversial idea at all.
Bullshit. Human comprehension will not continue, it will reduced to "when the AI tells us to add Bismuth, strontium, and copper together in this way, we get a superconductor", but that doesn't mean we understand high temperature superconductivity.
For what it's worth, we've known about BSCCO for nearly 40 years, and we still don't have a great physical understanding of how it works, thought we made decent progress in 2022.
That’s still comprehensible. Incomprehensible chemistry instructions simply could not be followed by any human. They would be of no use to anyone.
You're using a very bad definition of incomprehensible. We don't understand the chemistry behind how these things work, it's that simple. If you're going to argue it's comprehensible, you'd better show how it works and go collect your Nobel prize.
All we know is that if we melt the right rocks together, we get a superconductor that works in some mysterious way that we can't explain. We know it's not Cooper pairs.
No, I’m using the right one [1]. You are the one using a bizarre definition for your own purposes.
I can see there is no further productive discussion to be had with you at all. I bid you good day.
[1] https://www.merriam-webster.com/dictionary/incomprehensible
Since it's comprehensible, please point me to someone that can comprehend high temperature superconductivity and how it works, and then explain why they didn't explain it.
With the definition you seem to want to use, it's impossible for anything to be incomprehensible, and therefore it's tautological that there's no incomprehensible LLM output: nothing at all is incomprehensible.
Do you have an example of anything incomprehensible? Anything at all? Even the things that Gödel would say are inaccessible could, in theory, become accessible: we just don't know with absolute certainly that the mathematics it was based on got all the axioms right, though we see no errors now.
Yeah, how many people do you think understand the Linux kernel in full? What percentage of the people using it to great commercial effect can understand it?
How's your understanding of Schroedingers "An Undulatory Theory of the Mechanics of Atoms and Molecules"? You seem to be using the results of it as applied to semiconductor engineering just fine. And, I promise you, most semiconductor engineers haven't read it in full, they just accepted the results as passed on by several layers of teacher.
I have a paper on routing algorithms, which I have attempted to read to my cat. I don't think my cat retained much, but they seem to be enjoying the cat food that got delivered using the results.
I'd suggest that we're going to be a lot closer to the cat than the author of the paper when AI takes off.
You've moved the goalposts. The original claim was about producing mathematics that are in principle impossible for any human to understand.
All the stuff you've listed is understood by some person, and that understanding is the source of its value.
Now that we've cleared that up, can you furnish an example that satisfies the original claim of incomprehensibility and value?
You asked for examples, for a technology that we're still building. Maybe you can see the issue with that?
Anyways, people benefitted greatly from Newton's laws of gravity, even though we still don't have a quantum-compatible set of laws for it. The laws of gravity are still incomprehensible for people, but the approximation that we've observed is still immensely valuable.
No, I gave you a lot more leeway than that. Take any utterly incomprehensible piece of writing from the entire history of civilization and demonstrate its value.
You keep falling back on "incomprehensible for some people" but that wasn't the claim. It was about a text which is incomprehensible in principle; that is, utterly impossible for any human to ever understand.
His examples work fine but you aren't accepting them because they don't confine to your paradox.
Every writing must be comprehensible to at least the author, regardless of whether it has commercial value or not. If I hit the keyboard a few times, I've created writing, but it doesn't mean anything. It is just gibberish and without meaning, so there is nothing to try to comprehend. So if there is something to be comprehended, then at minimum the author should know it.
Therefore what you keep claiming is the only refute of your argument of "an [...] incomprehensible [...] writing" is actually a paradox, and cannot be disproved itself. However "humans comprehending things" is not a paradox, which means that your specific request to beat your paradox is not actually related at all.
His examples disproving the non-paradox version of your challenge (writing incomprehensible to folks other than the original authors) are sufficient to disprove your statement, as he gave examples of both people not comprehending human made and 'God' made writing (the universe/gravity)
That's because his examples are obvious and not at all interesting, since we've already seen them.
His original claim amounts to creating an AI that takes its place above humans as some kind of electronic God, delivering edicts to humanity that we cannot comprehend, but which somehow have value to us. It's unskeptical, pseudo-religious nonsense.
I don't see that at all. Nowhere did this person evoke placing AI above people or even remotely evoke religious imagery. You imagined that yourself, and that creeps me out.
It’s right here:
The age of humans comprehending things is coming to an end [1]
[1] https://news.ycombinator.com/item?id=49312914
That does not place AI above humans and neither does it evoke anything religious.
Look at it this way. AI has destroyed humanity at chess for decades now. That doesn't mean we have placed AI above humans in the general sense.
You have basically totally fabricated what you thought the other person was saying.
Can you give me an example single piece of writing that the author didn't claim they understood? I don't think that any utterly incomprehensible writing exists.
If you want examples where nobody but the author understands it, examples are a dime a dozen.
So you concede the point then. The age of humans comprehending things is not coming to an end. And therein lies the rub. When it comes to intellectual labour:
Understanding == Value
If a mathematician produces something incomprehensible then it has no value. It's meaningless. Indistinguishable from random noise.
An AI which produces incomprehensible text is producing no value. We didn't need to spend trillions of dollars on LLMs to figure that out. Markov chains can do that job perfectly well.
You may have missed the present tense. I didn't say it's over, but that it will be. Please pay attention and don't make straw men.
Again, do you believe that there are documents, of any value, that humans don't understand?
Maybe an LLM could help you notice what I was saying, since it's clearly beyond at least one human's comprehension!
Again, do you believe that there are documents, of any value, that humans don't understand?
There is no value in an undeciphered document until understanding is achieved, just as a lode of gold ore in some asteroid orbiting a distant star has no value until we can fly there and extract it.
If an LLM can help us understanding something then it was not incomprehensible, by definition.
But this is just arguing over word tenses. You think "cannot", they think "could". Somewhere between there is the actual crux of disagreement.
Nothing is in principle impossible to understand. It just takes too long, is inconvenient and/or economically unviable.
I don’t find it hard at all to imagine that an AI comes up with a fundamental proof applicable to physics which results in some widget we can now produce that would otherwise not have been produced yet nobody takes the time to fully comprehend why it works. Somebody could, in principle, devote their lives to it and possibly get it, but for what purpose?
There are plenty of artifacts that, if not impossible for humans to understand, then at least no human has ever completely understood. To start with, the universe as a whole. Despite that, we are able to choose legible pieces of it to model and perform useful actions from.
This will shift your argument--that doesn't count! etc., to the point where it's by construction unsatisfiable and vacuous. And it doesn't matter: an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
One person need not completely understand something for it to have value; it's sufficient for there to exist a shared understanding.
an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
No, that is the entire point. If it is an algorithm which accomplishes something useful, then it is intelligible as such. That which is incomprehensible cannot be understood even in part, so it provides no value as a bit of knowledge (unless you're looking for a strong random number source, I suppose).
So, what you're asking for is an example of someone extracting value out of something that is fundamentally irreducible random noise?
> how many people do you think understand the Linux kernel in full?
"Not many people understand some things fully" is so massively different from "the human mind is incapable of understanding some things that AI will understand for us"
Yeah. We're only starting the journey of building tools better at thinking than the human mind, so expecting me to have examples of things it produces is a little hard, don't you think?
The best I can do is things that are incomprehensible to nearly everyone, but still provide value. There's a small leap of imagination to consider an author that understands it and can show others how to leverage results without understanding be mechanical rather than biological.
Can you give an example of a 4-wheeled vehicle that could move at tens of miles per hour before the automobile was invented?
I know right? And there’s only a market for maybe 5 computers in the whole world.
Paul Krugman (1998): predicted the internet’s economic impact would be no greater than the fax machine’s.
The 1876 Western Union memo dismissing the telephone as having too many shortcomings, and the banker telling Horace Rackham not to invest in Ford because the automobile was a novelty.
We are in good company!
Past performance does not guarantee future returns.
The idea about the goal of mathematics being shared understanding seems to come at a convenient time.
Mathematicians have never been known to communicate their ideas very clearly.
Regardless, even that target llms will likely win - an llm will likely be more efficient at teaching me string theory than a professor in a room with 463 other students.
The llm is the shared understanding.
I don't think "shared knowledge" means "shared knowledge between mathematicians and lay persons" (there is no much point in that, the same way a smartphone technician knows how a smartphone works deep down to the details but there is no big interest for society to have every lay persons being informed about it). I think it means "shared knowledge between mathematicians".
And at this level, while there are anecdotical exceptions, mathematicians have always been pretty decent (with their conferences, workshops, paper publications, international collaborations, ...).
So, it does not mean "teaching the subject", it means "creating a human network of people that share the understanding". LLM can be useful at telling a human, but you still need a human. The point of Tao is not that LLM is not good at providing explanations, it is that "providing explanations" is not the contribution to science, "the human network" is. It's like saying "LLM are great cook, they generate tons of food in space", but the point of having cooks is so that people can eat food and not die. Having LLM generating mathematical proofs is as useless as having LLM generating food that no one can access: the point was never to "generate proofs" or "generate food", the point was "creating a shared human understanding" or "eating the food so human can survive".
I don't get the argument.
I get that there is a cultural benefit to keeping it alive. Just like we ideally want the languages represented at the universities.
But keeping humans in the loop does not appear to be necessary in order to call it science, and certainly not in order to have progress or dessiminate that progress.
I don't have a problem with people doing math. As long that we don't idiomatically hold on to that way of doing things.
I do, however, find it hard to belive that individual humans will play a big role from here and forward, in any scientific desciplines.
The goal is indeed to have progress or disseminate that progress.
The point of Tao is that people see LLM providing "proofs" and are concluding that this is all that is needed to "have progress or disseminate that progress". That is the same mistake of thinking that "generating food" is all that is needed to "have people not dying of hunger".
The hard part of "have progress or disseminate that progress" is the human network. A fundamental point of this human network is that it generate trust, accountability and reliability. Generating "useful new theorem" is useless unless the society also built the trust around the theorem to distinguish it from a fake theorem.
Maybe in the future, we will have AI doing some part of it, but this is a totally different AI animal than the one we are able to have now, and people who think the current AI that we see now is able to do that have no understanding how it works. This is demonstrated by the facts in math: current AI is able to provide math proofs, and yet, a lot of human work is still needed to get progress out of current AI.
I would not bet that individual humans will still play a role as big as today in the future. Maybe AI will be different in the future, but the reality is that we don't have any indication if this is even possible.
Why is there a cultural benefit to keeping it alive, or to having universities? It seems like education is toil that could be automated for those that don't have fun with it.
Humans soon won't need it.
The entire point of writing proofs is for advancing human understanding. A giant dump of symbols that passes the lean compiler is meaningless besides human beings understanding it.
In the field of pure mathematics this might be true, but it has implications regardless for applied math, engineering, and physics.
> The entire point of writing proofs is for advancing human understanding.
Proofs also enable AIs to direct search and generate knowledge. Verifiability is immensely useful for keeping AI grounded.
One might imagine AI generating enormous numbers of hypotheses and then trying to prove or disprove them, and then mine that data for new abstractions and heuristics.
But what does it mean? The theorems are just symbols in lean. The conjectures humans chose are carefully selected to be the questions that are interesting and relevant to our intuition about the real world.
Math often doesn't have applications for hundreds of years and that application is only possible because people deeply understand it and how it applies to the real world.
Generating an endless list of true statements doesn't really do anything, those things are already true regardless of whether someone has written a lean program to model them.
An AI may still be able to apply the results without humans understanding the proof.
Sometimes the purpose of the proof is simply to demonstrate that some construct is a safe assumption for other more interesting work-- and could still serve that purpose even if it was entirely a black box.
No it isn’t, it’s putting it into the corpus which means another LLM doesn’t have to spend a few billion credits the next time.
Is it the AI's fault we can't understand? If the GUT is beyond human comprehension does it matter less? We don't apply this reasoning to other animals or even to less capable humans. Besides, the robots may want to ponder maths for their pleasure.
was. Not is. Was.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
That sentiment makes me cringe. If you understand how LLMs work, you'd know it'll never be possible without a fundamental change in how these work.
We're also supposed to be reaching that point, somehow, without the LLMs ever being intelligent (in the dictionary definition sense, not the "high reasoning model" marketing sense).
Based on observations, the ones who are fooled by the supposed emergent properties, are just that, fools. Any sufficiently unintelligent agent will perceive transformer based LLM text predictors as possessing high intelligence.
Maybe you can enlighten us. In what way aren't LLMs able to make new contributions.
LLMs in agentic harnesses are Turing complete.
To my best knowledge, we don't know of any greater computational model that the brain is a part of, that LLMs are not.
First of all, the argument isn't that LLMs (with I assume some automation) cannot be used in searching a problem space. I'm assuming this is what you're referring to, in terms of contributions?
That's the part where LLMs are used as tools. Which there are plenty of places where they are useful.
Also, do you know what turning completeness is? Why are you bringing that up here?
The crowd that AI psychosis has brought to HN is interesting. But not in the "I'd love to learn more" kind
> That sentiment makes me cringe. If you understand how LLMs work, you'd know
> But not in the "I'd love to learn more" kind
I hope you are able to see the problem in your own communication here.
Computation classes are interesting because they say something about fundamental capabilities.
Two machine that are Turing complete are in theory able to carry out the same computations. They are isomorph mediums of computation.
Regardless. Please keep it sober. If you think you know something, enlighten us. But don't just propagate out lies.
The thing about discussing or explaining constraints, is that it rarely is useful or productive if the other side does not accept (or understand) the reality of them.
Turing completeness is not exactly a high bar, and it's genuinely confusing as to why you bring it up. Your C++ precompiler is exactly as intelligent as whatever is your favorite agentic workflow with whatever harness you're referring to. Both might be Turing complete. Neither are intelligent. But one of them seems to be fooling you to think otherwise.
There have been many times that the C++ precompiler produced some output I couldn't understand. I might even at some point thought it was trying to tell me something profound I was too dumb to comprehend. Turns out it was just a missing semicolon.
Yes, so there is not reason to believe that you can do computations that the precompiler can not do.
There is no reason to believe that that you can not fully simulate intelligence in a C++ precompiler.
The precompiler can be simulated by human intelligence, and human intelligence can simulate a c++ precompiler.
Again, you are the one who arrogantly say they llms can not be intelligent without supplying any argument for such.
> Again, you are the one who arrogantly say they llms can not be intelligent without supplying any argument for such.
Not really. You've provided the arguments yourself, just now. But, you don't understand them. Which, brings me back to the initial remark, as to why this engagement is bound to be unproductive. I'm off to bed. Have a good one.
I'll bite, I think Turing completeness is relevant in that it has to be used to informally argue via the Church-Turing thesis that biological intelligence cannot exceed the power or expressivity of formal neural network models. It thus is a good counterargument to stochastic parrot dismissals of neural net based AI such as LLMs, which really are still black boxes. The issue is not simply "So you think that humans are as powerful as LaTeX (or vice versa), ha ha how absurd!"
Obviously actual intelligence has an ineffable essential aspect, just like unicorn farts do.
"The age of humans comprehending things is coming to an end"
That's something AI companies would really want you to believe.
> That's something AI companies would really want you to believe.
Why would I care what they want me to believe?
Intuitively it would make sense that you can put math ability on a chart with a value for “general public” “smart high schooler” “smart undergrad” “smart PhD/ professional”. And you could place frontier AI somewhere on that chart over time from GPT 2 to now and see the trend.
Then you’d have to consider that either you believe there is a fundamental limit that is below peak human mathematician level or there’s not.
> Why would I care what they want me to believe?
How would you not care? Are you a robot?
They can say random stuff with the goal of increasing their shareholder value. Things they spit out do not have to be true. It is not easy to verify things they say, therefore, everything they say should be taken with a huge grain of salt.
Consider Enron and Amazon at the turn of the millennium. They were both telling you what the future would look like. The right action would’ve been to just ignore what they are saying and try and get data and reason about the world. It didn’t really matter that both Bezos and Jeff Skilling wanted you to believe various things - one was right and one was a scammer.
So that’s what I’m doing here. For what it’s worth I find a lot of the AI people’s worldview very consistent. They believed AI would be the most important technology of our life times and committed their work to it. Some of these same people are total liars so yeah I won’t really hang onto their every word.
He's saying he can connect the dots without their help.
Yeah, my bad.
that does not make it not true, nor does it make those companies or their products not dangerous. I like looking at videos of animals that tear other animals apart and eat them; lion cubs are super cute; but that does not mean I want to be thrown into a cage with a model of a lion that has not been programmed to be disinterested when it is sated. AIs appear never sated; humans using or making AI wanting money, even less so. I suspect the AIs will understand the cost long before the humans will, not that anyone making money would care.
You’re prescribing elegance to a stochastic generator trained on the wealth of humanity, including 4chan. Let’s set our expectations a bit.
Your brain is a stochastic generator. Have you read 4chan?
Whenever someone tell me they can be reduced to an LLM, I believe them. I see you and I believe you that you are qualitatively the same as a language model.
Your brain is not just a stochastic generator.
It also creates and updates models, uses those models to make predictions, guides the stochastic generation by comparing the output to those models and reworks them in real time.
what is it?
It also creates and updates models, uses those models to make predictions, and guides the stochastic generation by comparing the output to those models and reworks them in real time.
Not sure if you're referring to LLMs or humans here
(¬_¬)
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
I agree.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
I don't know if I see this being true for quite a while, if ever.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers. > I agree.
There's an infinite space of possible statements and proofs. The only thing that makes certain proofs significant is that human mathematicians consider them significant; if AI came up with a proof of some statement that no humans could understand then no humans would bother investing further resources in building upon it, for the same reason we don't waste computational resources iterating over the infinite space of true statements in first-order logic.
In limited experimentation: AI will certainly make statements that are extremely intricate and hard to understand, in part because they're overcomplicated and in part because they use a bunch of unnecessary terminology.
This is not to say that a human couldn't understand a streamlined version or that the AI would not be better if it made more streamlined statements to begin with.
(I am not saying that everything mathematical that an AI produces is in any sense trivial.)
It’s possible, but there’s a difference between vastness and difficulty.
Humans can’t compete with AIs on vastness of material they are familiar with, or the depth of effort they are willing and able to throw at a problem.
But scale isn’t the only aspect of difficult scientific endeavours. There’s also theory. And advancements sometimes come through hard graft of knotting together many things. And sometimes they come through the revelation of a deeper truth, or a new framework, a fundamental insight.
AI might help us reach the next level. But that doesn’t mean we won’t understand anything. It could be we have periods of vast intricacy we cannot follow, punctuated by profound elegance we (or at least experts) relatively easily can. And then the scaffolding we needed to get there falls away.
I don't think that's true. Human intelligence is limited, and our brains are inefficient machines.
The tools we built to replace muscles have mostly obsoleted raw strength for tasks like excavating earth.
There's no reason to think we can't do the same for brains. And then we'll never need to think for a living again. Some people may want to do it as a commercially insignificant hobby, of course, the way people lift and compete in strongman competitions today.
We'll have AI taking care of our needs, the way a good mother takes care of their children.
>our brains are inefficient machines
The human brain is exceptionally efficient.
Evidently some already stopped thinking way before the advent of these mythical thinking machines
A good mother doesn't raise children to be dependent upon her for all their needs.
For this to actually work in a way that benefits our species, humans will need to become something else/next through their interaction with the technology.
It's inevitable -- we won't have the machinery to compete, so we either have an aligned AI taking care of us, or we end up with a big problem.
Anyways, sipping wine on the beach and doing puzzles when I feel like sounds nice.
Permanant vacation is great for an individual. I'd love one.
However, for our species, extinction follows domestication.
looking for scraps in the gutter of never seen sun city more like
You could be right, but you're making a lot of assumptions about how complexity, scientific understanding, and explanations scale. One of the features of a good scientific discovery is that it often simplifies and compresses things that were previously a bunch of scattered facts. Also, as AI systems improve they'll get better not only at making scientific discoveries, but also at producing understandable explanations.
Then it's pretty bad that LLMs don't understand anything.
They don't know and can't know. Without an external source of input that corrects them, their output can never be verified.
If and only if that is actually true, then perhaps nothing matters. Until then, calling out shenanigans remains a noble art.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
Ask yourself: is that really the world you want to live in? It's a world where people, all people, are sidelined.
I think the happy ending of that path is something like Idiocracy. And the more likely ending is something like "automated capitalist economy without the people, because the people couldn't compete."
That maybe true at some point, but i don't think we are there yet.
Yeah, it's probably a few years out.
But apparently we can teach machines to do it for us
Yeah. We can also teach machines to move hundreds of miles an hour, but we could never do it ourselves.
>produce proofs far more intricate than humans can understand
Math is not magic, a proof is just a series of applications of a set of rules on some axioms. A mathematician could understand any proof given enough time to study it; the only way for AI to make proofs that a human couldn't understand is by making really, really long proofs.
It's going to produce proofs far more intricate than humans can understand
The thing is, some number of advanced proofs start out "too intricate for most mathematicians to understand" but many of these get rephrase and reframed until they're accessible to undergraduates. Hopefully, AI math can be guided to do that sort of reframing to increase the level of accessible math as well as extend the border of math.
Why would you want something you don't comprehend? How can you be sure it empowers you?
I think perfect rationality doesn't exist, because it is rational to reject something that you don't understand. So rationality of a given physical system will always be bounded.
Let’s not jump the gun. Where they are right now they can somewhat match our abilities. We haven’t even gotten to the point where they can self improve.
This means some AI proofs might be impossible to comprehend by humans, right? I guess AI still lacks human intuition for many concepts, but AI might beat humans in narrow areas, such as discrete math and combinatorics.
Exactly, it's making connection across vast set, not bringing the magic intuition.
Has anyone tried feeding all of human knowledge to an LLM prior to Einstein's work and tried to have it reinvent physics?
Where would you get enough pre internet tokens to train a near-frontier model?
C'mon AI companies, pivot to lawyers or doctors already.
Trying to convince us that mathematics and software engineering are "solved" is getting very tiring.
The pushback would probably be too much for the soon-to-be IPO-ed companies.
Mathematics is typically concerned with "proofs" [1], which similarly to code, often allow for strict validation. Thanks to reinforcement learning techniques, it is now possible to train LLMs to perform very well on code generation, and mathematical proof generation.
Law and medicine are fundamentally harder fields to obtain decent training data for, and LLM results are therefore expected to be less powerful. Also, making mistakes in these fields is costly, but perhaps you were alluding to that already.
[1] https://en.wikipedia.org/wiki/Mathematical_proof
Mathematician here. There is a lot of recent work on the Lean project -- when a proof can be translated into Lean code, then it can be strictly and formally validated.
https://lean-lang.org/
But otherwise, mathematical proofs are read and written by humans, and at the end of the day the relevant standard of proof is what other mathematicians will accept.
Occasionally, mathematicians don't agree. For a prominent example, you can read about Shinichi Mochizuki's claimed proof of the so-called ABC Conjecture:
https://en.wikipedia.org/wiki/Abc_conjecture#Claimed_proofs
For that particular example, it's now been proved that Mochizuki's proof is incomplete: https://zeli.app/en/story/48963019
I guess whether he will eventually fix those gaps and resolve the issues remains to be seen.
> Law and medicine are fundamentally harder fields to obtain decent training data for
I think this bubble has given a lot of people software brain and are trying to apply it to fields it is wholly inappropriate for, though. Law is about argumentation and rhetoric. It is about providing a persuasive argument. This is how it is taught. The actual legal code is a way to formalize parts of it, but increasingly I see people angrily insisting that the only thing that matters is the text.
As you might imagine, I find textualism a load of applesauce, but I don’t think the vast majority of people making this argument even understand textualism as jurisprudence. It seems to stem from Crypto bros and the whole “code is law” argument which is just codswallop.
A neighbor of mine whose husband is a lawyer said it's already part of his regular workflows. OpenAI also already have HIPAA compliant offerings targeting healthcare uses, etc. Of course they already do these things.
They try to sell AI as lawyer or doctor replacements as well. But because it’s HackerNews we are biased towards our domains to see them more often.
Well also you can't just throw AI slop as doctors or lawyers advice and fail multiple times until you find the right answer. With code and math you can have failures 1000 times for every success and still get rewarded.
They can rely on compilers, solvers, theorem provers to validate the generated softwares and maths. That’s what makes it possible to iterate quickly in a loop and self correct. You cannot do that in soft industries like legal and medicine
That is not the point I was making. I am not talking about validating software or maths. It can generate stuff that is valid, but bad and incomprehensible.
What I’m saying is that AI labs are talking so much about software and maths because we already have tools that can say « it’s all good ». That makes it possible and worth it for them to spend 1 week of compute on a problem until the validator passes, then publish marketing pieces. You cannot do the same in medicine or laws (modulo some niche areas)
Look up OpenEvidence
Most used as a Wikipedia-on-the-fly. Replaces UpToDate and other POC tools NOT professionals operating them.
and yet very little access to common sense :)
the article cites the ability to keep a ton of details in working memory as an advantage, I'm not so sure that it is - perhaps it is quality over quantity; a compression of everything known into a smaller set of interlocking patterns should provide a more useful generalization (if correct, of course). The human's ability to perceive 7-10 concepts might just be the tips of the icebergs composed of a gazillion micro-concepts, i.e. our working memory is enormous, but not consciously accessible
also, the older neural capacity estimates are way wrong [1]: dendrites also compute
[1] https://www.yahoo.com/news/science/articles/neuroscientists-...
Is there anyone on the planet who doesn’t think ”thinking” also includes memory?
Choosing what to remember is my biggest challenge!
I love the term "Out-Remembering"! I have been trying to find a way to communicate that "intelligence", "creativity" and so on might be misleading about the true nature of LLMs, and they would better be described as genious "reproducers" as in, they are very capable at reproducing what they have already seen - and they are a bit less capable, but for many use cases still good enough, at reproducing a mix of concepts seen previously.
This also nudges into how to use it best: By knowing where the "piles" of if training data are (i.e. when it comes to a CLI in rust, I just briefly describe the use cases, and I have a very high confidence the code will work exactly as intended by me since there will be a multitude of examples in the training data), one can predict where the LLM is likely to go wrong an prompt/guard accordingly. This skill grows with domain expertise, and is one of the many reasons LLMs can be (and probably should be) used to outsource busy work, but never understanding and learning. ("never" is a not meant literaly of course - I for one am glad that I do not have to wrap my head around CSS and other frontend topics and go straight to the topics that interest me most)
"Out-Remembering" captures that perfectly, I feel. Also goes nice along with "asking it leading questions" as we know how to do in real live; if you want a person (LLM) to confess (produce output tokens) something, sometimes you do that by leading the interogation (chat, context) to where you think the truth lies.
So build a better proof/paper/technique search engine?
It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.
We offload working memory to paper if we want to understand something that does not fit into the regular meat bits.
>Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.
“is obvious” -- that’s what my Russian math professor said in college before skipping the rest of a proof.
But was it?
Mathematicians don’t forget things they haven’t learned. So is it that forgot history, or they didn’t learn parts of history to begin with?
Nobody IRL cares about nerds btw.
Yet it still can't make a symphony
"AI" does not exist, it knows nothing and has no experience of doing or seeing anything to "remember", a data bank filled with the jumbeled accounts of things that humans have said or written about what they may have made up or may have done or may have heard someone else say, and changing the label to "memory" is only that , and can only have the predictable outcomes.
Seeing what AI has been doing to maths conjectures recently, I decided to give theoretical physics a go.
I threw some random "crank" ideas into GPT 5.6 Ultra and let it consume most of my weekly Pro subscription quota.
I'm actually pretty impressed with what can be achieved! If your simply ask it to use adversarial agents and include honest self-ratings, it can produce output competitive with a smart but not exceptional PhD. Not imaginative, but the effort that it sinks into even the smallest matters is just amazing to watch.
Did I get anywhere by applying AI to my toy models? Probably not! I ran out of quota before I could tackle anything really interesting.
But... GPT did seem to discover something genuinely new by essentially brute-force effort of combing through hundreds of papers on ArXiV: a new "constant" extracted from a formula over quark masses that seems to evaluate to exactly 3/4 at the one-loop level and then with various corrections at two-loop and higher. Link: https://chatgpt.com/s/t_6a80f2c794388191971350733a6f4378
This is "novel" in the sense that it found a vaguely similar formula in a paper where the authors simply missed the opportunity to extract a simple ratio from quark masses, combined with updated constants the AI found in in later papers that allowed it to guess that this might be exactly 3/4.
This is the super-power of mechanised attention!
We've produced a truly ludicrous amount of scientific and mathematical output, far past the ability of any single human mind to be even vaguely aware of all of it.
Now we can scrape together all of the jigsaw pieces we have made and see what fits together.
I wouldn't be surprised if someone, perhaps even a hobbyist comes up with a viable theory-of-everything in the next year or two simply by feeding in some clever starting point and then turning the handle on the machine until a complete and consistent theory pops out the other side.
Exciting times!
This misses a very important point.
It’s not just about out remembering, it’s about breadth.
Mathematicians are all about depth. It’s pretty much impossible to become an expert in more than one narrow field of mathematics.
AI is happily applying techniques and abstractions across these silos.
Euler would like a word. Maybe it's our institutions.
which is why the programming language for LLMs hasn't been invented yet. I cannot read someone else's code that is filled with single letter very real names, but I can write it but as a human with human working memory, I can only hold so many variables in my head to reason about an LM is a much greater working memory unless does not need the same abstractions as a human does if I'm no longer riding the code and I don't need to read it, then not only is goto's considered harmful is holding us back, but bring on the big ball of mud spaghetti code that outperforms human written code, that no human has a chance of maintaining or reasoning about.
The problem with this is there is going to be a bug that needs fixing that the LM can't fix and if the humans can't understand the bug that we're just gonna have to live with the bug! But let's not talk about that.
There is a certain amount of slack that we have. Things that are obvious consequences of what we've already discovered but that we haven't taken advantage of yet. AI is reaping that slack. It isn't adding brand new ideas at the moment. We will run out of this slack pretty quickly.
"OK, it might be much better at math than me, but it's not smarter, it just remembers more".
Heh. Every day, a new type of cope. It's like, coping as hard as possible.
If this was an ML researcher trying to come up with a way to improve performance then you could say it wasn't cope but rather practical observation to serve a goal.
But it's just cope.
Also, AI is going to continue to get smarter. A lot smarter. There are already systems in R&D that will continue increasing efficiency and performance of hardware by more orders of magnitude.
As AI keeps improving, the definition of intelligence will keep changing in order to exclude it without explicitly saying so.
Well, some folks are going to keep trying to exclude it, but the reality is we are almost at the point even those folks would concede that they are not smarter than LLM. I suspect we will see then see more discourse about consciousness, morality, agency, emotions as being the most important part of intelligence.
That's exactly what the strength of AI is, no? Reading all the world's knowledge and recalling it in an instant and seeing where it applies.
Precisely perfect for replacing lawyers, if nothing else..
For instance, Opus 5 yesterday critiqued my resume and mentioned a ridiculous little detail-- my phone number area code didn't match the state in which I currently work (I know, I should anonymize but I couldn't be bothered). It failed to notice that I both worked in (previously) and studied in the state of my area code. This was just two pages of text, set to highest effort.
Why would you want to replace your lawyer with a set of tensors that does not actually think and makes mistakes like this? Lawyers tend to get hired in high stakes situations. Why wouldn't you instead say that this would be a great tool for lawyers to use judiciously in researching precedents, etc?
I don't understand what people are doing with models that makes them assign agency or intelligence to them. When I manage to forget the financial fuckery of the AI buildout and its implications, when I manage to forget scaremongering by loathsome CEOs, I still have the same fascination and excitement at the idea of LLMs as I did when I was playing with the GPT API prior to the release of ChatGPT.
LLMs are, to me, truly amazing tech. It's so fascinating to me that they now DO have emergent properties that look at face value like reasoning and intelligence. But every day that I work with them, I am repeatedly clobbered over the head with the fact that they do NOT reason and are NOT intelligent.
Why can't we be fascinated by emergent properties of intelligence without immediately jumping 10 steps into the future and, like a limit in calculus, assume that "this is it-- we're on the cusp of AGI"? To me, the fact that LLMs can combine existing ideas that people hadn't thought of combining in solving a novel problem is extremely cool. But my first thought is-- this is an amazing new tool for mathematicians and researchers. Instead, most everyone seems to jump the gun to the "humans are obsolete next year" conclusion.
> Why can't we be fascinated by emergent properties of intelligence without immediately jumping 10 steps into the future
Because, when computers first came out, and filled entire rooms, people predicted they'd eventually shrink down to fit the crevice of your bum and everyone on the planet would have them
When the first cars came out, people predicted that within 10 years, city streets would no longer be drowning in horse dung
Do people who rant and rail against every new technology in its infancy, actively choose to forget history, or simply didn't learn about the many similar instances in the past?
Specifically working memory. If you can’t hold enough concepts in your head then you can’t see how they all relate in one giant theory.
> But chunking does not eliminate the limit. It merely compresses the information.
Yeah, as expected, an article about AI that's at the very least been polished using AI. For fucks sake we need an LLM flag to filter out slop.
Aka - it’s a stochastic parrot with a good memory, for anyone still struggling to understand this. It should be obvious, imo, but some people seem to have trouble with the concept.
I don’t understand why this is such big news. OpenAI has essentially made a bunch of marketing copy by gussying up an algorithm being given a near unlimited budget to stochastically permute through its lossy memory.
I think the real scandal is that we are almost 3-4 years into this (I think the release of GPT 3.5 is a good marker of when this public frenzy started) and all we’ve seen is OpenAI and the other major AI frontier companies constantly retracting their preposterous claims every time. We appear to have reach a local maxima in that it has some value in places that tend to be a little easier to scope and limit (computer programming, mathematical proofs). So, given the actual useful economic value this has provided, does this justify the investments? I think we are approaching 1 trillion in CapEx for AI [0]. For context, I believe the annual GDP of Norway is $600 billion.
[0] https://www.fool.com/research/ai-companies-spending-on-data-...
It's crazy, you see someone spin up 50 instances of chatGPT or gemini or whatever to tackle a math problem, it succeeds by brute forcing through a ton of existing theories to find one that extends the problem, and the takeaway is that this technology is magic and going to solve all of our problems.
Whereas I see that and say - if we properly funded the sciences we could have had a bunch of grad students tackling that problem and found this application 20-30 years ago. Sure it's 'nice' that LLMs can fill in for people in brute force work like that but people are perfectly capable of doing that work and if we focused on properly staffing our research institutions we would achieve a lot more a lot faster. Instead this is obviously going to be used to replace staff and further reduce headcounts.
Metacommentary: how did this post get to #5 on the front page with 1 upvote within 2 minutes of submission?
That’s how HN works, I had that multiple times over the years with my own submissions. Sometimes it gets picked up quickly, sometimes not. A post can also down rank very, very fast. It depends a lot on the level of engagement and the type of engagement
Follow the money.
How fast the upvote happened?
And the goalposts must move once again..
"It's not X, it's Y" hot take AI slop.
100%. Context is big for AI, but it's nothing compared to everything a human can learn. If you efficiently represent everything in context, it may be many papers, but if AI is actively working through proofs, it will quickly fill up. They're no denying AI is making strides, but pinning it to memory is an oversimplification.
I don't have to open the article to be confident it's not worth reading. Anybody knowledgeable in the field should be familiar with AI writing tells and the message they send. It only takes a few seconds thought to transform the title into something like "AI beats mathematicians by out-remembering, not out-thinking." Regardless of whether the article is slop or not, I expect any competent writer to avoid slop phrasing in their titles. To do otherwise signals laziness.
Most mathematicians are quite simple creatures. I can do basic math, some derivations, but my bright days of solving differential equations are far gone!
Computers are simply better at math now, like in chess or go!