It's fascinating to see these benchmarks being saturated where, a priori, solving it seems to require human-level general intelligence, yet when you try the models they are still clearly not broadly human-level. I am very curious to see if a particularly clever set of benchmarks is all you need to elicit genuine human-level intelligence.
I think also there's an element of these benchmarks being static (and therefore gameable) which is holding us back? I know we do multi turn assessment yadda yadda but I wonder if we had an adaptive framework of sorts which could actually adapt to model behaviour whilst dynamically leveraging some kind of internal scoring criteria knowledge graph it might at least address the issue slightly? Reality is just not matching these benchmarks (or any of those which I've built privately for assessing models even!)
It's fascinating to see these benchmarks being saturated where, a priori, solving it seems to require human-level general intelligence, yet when you try the models they are still clearly not broadly human-level. I am very curious to see if a particularly clever set of benchmarks is all you need to elicit genuine human-level intelligence.
Maybe the problem is the concept of "benchmark" itself.
Maybe we do better at things that cannot be a benchmark, i.e. uncertainty and randomness.
You cannot put things that cannot be graded into a benchmark.
I think also there's an element of these benchmarks being static (and therefore gameable) which is holding us back? I know we do multi turn assessment yadda yadda but I wonder if we had an adaptive framework of sorts which could actually adapt to model behaviour whilst dynamically leveraging some kind of internal scoring criteria knowledge graph it might at least address the issue slightly? Reality is just not matching these benchmarks (or any of those which I've built privately for assessing models even!)
And yet we make children take exams to prove their value
Looking forward to see what they come up with for ARC-AGI-4.
100% 'with adaptor' and 70.3% in standard mode...
[dead]
So is this AGI?
François Chollet answered that here:
> Many of you will ask, "if it saturates ARC 3, is it AGI?"
https://x.com/fchollet/status/2095599835932135919
Better: it passes an AGI benchmark
Yes, according to this vibe coded https://arcprize.org/ website that is the "North Star for AGI" and is sponsored by OpenAI, Google, xI and Anthropic.
Time for a new acronym
AGOD
Super Intelligence inbound
"Big, Beautiful Intelligence": BIBI.